WasteWorks FixMyStreet Pro Network
We have maintained the increased levels of compute resources across our network edge; this remains under review.
The used domain name has remained behind Cloudflare’s service.
As some of our external IP addresses were cached during the attack, we have deployed additional addresses and are reviewing distribution of services across our front-end systems.
We have added further additional capacity to our front-end to permit complete segregation for at-risk services to third-party CDNs and WAF (web application firewall) services who provide these services at scale.
20th June, 07:30 The attack ended - traffic levels returned to normal.
21:30 The level of malicious traffic dropped back to the previous level of c. 7,500 requests a second.
19:00 (onwards) We maintained an increased level of active monitoring by staff through the evening as the attack continued.
18:30 The level of the attack increased from c. 7,500 requests a second to c. 9,500 requests a second. We were able to absorb the increase in traffic without further impact to service.
16:49 By this time, the emergency change to move the domain name behind Cloudflare was complete, however the attack against our systems continued as our internet-facing IP addresses appeared to have been cached at an earlier phase of the attack.
16:36 Further service advisory was sent to all clients.
14:44 Final scaling up of our front-end infrastructure to a level where the residual malicious traffic was contained was completed. At this point, although still under a sustained attack of approximately 7,500 requests a second, our service was now responding normally.
14:00 Joint SocietyWorks/client conference call. Agreed to begin the process of raising an emergency change to move their domain behind their existing Cloudflare CDN (content distribution network) service.
13:45 Initial phase of scaling front-end systems completed and being actively monitored for impact on service.
13:35 We received confirmation on a phone call that the client was under a wider attack and that this wasn’t limited to us.
13:20 Geofencing implemented for traffic originating from Russian and Ukrainian IP addresses, further reducing the impact.
12:50 Several network providers identified from where a significant proportion of the malicious traffic originated. Blocked these at the network level, reducing impact on our systems.
12:20 Service advisory was sent to all clients.
11:24 Incoming traffic analysed and measures to handle the malicious requests put in place close to the edge of our network, largely restoring service. However, the sheer amount of traffic meant that not all requests for pages were being serviced, or that responses were slower than expected.
11:17 Automated monitoring systems alerted us to the problem and we commenced work on restoring service as a priority following our internal incident management procedures.
Last updated: 19 November 2024 at 12:22 PM