WasteWorks on SocietyWorks Status

[Resolved] Scheduled Maintenance on 25th February 2026

Wed, 25 Feb 2026 05:00:00 +0000

We will be carrying out some scheduled maintenance on our services on 25th February 2026 between the hours of 5am and 7am. There may be short periods of downtime, which should last no longer than a few seconds in most cases, while this is completed.

Thanks for your understanding.

[Resolved] DDOS attack

Thu, 04 Dec 2025 08:45:00 +0000

Services are now working as normal. We’ll now look into further details of the source of the DDOS and any further mitigations.

09:06: We have temporarily removed the stats from the front page as a potential cause of the thundering herd problem. The services should be working again.

08:59: We have blocked the main source of traffic; the database has undergone a ’thundering herd’ problem due to the traffic; we are looking at calming it down.

We appear to be undergoing a DDOS attack which is affecting our services and making them intermittently unavailable; we are actively looking at mitigations and will update here.

[Resolved] Scheduled Maintenance on 27th August 2025

Wed, 27 Aug 2025 05:10:00 +0000

We will be carrying out some scheduled maintenance on our services on 27th August 2025 between the hours of 5am and 7am. There may be short periods of downtime, which should last no longer than a few seconds in most cases, while this is completed.

Thanks for your understanding.

[Resolved] Intermittent Echo availability

Wed, 30 Jul 2025 09:39:00 +0000

11:57 The issue with Echo has been resolved and WasteWorks & FMS Pro are operating normally.

09:39 We are aware of an ongoing issue with the third-party system Echo, meaning some property lookups on WasteWorks are failing and reports made on FixMyStreet Pro are being held in a queue.

[Resolved] Scheduled Maintenance on 28th May 2025

Wed, 28 May 2025 05:00:00 +0000

We will be carrying out some scheduled maintenance on our services on 28th May 2025 between the hours of 5am and 7am. There may be short periods of downtime, which should last no longer than a few seconds in most cases, while this is completed.

Thanks for your understanding.

[Resolved] Scheduled Maintenance on 25th February 2025

Tue, 25 Feb 2025 05:00:00 +0000

We will be carrying out some scheduled maintenance on our services on 25th February 2025 between the hours of 5am and 7am. There may be short periods of downtime, which should last no longer than a few seconds in most cases, while this is completed.

Thanks for your understanding.

[Resolved] Scheduled maintenance on 27 November 2024

Wed, 27 Nov 2024 06:00:00 +0000

We will be carrying out some scheduled maintenance on our services on 27 November 2024 between the hours of 6am and 7am. There may be short periods of downtime, which should last no longer than a few seconds in most cases, while this is completed.

Thanks for your understanding.

[Resolved] Scheduled maintenance Wednesday 21 August

Wed, 21 Aug 2024 05:00:00 +0000

We will be carrying out some scheduled maintenance on SocietyWorks services on Wednesday 21 August between 5 and 7AM.

This will involve some short periods of downtime which should last no longer than around 10 minutes as we perform essential maintenance on our database systems. Thank you in advance for your patience and understanding.

[Resolved] Scheduled maintenance 25 June 2024

Tue, 25 Jun 2024 05:00:00 +0000

We will be carrying out some scheduled, proactive maintenance on our file storage systems as part of capacity planning on Tuesday 25 June 2024.

This will involve some short periods of downtime between the hours of 5am and 7am affecting all SocietyWorks services.

Thank you for your understanding.

[Resolved] Scheduled maintenance on 22 May 2024

Wed, 22 May 2024 05:00:00 +0000

We will be carrying out some scheduled maintenance affecting all of our services on Wednesday 22 May 2024 between 5am - 7am. This may involve some short periods of downtime which should last no longer than a few minutes.

[Resolved] Quarterly maintenance window

Thu, 29 Feb 2024 05:00:00 +0000

We will be carrying out some scheduled database maintenance affecting all of our services on Thursday 29 February 2024 between 5 AM - 6 AM. This may involve some short periods of downtime which should last no longer than a few minutes.

File server maintenance

Wed, 21 Feb 2024 05:00:00 +0000

Between the hours of 5am - 7am we will be performing maintenance on our file storage clusters. We don’t anticipate this will cause any downtime, but some services may be at risk while this is carried out.

[Resolved] Echo unavailable

Wed, 24 Jan 2024 18:17:00 +0000

08:16, 25 January WasteWorks availability was restored when the Echo API came back online.

18:17, 24 January The Echo API has stopped responding, meaning our WasteWorks services that use Echo (Brent, Bromley, Kingston, Sutton) are currently unavailable.

[Resolved] Database upgrade

Fri, 12 Jan 2024 05:00:00 +0000

The database server and associated websites were taken offline briefly in order to increase the amount of cores and memory on the server.

[Resolved] Database disk upgrade

Tue, 09 Jan 2024 22:45:00 +0000

The database server and associated websites were taken offline briefly in order to increase the size of the main disk and reclaim unused space.

[Resolved] Database disk space

Tue, 09 Jan 2024 17:00:00 +0000

On the morning of Tuesday the disk space used by the database servers started expanding at a much higher rate than previously. We investigated the cause of the issue, which seemed potentially linked to cached waste data being stored in sessions for performance reasons, and made the decision at 5pm to bring forward the maintenance downtime already scheduled for Echo (starting just before 8pm) in order to try and stop the increase in disk use and prevent the system from running out of space entirely. This did halt the increase of disk use any further.

Later that evening we doubled the size of the disk on the database server, and ran a full vacuum on the session table causing the issue, which successfully reclaimed all the space used up earlier in the day. The service was then fully restored at 11pm.

[Resolved] High load and traffic on FixMyStreet

Tue, 02 Jan 2024 12:27:00 +0000

3rd January

We alerted you yesterday to some elevated traffic levels across FixMyStreet and WasteWorks which had intermittently affected the performance of both services.

We identified and blocked several sources of problematic traffic, but were unable to find a definitive single root cause. We also identified record-breaking levels of usage, even for the start of January when it’s not uncommon to see an uptick in reports made due to, for example, bad weather.

We have put more resources in place to handle the increased traffic and are confident that everything is now stable and working as it should. We will continue to monitor the situation, and will get back in touch if anything else changes.

2nd January

This service announcement is to acknowledge that FixMyStreet and WasteWorks have been experiencing elevated traffic levels this afternoon, 2 January 2024, which has had an impact on performance of the web service.

We have identified and blocked several sources of this traffic, but have not yet got to the underlying cause. We are actively investigating this issue and working on a resolution. Integrations are unaffected - reports and updates are continuing to be processed and sent as usual.

Quarterly maintenance window

Thu, 23 Nov 2023 05:00:00 +0000

This is our regular scheduled maintenance window.

During this window we performed database upgrades and restarts.

[Resolved] DDOS attack

Mon, 19 Jun 2023 11:17:18 +0000

11:24 Incoming traffic analysed and measures to handle the malicious requests put in place close to the edge of our network, largely restoring service. However, the sheer amount of traffic meant that not all requests for pages were being serviced, or that responses were slower than expected.

11:17 Automated monitoring systems alerted us to the problem and we commenced work on restoring service as a priority following our internal incident management procedures.

[Resolved] DDOS attack

Mon, 19 Jun 2023 11:17:18 +0000

Post Incident Actions

We have maintained the increased levels of compute resources across our network edge; this remains under review.

The used domain name has remained behind Cloudflare’s service.

As some of our external IP addresses were cached during the attack, we have deployed additional addresses and are reviewing distribution of services across our front-end systems.

We have added further additional capacity to our front-end to permit complete segregation for at-risk services to third-party CDNs and WAF (web application firewall) services who provide these services at scale.

Timeline

20th June, 07:30 The attack ended - traffic levels returned to normal.

21:30 The level of malicious traffic dropped back to the previous level of c. 7,500 requests a second.

19:00 (onwards) We maintained an increased level of active monitoring by staff through the evening as the attack continued.

18:30 The level of the attack increased from c. 7,500 requests a second to c. 9,500 requests a second. We were able to absorb the increase in traffic without further impact to service.

16:49 By this time, the emergency change to move the domain name behind Cloudflare was complete, however the attack against our systems continued as our internet-facing IP addresses appeared to have been cached at an earlier phase of the attack.

16:36 Further service advisory was sent to all clients.

14:44 Final scaling up of our front-end infrastructure to a level where the residual malicious traffic was contained was completed. At this point, although still under a sustained attack of approximately 7,500 requests a second, our service was now responding normally.

14:00 Joint SocietyWorks/client conference call. Agreed to begin the process of raising an emergency change to move their domain behind their existing Cloudflare CDN (content distribution network) service.

13:45 Initial phase of scaling front-end systems completed and being actively monitored for impact on service.

13:35 We received confirmation on a phone call that the client was under a wider attack and that this wasn’t limited to us.

13:20 Geofencing implemented for traffic originating from Russian and Ukrainian IP addresses, further reducing the impact.

12:50 Several network providers identified from where a significant proportion of the malicious traffic originated. Blocked these at the network level, reducing impact on our systems.

12:20 Service advisory was sent to all clients.

11:17 Automated monitoring systems alerted us to the problem and we commenced work on restoring service as a priority following our internal incident management procedures.