SocietyWorks Status

[Resolved] OS API unresponsive

Fri, 05 Jun 2026 10:40:00 +0000

There is currently an issue with the OS API, which means OS background tiling is not working. We are switching FixMyStreet to OpenStreetMap map tiling until it is resolved.

Please note the OS status page seems to always show things are okay between 0 and 10 past the hour, even if they are not.

[Resolved] Echo availability

Fri, 05 Jun 2026 08:00:00 +0000

08:00 There is currently an issue with Echo, leading to possible issues fetching information from and sending data to Echo.

[Resolved] OS API unresponsive

Thu, 04 Jun 2026 11:25:00 +0000

There is currently an issue with the OS API, which means OS background tiling is not working. We are switching FixMyStreet to OpenStreetMap map tiling until it is resolved.

Please note the OS status page seems to always show things are okay between 0 and 10 past the hour, even if they are not.

[Resolved] Intermittent availability issues

Wed, 20 May 2026 09:15:00 +0000

11:00 Echo appears to be responding normally and WasteWorks service has been restored.

09:50 FMS Pro service has been restored. WasteWorks Echo integrations are still offline due to ongoing issues with availability of the third-party Echo API.

09:45 We’re aware of intermittent issues affecting availability of FMS Pro and WasteWorks this morning. We are actively investigating.

[Resolved] Scheduled Maintenance on 20th May 2026

Wed, 20 May 2026 05:00:00 +0000

We will be carrying out some scheduled maintenance on our services on Wednesday 20th May 2026 between the hours of 5am and 7am. There may be short periods of downtime, which should last no longer than a few seconds in most cases, while this is completed.

Thanks for your understanding.

[Resolved] Echo availability

Thu, 30 Apr 2026 10:45:00 +0000

10:45 There is currently an issue with Echo, leading to possible issues fetching information from and sending data to Echo.

[Resolved] Scheduled Maintenance on 21st April 2026

Tue, 21 Apr 2026 05:00:00 +0000

We will be carrying out some scheduled maintenance on our services on Tuesday 21st April 2026 between the hours of 5am and 7am. There may be short periods of downtime, which should last no longer than a few seconds in most cases, while this is completed.

Thanks for your understanding.

[Resolved] Echo availability

Fri, 17 Apr 2026 06:30:00 +0000

08:15 The issue appears to be resolved.

06:30 There is currently an issue with Echo, leading to possible issues fetching information from and sending data to Echo.

[Resolved] Scheduled Maintenance on 25th February 2026

Wed, 25 Feb 2026 05:00:00 +0000

We will be carrying out some scheduled maintenance on our services on 25th February 2026 between the hours of 5am and 7am. There may be short periods of downtime, which should last no longer than a few seconds in most cases, while this is completed.

Thanks for your understanding.

[Resolved] DDOS attack

Thu, 04 Dec 2025 08:45:00 +0000

Services are now working as normal. We’ll now look into further details of the source of the DDOS and any further mitigations.

09:06: We have temporarily removed the stats from the front page as a potential cause of the thundering herd problem. The services should be working again.

08:59: We have blocked the main source of traffic; the database has undergone a ’thundering herd’ problem due to the traffic; we are looking at calming it down.

We appear to be undergoing a DDOS attack which is affecting our services and making them intermittently unavailable; we are actively looking at mitigations and will update here.

[Resolved] Echo availability

Thu, 23 Oct 2025 10:40:00 +0000

12:03 Selected Interventions have resolved the issue with Echo and WasteWorks is now fully operational again.

11:11 The WasteWorks Echo integration is now offline while the issue persists.

11:06 There is currently an issue with Echo, preventing our Echo-based waste sites from being able to display the correct information.

[Resolved] OS API unresponsive

Mon, 20 Oct 2025 09:00:00 +0000

12:45: FixMyStreet has switched back to using the OS API.

There is currently an issue with the OS API, which means OS background tiling is not working. We are switching FixMyStreet to OpenStreetMap map tiling until it is resolved.

Please note the OS status page seems to always show things are okay between 0 and 10 past the hour, even if they are not.

[Resolved] Echo availability

Mon, 06 Oct 2025 12:45:00 +0000

There is currently an issue with Echo, preventing our Echo-based waste sites from being able to return information.

[Resolved] OS API unresponsive

Tue, 09 Sep 2025 14:40:00 +0000

07:30: FixMyStreet has switched back to using the OS API.

19:15: The OS API status page says all is resolved, we will leave it overnight and revert to using in the morning if still clear.

16:40: The OS API is still degraded, so we are switching FixMyStreet to OpenStreetMap map tiling until it is resolved.

15:00: There is currently an issue with the OS API, which means background tiling may be timing out (note the OS status page always shows things are okay between 0 and 10 past the hour).

If you use OS Places, this may also affect usage of that, e.g. if you include an address in your CSV export. We will look to backfill missing entries once the API is operational again.

[Resolved] Scheduled Maintenance on 27th August 2025

Wed, 27 Aug 2025 05:10:00 +0000

We will be carrying out some scheduled maintenance on our services on 27th August 2025 between the hours of 5am and 7am. There may be short periods of downtime, which should last no longer than a few seconds in most cases, while this is completed.

Thanks for your understanding.

[Resolved] Agile unavailable

Sat, 16 Aug 2025 14:45:00 +0000

11:05 Agile has started responding again.

We are aware of an ongoing issue with the third-party system Agile, meaning garden subscriptions cannot be looked up on councils using that system.

[Resolved] Intermittent Echo availability

Wed, 30 Jul 2025 09:39:00 +0000

11:57 The issue with Echo has been resolved and WasteWorks & FMS Pro are operating normally.

09:39 We are aware of an ongoing issue with the third-party system Echo, meaning some property lookups on WasteWorks are failing and reports made on FixMyStreet Pro are being held in a queue.

[Resolved] Scheduled Maintenance on 28th May 2025

Wed, 28 May 2025 05:00:00 +0000

We will be carrying out some scheduled maintenance on our services on 28th May 2025 between the hours of 5am and 7am. There may be short periods of downtime, which should last no longer than a few seconds in most cases, while this is completed.

Thanks for your understanding.

[Resolved] Echo unresponsive

Tue, 01 Apr 2025 09:50:00 +0000

There is currently an issue with Echo, preventing our Echo-based waste sites from being able to return information.

[Resolved] Scheduled Maintenance on 25th February 2025

Tue, 25 Feb 2025 05:00:00 +0000

We will be carrying out some scheduled maintenance on our services on 25th February 2025 between the hours of 5am and 7am. There may be short periods of downtime, which should last no longer than a few seconds in most cases, while this is completed.

Thanks for your understanding.

[Resolved] Whitespace unresponsive

Thu, 28 Nov 2024 16:05:00 +0000

There is currently an issue with Whitespace, preventing our Whitespace-based waste sites from being able to return information.

[Resolved] Echo unresponsive

Wed, 27 Nov 2024 14:30:00 +0000

There is currently an issue with Echo, preventing our Echo-based waste sites from being able to return information.

[Resolved] Scheduled maintenance on 27 November 2024

Wed, 27 Nov 2024 06:00:00 +0000

We will be carrying out some scheduled maintenance on our services on 27 November 2024 between the hours of 6am and 7am. There may be short periods of downtime, which should last no longer than a few seconds in most cases, while this is completed.

Thanks for your understanding.

[Resolved] Echo unresponsive

Wed, 02 Oct 2024 18:30:00 +0000

There is currently an issue with Echo, preventing our Echo-based waste sites from being able to return information.

[Resolved] Scheduled maintenance Wednesday 21 August

Wed, 21 Aug 2024 05:00:00 +0000

We will be carrying out some scheduled maintenance on SocietyWorks services on Wednesday 21 August between 5 and 7AM.

This will involve some short periods of downtime which should last no longer than around 10 minutes as we perform essential maintenance on our database systems. Thank you in advance for your patience and understanding.

[Resolved] Backend systems with delivery problems

Fri, 19 Jul 2024 10:00:00 +0000

12:45 Confirm also now appears to be working and has accepted all waiting reports.

10:45 Symology now appears to be working and has accepted all waiting reports.

10:00 Whilst SocietyWorks itself is unaffected, some of the endpoints to which we send reports are currently reporting errors which we assume are due to the update to Crowdstrike anti-virus software causing issues with many Windows systems. Currently, this appears to include some, but not all, Confirm and Symology endpoints.

[Resolved] Echo having connection issues

Fri, 12 Jul 2024 10:45:00 +0000

There is currently an issue with postcode lookup on Echo, preventing our Echo-based waste sites from being able to return information.

[Resolved] Scheduled maintenance 25 June 2024

Tue, 25 Jun 2024 05:00:00 +0000

We will be carrying out some scheduled, proactive maintenance on our file storage systems as part of capacity planning on Tuesday 25 June 2024.

This will involve some short periods of downtime between the hours of 5am and 7am affecting all SocietyWorks services.

Thank you for your understanding.

[Resolved] Scheduled maintenance on 22 May 2024

Wed, 22 May 2024 05:00:00 +0000

We will be carrying out some scheduled maintenance affecting all of our services on Wednesday 22 May 2024 between 5am - 7am. This may involve some short periods of downtime which should last no longer than a few minutes.

[Resolved] Bulky collections failing to show dates

Mon, 20 May 2024 09:15:00 +0000

There is currently an issue with bulky or small items collections failing to return dates for booking a collection.

[Resolved] Bulky collections failing to submit

Tue, 26 Mar 2024 10:30:00 +0000

27th March, 09:30 Bulky/small item collection has been re-enabled, after confirmation the issue in Echo with pre-bookable slots has been resolved. We will make some changes to our system to try and work around this situation in future should it happen again.

26th March, 16:45 Bulky/small item collection has currently been switched off until this is rectified.

26th March, 10:30 There is currently an issue with bulky or small items collections returning an error when submitted to Echo.

[Resolved] Quarterly maintenance window

Thu, 29 Feb 2024 05:00:00 +0000

We will be carrying out some scheduled database maintenance affecting all of our services on Thursday 29 February 2024 between 5 AM - 6 AM. This may involve some short periods of downtime which should last no longer than a few minutes.

File server maintenance

Wed, 21 Feb 2024 05:00:00 +0000

Between the hours of 5am - 7am we will be performing maintenance on our file storage clusters. We don’t anticipate this will cause any downtime, but some services may be at risk while this is carried out.

[Resolved] Intermittent Echo errors

Tue, 06 Feb 2024 09:00:00 +0000

When attempting to post new events to the Echo API, we are intermittently receiving errors such as “503 Service Available” or “500 System.ServiceModel.ServiceActivationException”. This will be slowing down the sending of new reports through to Echo. Reports will be retried until they are accepted by the API.

[Resolved] Echo unavailable

Wed, 24 Jan 2024 18:17:00 +0000

08:16, 25 January WasteWorks availability was restored when the Echo API came back online.

18:17, 24 January The Echo API has stopped responding, meaning our WasteWorks services that use Echo (Brent, Bromley, Kingston, Sutton) are currently unavailable.

[Resolved] Database upgrade

Fri, 12 Jan 2024 05:00:00 +0000

The database server and associated websites were taken offline briefly in order to increase the amount of cores and memory on the server.

[Resolved] Database disk upgrade

Tue, 09 Jan 2024 22:45:00 +0000

The database server and associated websites were taken offline briefly in order to increase the size of the main disk and reclaim unused space.

[Resolved] Database disk space

Tue, 09 Jan 2024 17:00:00 +0000

On the morning of Tuesday the disk space used by the database servers started expanding at a much higher rate than previously. We investigated the cause of the issue, which seemed potentially linked to cached waste data being stored in sessions for performance reasons, and made the decision at 5pm to bring forward the maintenance downtime already scheduled for Echo (starting just before 8pm) in order to try and stop the increase in disk use and prevent the system from running out of space entirely. This did halt the increase of disk use any further.

Later that evening we doubled the size of the disk on the database server, and ran a full vacuum on the session table causing the issue, which successfully reclaimed all the space used up earlier in the day. The service was then fully restored at 11pm.

[Resolved] High load and traffic on FixMyStreet

Tue, 02 Jan 2024 12:27:00 +0000

3rd January

We alerted you yesterday to some elevated traffic levels across FixMyStreet and WasteWorks which had intermittently affected the performance of both services.

We identified and blocked several sources of problematic traffic, but were unable to find a definitive single root cause. We also identified record-breaking levels of usage, even for the start of January when it’s not uncommon to see an uptick in reports made due to, for example, bad weather.

We have put more resources in place to handle the increased traffic and are confident that everything is now stable and working as it should. We will continue to monitor the situation, and will get back in touch if anything else changes.

2nd January

This service announcement is to acknowledge that FixMyStreet and WasteWorks have been experiencing elevated traffic levels this afternoon, 2 January 2024, which has had an impact on performance of the web service.

We have identified and blocked several sources of this traffic, but have not yet got to the underlying cause. We are actively investigating this issue and working on a resolution. Integrations are unaffected - reports and updates are continuing to be processed and sent as usual.

Quarterly maintenance window

Thu, 23 Nov 2023 05:00:00 +0000

This is our regular scheduled maintenance window.

During this window we performed database upgrades and restarts.

[Resolved] Memory usage on FOIWorks

Sun, 10 Sep 2023 18:38:00 +0000

A process handling a particular PDF caused the server to use all available memory, which then led to the web server being killed. The server was automatically restarted at 19:02.

Quarterly maintenance window

Wed, 16 Aug 2023 05:00:00 +0000

This is our regular scheduled maintenance window, if required. Non-essential maintenance that requires downtime can be scheduled in this slot. Currently no work is scheduled.

[Resolved] DDOS attack

Mon, 19 Jun 2023 11:17:18 +0000

11:24 Incoming traffic analysed and measures to handle the malicious requests put in place close to the edge of our network, largely restoring service. However, the sheer amount of traffic meant that not all requests for pages were being serviced, or that responses were slower than expected.

11:17 Automated monitoring systems alerted us to the problem and we commenced work on restoring service as a priority following our internal incident management procedures.

[Resolved] DDOS attack

Mon, 19 Jun 2023 11:17:18 +0000

Post Incident Actions

We have maintained the increased levels of compute resources across our network edge; this remains under review.

The used domain name has remained behind Cloudflare’s service.

As some of our external IP addresses were cached during the attack, we have deployed additional addresses and are reviewing distribution of services across our front-end systems.

We have added further additional capacity to our front-end to permit complete segregation for at-risk services to third-party CDNs and WAF (web application firewall) services who provide these services at scale.

Timeline

20th June, 07:30 The attack ended - traffic levels returned to normal.

21:30 The level of malicious traffic dropped back to the previous level of c. 7,500 requests a second.

19:00 (onwards) We maintained an increased level of active monitoring by staff through the evening as the attack continued.

18:30 The level of the attack increased from c. 7,500 requests a second to c. 9,500 requests a second. We were able to absorb the increase in traffic without further impact to service.

16:49 By this time, the emergency change to move the domain name behind Cloudflare was complete, however the attack against our systems continued as our internet-facing IP addresses appeared to have been cached at an earlier phase of the attack.

16:36 Further service advisory was sent to all clients.

14:44 Final scaling up of our front-end infrastructure to a level where the residual malicious traffic was contained was completed. At this point, although still under a sustained attack of approximately 7,500 requests a second, our service was now responding normally.

14:00 Joint SocietyWorks/client conference call. Agreed to begin the process of raising an emergency change to move their domain behind their existing Cloudflare CDN (content distribution network) service.

13:45 Initial phase of scaling front-end systems completed and being actively monitored for impact on service.

13:35 We received confirmation on a phone call that the client was under a wider attack and that this wasn’t limited to us.

13:20 Geofencing implemented for traffic originating from Russian and Ukrainian IP addresses, further reducing the impact.

12:50 Several network providers identified from where a significant proportion of the malicious traffic originated. Blocked these at the network level, reducing impact on our systems.

12:20 Service advisory was sent to all clients.

11:17 Automated monitoring systems alerted us to the problem and we commenced work on restoring service as a priority following our internal incident management procedures.

Quarterly maintenance window

Wed, 17 May 2023 05:00:00 +0000

This is our regular scheduled maintenance window, if required. Non-essential maintenance that requires downtime can be scheduled in this slot.

During this window, we upgraded the databases, activated new kernels on some servers, and migrated the extra data columns of FixMyStreet to use a more standard storage format.