On the morning of Tuesday the disk space used by the database servers started expanding at a much higher rate than previously. We investigated the cause of the issue, which seemed potentially linked to cached waste data being stored in sessions for performance reasons, and made the decision at 5pm to bring forward the maintenance downtime already scheduled for Echo (starting just before 8pm) in order to try and stop the increase in disk use and prevent the system from running out of space entirely. This did halt the increase of disk use any further.
Later that evening we doubled the size of the disk on the database server, and ran a full vacuum on the session table causing the issue, which successfully reclaimed all the space used up earlier in the day. The service was then fully restored at 11pm.
Last updated: 27 March 2024 at 9:36 AM