AU Diary Application Slow/Unresponsive

Incident Report for ResDiary

Resolved

At 08:03 UTC on Saturday 15th December internal monitoring alerted emergency engineers to a build up of traffic on the web servers in the AU region. This build up resulted in requests being slow to return and meant the diary did not respond in a timely fashion.

Engineers monitored the situation and at 08:11 the servers began recovering without manual intervention. The recovery continued until roughly 08:20 at which point the application was back to normal.

The cause of the traffic build up was traced to a slow running database query. The database recognised that the query was being executed inefficiently and sometime between 08:05 and 08:10 automatically generated a more efficient version.

ResDiary engineers have also rewritten the particular query to greatly reduce the chances of it being used inefficiently in the future.

Posted Dec 15, 2018 - 08:03 UTC