AU Diaries Degraded Performance

Incident Report for ResDiary

Postmortem

All the times in this report are in AEST.

At around 12:24 a large number of requests started to build up on the web servers hosting au.resdiary.com. An engineer was notified at 12:29, and after an initial investigation, began recycling the web server processes at 12:40 in order to get rid of the backlog. This appeared to reduce the problem, but did not completely solve it. At around 12:49 our engineer tried recycling the web server process again on two of the web servers, after which normal operations were restored.

We are currently investigating ways of protecting our servers from incidents like this, and of improving our resilience in general, and we sincerely apologise for any inconvenience caused.

Posted Mar 27, 2019 - 10:18 UTC

Resolved

AU diaries appear to be working correctly again. We will continue to monitor the situation, and will provide a full post-mortem once we know more.

Posted Mar 25, 2019 - 03:22 UTC

Identified

Engineers have identified an issue causing a large build up of queued requests on the Australian server. Long running requests that were never going to complete have been stopped, which has allowed normal functionality to be restored.

Posted Mar 25, 2019 - 02:59 UTC

Investigating

We are currently investigating issues reported affecting diaries hosted in our au.resdiary.com environment.

Posted Mar 25, 2019 - 02:36 UTC

This incident affected: ResDiary Application (Australia).