AU Diaries Degraded Performance
Incident Report for ResDiary
Postmortem

All the times in this report are in AEST.

At around 12:24 a large number of requests started to build up on the web servers hosting au.resdiary.com. An engineer was notified at 12:29, and after an initial investigation, began recycling the web server processes at 12:40 in order to get rid of the backlog. This appeared to reduce the problem, but did not completely solve it. At around 12:49 our engineer tried recycling the web server process again on two of the web servers, after which normal operations were restored.

We are currently investigating ways of protecting our servers from incidents like this, and of improving our resilience in general, and we sincerely apologise for any inconvenience caused.

Posted about 2 months ago. Mar 27, 2019 - 10:18 UTC

Resolved
AU diaries appear to be working correctly again. We will continue to monitor the situation, and will provide a full post-mortem once we know more.
Posted about 2 months ago. Mar 25, 2019 - 03:22 UTC
Identified
Engineers have identified an issue causing a large build up of queued requests on the Australian server. Long running requests that were never going to complete have been stopped, which has allowed normal functionality to be restored.
Posted about 2 months ago. Mar 25, 2019 - 02:59 UTC
Investigating
We are currently investigating issues reported affecting diaries hosted in our au.resdiary.com environment.
Posted about 2 months ago. Mar 25, 2019 - 02:36 UTC
This incident affected: ResDiary Application (Australia).