Login Service, Widget, and API Outage
Incident Report for ResDiary
Postmortem

All dates and times in this report are in BST.

On Sunday 06/09/2020 for around 40 minutes between 19:00 and 20:00, and for a short period between 20:00 and 20:15, login.resdiary.com received a much larger volume of requests than normal, resulting in degraded performance. As a result of this, users may have been unable to login to diaries during this period, and some of our services including api.resdiary.com also experienced outages.

The root cause of this was a malfunctioning integration with a restaurant that caused an extreme amount of bookings to be created in a short period of time. This in turn created an extreme workload for another one of our applications responsible for delivering webhook notifications to users, which had a knock-on effect of flooding login.resdiary.com with a large volume of requests.

The circumstances leading to this outage were highly unusual and highlighted several improvements which we can make to reduce the impact caused by traffic like this. We'd like to apologise for any inconvenience caused by this incident, and thank you for your patience.

Posted Sep 07, 2020 - 13:07 UTC

Resolved
The Login service and applications depending on it, such as the API and Widgets, were unavailable for a period between 19:00 and 20:00 (BST).
Posted Sep 06, 2020 - 18:00 UTC