Slow Performance of USNC widget
Incident Report for ResDiary
Postmortem

All of the times in this report are in BST (UTC+1).

On Friday 11/10/2019 at around 10:00 credentials were updated for the component used to show what days are available for reservations on USNC Classic Widgets. We re-deployed this component in order to switch to the new credentials but missed one of the configuration variables that needed updated.

As a result, USNC Classic Widgets may not have been displaying the correct availability on their calendar, and users would have experienced very slow performance when loading widgets, and when trying to choose days on the calendar.

At 20:59 on Saturday our on-call engineers were notified that customers were experiencing problems using USNC Classic Widgets. They investigated and realised that a configuration issue had been made. At 21:19 the configuration issue was resolved, and by 21:22 the widgets were working normally again. Our engineers continued monitoring the system after the incident to make sure that the problem had been fully resolved.

We apologise for any inconvenience caused to customers by this problem. The engineers involved in making the change have been made aware of the mistake that was made to avoid a repeat.

In addition, we will be making a number of changes to our processes and alerting systems as a result of this to prevent the same or similar problems happening in future.

Posted Oct 16, 2019 - 09:02 UTC

Resolved
USNC Classic Widgets were experiencing performance problems caused by a server misconfiguration. We have fixed the configuration issue, and the widgets should now be performing normally again.
Posted Oct 12, 2019 - 20:30 UTC
Investigating
We are currently investigating this issue.
Posted Oct 12, 2019 - 20:13 UTC
This incident affected: ResDiary Application (North Central US).