All of the times in this report are in BST (UTC+1).
On Friday 11/10/2019 at around 10:00 credentials were updated for the component used to show what days are available for reservations on USNC Classic Widgets. We re-deployed this component in order to switch to the new credentials but missed one of the configuration variables that needed updated.
As a result, USNC Classic Widgets may not have been displaying the correct availability on their calendar, and users would have experienced very slow performance when loading widgets, and when trying to choose days on the calendar.
At 20:59 on Saturday our on-call engineers were notified that customers were experiencing problems using USNC Classic Widgets. They investigated and realised that a configuration issue had been made. At 21:19 the configuration issue was resolved, and by 21:22 the widgets were working normally again. Our engineers continued monitoring the system after the incident to make sure that the problem had been fully resolved.
We apologise for any inconvenience caused to customers by this problem. The engineers involved in making the change have been made aware of the mistake that was made to avoid a repeat.
In addition, we will be making a number of changes to our processes and alerting systems as a result of this to prevent the same or similar problems happening in future.