At 12:10 on 1st Sept 2018, on-call emergency support engineers received reports that the UK diary application was not responding in a timely manner.
Initial investigation indicated that one of our database servers was under very high load. The engineers identified two separate database queries that were potential issues and individually forced those queries to use more efficient versions. This had the immediate effect of reducing server load back to normal levels and the application became responsive. Engineers then continued to monitor the situation.
At 14:50 engineers noticed a similar problem with one other query and again forced the database to use a more efficient version. This rectified the issue and server load returned to normal levels.
The three troublesome queries are being investigated with a view to ensuring that they don’t cause similar problems in the future.
*NOTE *We are currently in the process of migrating our UK infrastructure to a new provider which will allow for automatic detection and rectification of poorly performing queries. We tentatively expect the migration process to take 4-6 weeks.