ASSE outage due to partial deployment of 12.5
Incident Report for ResDiary
Postmortem

At 16:35 UTC on 06/03/2019, we attempted to deploy version 12.5 of the diary application to our Southeast Asia server. Unfortunately the deployment process failed before all of the steps in the process had completed, meaning that the web server was running the latest version, but its database had not been updated correctly.

At the time, engineers noticed that the deployment process had failed and manually updated the web application to 12.5, without noticing that the database updates had not been run.

At 00:11 UTC one of our on-call engineers was notified that users on the Southeast Asia server were having issues accessing their diaries or performing certain operations against bookings. He investigated and realised that the deployment had failed part of the way through, without the database updated being run. At this point he rolled back to the previous version of the application, restoring functionality.

We sincerely apologise for any inconvenience caused. We are examining our processes to try to prevent something like this from happening in future.

Posted 2 months ago. Mar 07, 2019 - 12:21 UTC

Resolved
The 12.5 release to asse.resdiary.com (Southeast Asia) failed part of the way through. Unfortunately we did not realise at the time that certain components hadn't been successfully updated. This in turn caused users issues accessing their diaries, or performing certain actions on their diaries.
Posted 2 months ago. Mar 07, 2019 - 12:14 UTC
This incident affected: ResDiary Application (S.E. ASIA).