We were alerted to degraded performance across our services at 04:37am on June 11th 2025. Our Engineers responded to the alert immediately, with additional engineers being called in at 05:52am.
A full investigation has been completed into the cause, and steps needed to safeguard our services moving forward:
Root Cause
An Azure CDN endpoint that our system is reliant on failed, without any warning from the 3rd party.
On 11th of June 2025 the Let’s Encrypt SSL certificate for `*.azureedge.net` expired leading all secure traffic to be blocked.
The PowerShell command “Install-PackageProvider” which was used in a start-up script for the application uses the CDN endpoint "onegetcdn.azureedge.net", which was impacted by the expired SSL certificate.
This was the root cause of the incident.
Corrective actions Taken
We are removing any instances of the Install-PackageProvider command in our scripts and migrating them to AnyPackage (Install-Module) which is more actively maintained. Measures are also being taken to improve pipeline build times.
The ultimate goal to safeguard our services is to re-platform away from VMSS, which is currently being investigated and planned.
We sincerely apologise for the disruption to service experienced, and will take every precaution and measure needed to avoid such incidents moving forward.
Thanks,
ResDiary.