Between 9:56 ET and 10:34 ET on October 18, 2023, Engage adblock instructions, VRMBot, and the subscription management site experienced complete outages worldwide.
During the timeframe visitors trying to subscribe or login were shown an error message. Visitors trying to load adblock disable instructions were also shown an error. Finally, for sites that have VRMBot enabled, it did not load correctly. All of these issues were simultaneously corrected.
A configuration change was deployed as part of routine maintenance led to the unavailability of part of our web serving platform. Admiral separates the serving web properties which limited the blast radius to only a few services.
Once the change was identified as the cause it was corrected and re-deployed to the relevant infrastructure. The corrected deployment started at 10:24 ET and finished at 10:34 ET.
Admiral will continue to improve our deployment tooling and early-warning systems to catch these types of issues. We will continue migrating away from our legacy web serving infrastructure to improve consistency and reliability. Finally, we are reviewing our internal processes and will be adjusting them to better prevent cross-region outages going forward.