Between 9:48 ET and 14:40 ET on August 8, 2023, Admiral experienced degraded targeting capabilities leading to the unavailability of Measure, Engage, and Consent products for some visitors located in the Eastern United States of America.
During the specified timeframe, for visitors located in the Eastern USA approximately 15% of requests to record Measure data failed and approximately 33% of of requests to target Engages and target Consents. When a targeting request fails, no Engage or Consent product is shown to the visitor. Admiral purposefully disabled targeting in the affected region from 12:23 ET to 12:39 ET to help recovery efforts. The cause of this incident was identified to be a an overload of a database cluster.
The root cause of the outage was a spike in abnormal traffic patterns that led to an overloaded database cluster. Admiral has found no evidence of unauthorized access and the traffic appears to be non-malicious in nature.
Admiral operates many separate database clusters to reduce the impact of any individual cluster. Automated mitigations and configuration changes were made to return the cluster to normal operational status.
Admiral will continue to invest in our software to ensure platform stability despite all forms of traffic. We’ve identified several changes to targeting to further improve resiliency and performance. Additionally, we will be focused on improvements to further isolate Consent from many types of outages, ensuring compliance availability.