At 20:54 GMT on the 1st of October 2022, Squiz internal monitoring started generating alerts indicating that multiple Squiz Edge customers hosted in our ASH and Newbury Data Centre suffered degradation of service. Our technical engineers alongside our Data Centre team were engaged. Performed investigations indicated multiple web requests coming in from various IP’s. This impacted services for some Matrix customers hosted in those data centres.
Our teams performed multiple containment actions, that included:
Above actions promoted recovery at 00:24 GMT on the 2nd of October 2022.
Between 20:54 GMT on the 1st of October 2022 and 00:24 GMT on the 2nd of October 2022, Squiz Edge customers hosted in our ASH and Newbury Data Centre’s may have been unable to access their applications.
Starting at 20:54 GMT on the 1st of October 2022, a very large number of different IP addresses from a number of different countries and Internet Service Providers started making 'cache busting' requests. It was identified that these IPs made more than 10000 requests causing a high server load, resulting in degradation of service for our customers.
The Squiz Data Centre team has confirmed that there are no further actions required from a Security perspective. Our teams are continuing to closely monitor the current situation but so far we have not seen anything requiring further action. In addition, we are working closely with our Distributed Denial of Service (DDoS) protection provider to ensure their reports are being monitored for any observed anomalies.