Executive Summary
On the 31st of May 2022 at 06:52 GMT, Squiz monitoring systems detected a degradation of service affecting customers hosted in our Sacramento and New York data centres. Investigation by the Squiz Data Centre team indicated a Distributed Denial of Service (DDoS) attack on a customer’s search website. Other customers hosted in Sacramento and New York data centres may have experienced packet loss and elevated response times resulting in intermittent degradation of service.
Once the issue was identified, the Squiz Data Centre team took remedial action to contain the attack by blocking the DDoS attack via DNS Blackholing, rerouting the incoming network traffic on the impacted website, resulting in partial stability. Recovery was achieved with changes to Firewall and Web Application Firewall rules at ~ 11:58 GMT.
Customer Impact
For the duration of the incident the targeted customers search website was disrupted and other customers in the Sacramento and New York data centres experienced increased response times and sporadic unreachability of services.
Root Cause
An application layer Distributed Denial of Service (DDoS) attack was launched on one of our customer sites causing a total disruption of service, and further impacting other customers due to being hosted in the same environments. The attack used a payload looking like normal web traffic that was not initially detected and blocked by security measures.The initial observable symptoms looked like normal internal traffic fluctuations with the initial small amount of packet loss.
Containment and Recovery
The attack against the targeted customer was contained by DNS Blackholing all traffic directed at the target, restoring normal operations to the data centres.
The targeted customer was recovered by updates to the Firewall and Web Application Firewall rules to recognise the attacker’s behaviour and block further attacks using this tactic.
Mitigation and Follow-up Actions
In response to this Incident, the Squiz Data Centre team will undertake the following actions:
If you require a PDF copy of this post incident report please contact your Squiz Service Experience Manager or Squiz Customer Care.