Squiz identified operational issues with the query processing resources in the UK DXP, leading to queuing and slower response times on searches being processed in the UK region.
A subset of UK Customers may have experienced delays in search results and timeouts when attempting to utilise the search function.
Squiz engineers identified an increase in query volumes on the processing compute layer. Whilst clients have independent query processing capabilities, the overall compute layer powering this has a finite ceiling. The increased traffic was identified as a potential DDOS masquerading as valid search traffic. At no stage was there any breach of our systems.
Once this was identified, Squiz Cloud Engineering was able to pinpoint the pattern of traffic and our web application firewall (WAF) was reconfigured to stop the the negative affect on our compute layer. This had an immediate impact on search performance and resulted in restoration of service.
The degraded service was restored at 11:08 GMT.
In light of this incident, we are reviewing our alerting thresholds to detect and prevent these issues sooner.