UK search performance issues

Summary
A spike in crawling traffic originating from Facebook (Meta) targeted search pages, caused excessive load on the DXP search infrastructure. Initial mitigation involved blocking the crawler at the individual account level. Upon identifying that the traffic was dynamically shifting across multiple accounts over time, a global rule was implemented to block the traffic across all search instances, successfully stabilizing the platform.

Customer Impact
A subset of UK-based clients experienced intermittent intermittent timeout / HTTP 500 errors when attempting to access their infrastructure during the period of elevated load.

Root Cause
The issue was caused by elevated crawling activity originating from Facebook infrastructure that targeted specific search pages, leading to temporary resource constraints on the DXP search infrastructure.

Resolution Actions

• Proactive Detection: Internal monitoring systems successfully flagged the traffic spike and elevated resource usage
• Targeted Mitigation: Implemented localized blocking rules on the specific client accounts initially affected by the crawling activity.
• Global Rule Implementation: Deployed a global rule across all search instances to comprehensively block the crawler traffic, fully restoring and stabilizing platform performance.
• Long-Term Prevention: Squiz is currently evaluating origin-agnostic bot mitigation strategies to comprehensively protect search infrastructure from aggressive or automated crawling activity, regardless of its source.