UK Funnelback Performance Degradation

Summary

On June 5, 2025, Squiz experienced a service degradation affecting UK Funnelback DXP customers.
A spike in automated bot traffic led to a strain on system resources, which slowed down search performance and caused occasional service issues for some customers.

A major incident was declared at 07:29 UTC.
This led to some traffic being temporarily rerouted to alternate endpoints. Mitigation efforts were put in place, including enhanced traffic filtering and bot management measures.

Service began to recover at 07:40 UTC following the introduction of targeted blocks and Cloudflare mitigation. The incident was declared resolved after a 10-minute monitoring period, with full service confirmed stable by 07:56 UTC.

‌‌Customer Impact

Customers in the UK using Funnelback DXP services experienced degraded performance, including slower search responses and intermittent 504 gateway timeout errors.

The impact was isolated to the UK region. No disruption was reported in other regions.

‌Issue, Resolution, and Mitigation

Root Cause

The incident was caused by a sharp increase in automated traffic, specifically from bot activity targeting UK customers' search services. This spike in non-human traffic led to congestion in the system, triggering rate-limiting protections which also impacted legitimate traffic.

Resolution Timeline

• 07:29 UTC – Major incident declared and incident channel initiated.
• 07:32 UTC – A large volume of automated traffic, primarily from a known search engine crawler was observed targeting UK customers’ endpoints.This surge in traffic caused excessive load on the system, triggering rate-limiting protections and impacting legitimate user traffic.
• 07:36 UTC – Mitigation initiated & Cloudflare traffic filtering enabled.
• 07:39 UTC – Click-ops rule blocks over 3,000 suspicious requests.
• 07:40 UTC – Service begins to recover.
• 07:56 UTC – Incident declared resolved following successful monitoring.

Mitigation Actions

• Bot traffic was identified and blocked using custom rules.
• Rate limits were reviewed and applied selectively.
• Cloudflare-based mitigation was enabled to provide immediate relief.
• Enhanced monitoring was put in place to verify system recovery.