UK DXP Funnelback Customers - Service Degradation

Postmortem

Summary

On March 26, 2025, Squiz observed a service degradation affecting several Funnelback DXP customers in the UK. The incident began at 11:57 GMT and lasted for several hours, during which customers experienced slow search responses and degraded service. Engineers identified a potential cause of the issue at 12:24 GMT, and by 13:20 GMT on March 27, signs of recovery were observed. Full service restoration was confirmed by 16:12 GMT, and the issue was resolved.

The degradation was primarily caused by unusually high traffic levels coming from a few specific actors, which overwhelmed the system and led to service disruptions. This traffic was subsequently rate-limited to mitigate the impact and restore service functionality.

Customer Impact

A subset of UK customers using Funnelback DXP services experienced slow response times and degraded search functionality. Some users also reported intermittent 504 service errors. The impact was localised to the UK, with no other regions affected.

Issue, Resolution, and Mitigation

Upon investigation, Squiz engineers identified unusually high levels of traffic coming from a small group of specific actors. This high traffic load caused congestion in the system, resulting in delays and intermittent service disruptions for affected customers.

The issue was addressed in the following steps:

1. The traffic from the specific actors was identified and rate-limited to reduce the load on the system.
2. Once the rate limiting was applied, service performance began to improve, and the degraded functionality started to recover.
3. Engineers continued to monitor the system for further anomalies and ensured that the rate-limiting measures were effectively restoring normal operations.
4. By 16:12 GMT, the system was fully stabilised, and all affected services were restored.