Squiz DXP Funnelback degraded performance
Incident Report for Squiz
Postmortem

Summary

Squiz identified operational issues with Funnelback services in the UK, leading to search function disruptions and latency for several customers.

Customer Impact

A small subset of UK Customers may have experienced delays in search results when attempting to utilise the Funnelback search function.

Issue and Resolution

We observed a steady increase in traffic, which was able to bypass previous rulesets, by appearing to be from more valid sources. As a result, more targeted and bespoke rules and rate-limiting controls were put in place to prevent systems from being negatively impacted.

The traffic these rules targeted were of an automated nature, and no "real person" traffic was impacted by them. We have found no evidence that this was an attack of any sort.

In rare circumstances some customers were negatively impacted as a result of these actions. Squiz support teams have liaised with these customers directly to resolve these issues.

Mitigation

We have Identified additional protection measures that have now been implemented across our FB DXP Services.

  • This include enhancing caching rules
  • Enhanced rate limiting for specific automated traffic sources eg Facebook and Tencent (WeChat)
Posted Sep 19, 2024 - 01:55 AEST

Resolved
We are pleased to confirm that the previously reported issue affecting the performance of our UK Funnelback DXP system has been successfully resolved.
Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance.
We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused.

A post mortem will be made available on https://status.squiz.cloud/ in the coming days.
Posted Sep 16, 2024 - 22:36 AEST
Update
We continue to monitor the situation and will confirm resolution as soon as possible.
Posted Sep 16, 2024 - 22:23 AEST
Update
Customers are now confirming that services are recovering and performing as expected.
Our team continues to monitor the situation closely to ensure everything remains stable.
Posted Sep 16, 2024 - 21:19 AEST
Update
We are still seeing service improvements in some areas of our Funnelback search performance.
We'll continue to monitor the situation and look to confirm resolution when possible.
Posted Sep 16, 2024 - 20:41 AEST
Update
We are noting sustained improvements in Funnelback search performance.
We will maintain close monitoring of the situation and will provide confirmation of resolution as soon as it is feasible.
Posted Sep 16, 2024 - 19:52 AEST
Monitoring
We are now beginning to see service improvements in our Funnelback search performance.
We'll continue to monitor the situation and look to confirm resolution when possible.
Posted Sep 16, 2024 - 19:27 AEST
Identified
Our engineering team has identified an issue with the DXP Funnelback and is in the process of implementing a resolution.
Posted Sep 16, 2024 - 19:11 AEST
Update
We are continuing to investigate the situation and our engineers are working to find the root cause of this issue.
Posted Sep 16, 2024 - 19:01 AEST
Investigating
Squiz monitoring has detected a degradation of service impacting some Funnelback DXP customers in the UK only.

Some customers are experiencing slow response times and/or timeouts.

We are currently working to find the root cause of this issue.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
Posted Sep 16, 2024 - 18:42 AEST
This incident affected: Squiz Funnelback Hosted Instances.