Major Incident - Funnelback SaaS (UK only)
Incident Report for Squiz
Postmortem

Summary

Squiz identified that a query processor node was not operating optimally, which may have impacted some of our Funnelback clients in the UK.

Customer impact

Customers being served from the impacted nodes may have seen slow search results and 500 errors when trying access search.

Issue and Resolution

Squiz engineers were alerted to an increase in 500 errors and time outs being served from one of Squiz Funnelback search nodes. Upon investigation it was found that a process responsible for serving search result was running out of available memory. The available memory was increased and the issue resolved on the 2nd February 12:25 GMT.

Mitigation

Squiz support staff have taken a review of our UK Funnelback systems to prevent further issues including increasing memory.

Posted Feb 05, 2024 - 23:48 AEDT

Resolved
This issue has now been resolved. We are no longer seeing a service degradation since the fix has been applied.
Posted Feb 02, 2024 - 23:42 AEDT
Monitoring
A fix for this has now been deployed. We will continue to monitor for further issues.

Further updates will be provided via https://status.squiz.cloud
Posted Feb 02, 2024 - 23:21 AEDT
Update
Our engineers are now working on deploying a fix for this.

A further update will be provided via https://status.squiz.cloud when this has been deployed.
Posted Feb 02, 2024 - 23:06 AEDT
Update
Our engineers are still working on resolving this issue.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
Posted Feb 02, 2024 - 22:45 AEDT
Update
Our engineers are still working on resolving this issue.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
Posted Feb 02, 2024 - 22:24 AEDT
Identified
The issue has been identified and we are currently working on a resolution. Further updates to follow.
Posted Feb 02, 2024 - 22:01 AEDT
Investigating
Squiz monitoring has detected a degradation of service impacted some Funnelback customers

Some customer may experience slow response and or degraded service.

We are working to mitigate this problem. More updates to follow shortly.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
Posted Feb 02, 2024 - 21:52 AEDT
This incident affected: Squiz Funnelback Hosted Instances.