During routine monitoring, Squiz identified operational issues with Funnelback services in the US, leading to search function disruptions and latency for several customers.
A subset of US Customers may have experienced delays in search results and encountered 500 errors when attempting to utilise the Funnelback search function.
Squiz engineers were alerted to errors and timeouts originating from our Squiz hosted Funnelback services in the US. This was isolated to the search sessions feature, which was subject to slow response times or termination due a build up of stored requests.
In response, we disabled the session storing feature and alleviated the database usage. This will not cause any disruption to search traffic.
As part of our standard process we initiated a period of heightened monitoring leading to resolution on May 13th at 17:20 BST
We have added new monitoring checks to flag excess database usage as well as utility scripts to help us debug slow performance in the future. Our Product team is investigating approaches to improve session performance in order to improve overall query performance going forward.