Performance Issues for Squiz UK hosted Clients

Incident Report for Squiz

Postmortem

Summary

Squiz identified performance issues with our Compute Service in the UK.

Customer impact

A subset of UK Customers may have experienced delays and timeouts when attempting to access Squiz UK hosted sites.

Issue and Resolution

The root cause of this incident was a DDoS attack. The majority of the attack was automatically blocked by our DDoS protection, but the remainder caused a performance impact to our Compute service in the UK. We performed additional manual intervention to completely mitigate the impact.

Mitigation

In light of this incident, We are investigating a number of approaches in order to reduce the effect of a similar event in the future.

Posted Dec 05, 2024 - 02:26 AEDT

Resolved

We are pleased to confirm that the previously reported issue affecting the performance of our UK hosted systems has been successfully resolved.
Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance.
We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused.

Posted Nov 25, 2024 - 22:53 AEDT

Monitoring

Following a thorough investigation from our engineering teams, we have performed mitigation and remediation actions.

These actions have been successful as we are currently experiencing no significant performance impact.

At this time we will continue to monitor the service.

Posted Nov 25, 2024 - 21:56 AEDT

Identified

We have identified this is as a result of DDoS attack which we have partially mitigated, however we are still experiencing issues at this time and our engineers are working to fully mitigate this.

Further update to be provided as soon as there is more information available.

Posted Nov 25, 2024 - 19:52 AEDT

Monitoring

We are now monitoring the performance after our engineering have been able to see some results so far.
An update shall be available once we identify no issues.

Posted Nov 25, 2024 - 18:36 AEDT

Update

Squiz are continuing to look into the potential causes of the performance degredation at hand and will have an update as soon as pertinent news has arisen.

Posted Nov 25, 2024 - 18:14 AEDT

Update

We are continuing to investigate this issue.

Posted Nov 25, 2024 - 17:42 AEDT

Investigating

Hey Everyone.
Squiz has identified issues with the performance of our UK servers and wish to let you know that our engineering team is looking into this matter with top priority and will have updates for you as soon as they are available.

Feel free to let us know by our Service Desk if you have any concerns.

Posted Nov 25, 2024 - 17:42 AEDT

This incident affected: Squiz Edge.