Squiz Cloud Incident - 25 March 2023
Incident Report for Squiz
Postmortem

Executive Summary

On the 22nd of March 20223, Squiz received notification of an ongoing malicious campaign against Australian government and private organisations. Squiz confirmed we have the detection capability, and mitigation strategies in place if an event should occur.

On 25th of March 2023 at ~19:49 AEDT, Squiz internal monitoring started generating alerts indicating a degradation of service / slow response times for some of our Squiz Edge customers. Additionally, some customers may also have experienced a "500 Internal Server Error” message upon accessing their sites. 

After a thorough investigation conducted by our team of experienced System Engineers, we detected abnormal traffic patterns and promptly initiated remedial actions. We immediately implemented a block on suspicious traffic in conjunction with our automated DDoS protection system from our trusted DDoS mitigation provider.

While carrying out the mitigation process, our Squiz Security team concluded that the suspicious traffic was directed only to a small number of Squiz customers.. The high volume of concurrent requests resulted in performance degradation of our Squiz Edge solution. Once we successfully implemented our mitigation measures, our systems regained their normal performance levels.

We continued to monitor the situation very closely and declared the incident as resolved at 01:23 AEDT on the 28th of March 2023.

Intelligence gathering and assessment done by our security team found that this was a targeted attack against a larger group of Australian infrastructure and facilities, and not directed at Squiz itself.  Our system and client systems remain uncompromised as the incident was restricted to a Distributed Denial of Service (DDoS) attack. No unauthorised access or data breaches have occurred, and the security of our infrastructure remains intact.

Customer Impact

During the attacks, customers may have encountered a temporary decline in service quality, such as slow response times or difficulty accessing their websites. In some cases, they might have seen a "500 Internal Server Error" message for brief periods of time.

Root Cause

A Distributed Denial of Service (DDoS) attack was launched against numerous Australian websites, including several hosted by Squiz, resulting in substantial load on Squiz's underlying infrastructure. This activity has been associated with a hacktivist group targeting multiple Australian organisations across numerous sectors.

Mitigation and Follow-up Actions

In response to the ongoing traffic anomalies, Squiz Security and Hosting teams have collaborated to implement additional security measures. 

At Squiz, we take security very seriously, and our core security capability includes DDoS protections that are routinely deployed to mitigate such attacks. In this instance, our DDoS protection measures functioned as intended, and our impacted customers were able to resume normal operations.

Our teams remain vigilant and are closely monitoring the situation to detect any future disruptions to our customers' websites.

Should you require any further information, please reach out to your Account Manager.

Posted Mar 31, 2023 - 12:39 AEDT

Resolved
We are pleased to confirm that the previously reported issue affecting the performance of our Edge system has been successfully resolved. Our team implemented corrective measures and closely monitored the situation, which led to significant improvements and the cessation of alerts. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused.
Posted Mar 26, 2023 - 00:05 AEDT
Monitoring
We are presently addressing a detected problem affecting the performance of our Edge system. Measures have been implemented, and we are noticing improvements. For now, alerts have ceased, and we are keeping a close eye on the situation.
Posted Mar 25, 2023 - 22:20 AEDT
Update
We are currently addressing an identified issue impacting our Edge system performance. Steps have been taken and we are observing improvements. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes
Posted Mar 25, 2023 - 22:03 AEDT
Identified
We are currently addressing an identified issue impacting our Edge system performance. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
Posted Mar 25, 2023 - 21:48 AEDT
Update
Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
Posted Mar 25, 2023 - 21:34 AEDT
Update
Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
Posted Mar 25, 2023 - 21:19 AEDT
Update
Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
Posted Mar 25, 2023 - 21:06 AEDT
Update
Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
Posted Mar 25, 2023 - 20:50 AEDT
Update
Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
Posted Mar 25, 2023 - 20:35 AEDT
Update
Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
Posted Mar 25, 2023 - 20:21 AEDT
Update
Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
Posted Mar 25, 2023 - 20:03 AEDT
Investigating
Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
Posted Mar 25, 2023 - 19:49 AEDT
This incident affected: Squiz Cloud Hosted Instances.