Major Incident - Sacramento Data Centre - 04/08/2022
Incident Report for Squiz

Executive Summary

On the 4th of August 2022 at 07:41 UTC, Squiz internal monitoring detected a degradation of services for customers hosted in the Sacramento Date Centre. 

The Squiz Data Centre team was engaged and identified an issue with a third party uplink service provider. As a workaround, all network traffic in the Sacramento Data Centre was diverted through a different border router to secondary uplink, promoting recovery at 07:58 UTC. The primary service was later restored externally by the third party uplink provider without any performed actions from our end.

Customer impact

Between 07:41 UTC and 07:58 UTC on the 4th of August 2022, services hosted in the Sacramento Data Centre were unreachable. 


Root cause

Performed investigations identified network issues with the third party uplink service provider. The Squiz Squiz Data Centre team performed a workaround and diverted all network traffic to a secondary uplink that restored the service at 07:58 UTC.

Mitigation and Follow-up actions

In response to this Incident, the Squiz Data Centre team has created a follow up ticket with the third party provider to identify the cause of the incident. 

Additionally, we will perform a configuration change to ensure traffic is routed through the secondary uplink: ensuring network connectivity and availability of services in the Sacramento Data Centre.

Posted Aug 05, 2022 - 17:48 AEST

Squiz support teams were engaged and switched services to a secondary uplink, this has resulted in recovery as of 07:58 UTC. Customer may have expiences a degredation of services from 07:41 UTC.
Posted Aug 04, 2022 - 18:03 AEST
Squiz monitoring has detected a degradation of service incident that is affecting customers hosted in our Sacramento Data Centre. Multiple Squiz teams are currently investigating.

A further update will be provided via in 15 minutes, or earlier if the situation or information changes.
Posted Aug 04, 2022 - 17:52 AEST
This incident affected: Squiz Cloud Hosted Instances.