On the 4th of August 2022 at 07:41 UTC, Squiz internal monitoring detected a degradation of services for customers hosted in the Sacramento Date Centre.
The Squiz Data Centre team was engaged and identified an issue with a third party uplink service provider. As a workaround, all network traffic in the Sacramento Data Centre was diverted through a different border router to secondary uplink, promoting recovery at 07:58 UTC. The primary service was later restored externally by the third party uplink provider without any performed actions from our end.
Between 07:41 UTC and 07:58 UTC on the 4th of August 2022, services hosted in the Sacramento Data Centre were unreachable.
Performed investigations identified network issues with the third party uplink service provider. The Squiz Squiz Data Centre team performed a workaround and diverted all network traffic to a secondary uplink that restored the service at 07:58 UTC.
In response to this Incident, the Squiz Data Centre team has created a follow up ticket with the third party provider to identify the cause of the incident.
Additionally, we will perform a configuration change to ensure traffic is routed through the secondary uplink: ensuring network connectivity and availability of services in the Sacramento Data Centre.