Service Degradation 'All services behind CloudFlare'

Summary

On June 12, 2025, Cloudflare experienced a significant service outage that impacted a wide array of Cloudflare services. This list of services included Workers KV, which Squiz utilizes for serving frontend content and backend administration UIs.

Squiz’s Support teams were alerted to this issue following alerts received through our internal monitoring. Some customers may have experienced 503 errors throughout the incident.

‌‌Customer Impact

Customer Impact:

• Incident Duration: 12 Jun 2025, 18:10 - 13 Jun 03:01 (UTC)

• Impact: Some customers experienced 503 errors when viewing pages and administration interfaces.

  • Customer systems/site pages were inaccessible during some/all of the impact window.
  • Cloudflare marked the issue as resolved at 20:28 UTC. However, some services were not fully resolved immediately; this included Workers KV. This was resolved at 03:01 UTC

Cloudflare's status page:
https://www.cloudflarestatus.com/incidents/25r9t0vz99rp

Root Cause

The cause of this outage was due to a failure in the underlying storage infrastructure used by Cloudflare's Workers KV service, which is a critical dependency for many of Squiz's Cloudflare configuration and DXP services. Cloudflare have stated that part of this infrastructure is backed by a third-party cloud provider, which experienced an outage on June 12th and directly impacted availability of the KV service.

Resolution Actions

• Identification:

Squiz Support Teams received a number of alerts indicating service disruption, leading to quick identification of the underlying cause.

• Cloudflare Actions:

Detailed in the Cloudflare Post Mortem; https://blog.cloudflare.com/cloudflare-service-outage-june-12-2025/

• Squiz Actions:

Squiz Support worked closely with Cloudflare, taking swift manual action to clear KV worker errors and restore services while full remediation was underway.

‌‌Follow-up Actions

Cloudflare Actions:

Squiz have engaged with Cloudflare to improve resiliency of the services that depend on Workers KV and their storage infrastructure. Cloudflare are already undertaking the actions detailed below:

• (Actively in-flight): Bringing forward our work to improve the redundancy within Workers KV’s storage infrastructure, removing the dependency on any single provider. During the incident window we began work to cut over and backfill critical KV namespaces to our own infrastructure, in the event the incident continued.
• (Actively in-flight): Short-term blast radius remediations for individual products that were impacted by this incident so that each product becomes resilient to any loss of service caused by any single point of failure, including third party dependencies.
• (Actively in-flight): Implementing tooling that allows us to progressively re-enable namespaces during storage infrastructure incidents. This will allow us to ensure that key dependencies, including Access and WARP, are able to come up without risking a denial-of-service against our own infrastructure as caches are repopulated.

Squiz Actions:

• (In progress): Change the Cloudflare layer of the DXP to remove dependencies on Cloudflare Workers KV where possible, and provide safe fallbacks when the service is unavailable.