Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Aug 18, 2020 - 22:00 AEST
Scheduled
NOTE: This maintenance only impacts Australian Squiz Cloud clients
Our software vendor's engineering team has identified a rare but dangerous bug which can cause storage corruption when ALL of the following conditions happen simultaneously:
- A storage server goes offline - Extremely high levels levels of disk IO are in progress - There is a very high disk-to-server ratio in the storage array - Data chunk CRC checksum validation is in progress - Data integrity algorithms detect the potential of bad data
None of the above conditions are a problem on their own, however if all of the above happen at once there is the potential for your server's data to be corrupted. This has not happened yet and we do not anticipate it happening in the near future, however we need to apply this software update to prevent any possibility of the bug being triggered.
As an extra precaution, we will be shutting down systems throughout the work in order to be 100% sure that the above conditions do not trigger while performing the update.
A full upgrade plan has been put together in conjunction with our storage vendor, and we will have expert engineering staff on hand throughout the work.
Predicted Impact:
- For clients with Squiz Edge: For up to 3 hours during the maintenance window, your Squiz Cloud services will remain online, however site edits and form submissions will be unavailable.
- For clients without Squiz Edge: For up to 3 hours during the maintenance window, your Squiz Cloud services will be offline.
Squiz appreciates that this is a long maintenance period. The reason for the extended downtime is that due to the critical nature of the storage system Squiz will be shutting down all Virtual Machines before performing the upgrade. Keeping systems offline during the upgrade will ensure that even in the event that something goes wrong there will be no risk of data loss or corruption.
Posted Jul 22, 2020 - 09:15 AEST
This scheduled maintenance affected: Squiz Cloud Hosted Instances.