Summary
On the 2nd April at approximately 14:49 (GMT+10) Squiz received indications of 500 Errors from customer pages.
Squiz’s Support teams alongside our product team, quickly identified that the issue was induced by a recent DXP Upgrade version 6.61.0. Hot patches were released whilst in parallel Matrix DXP was rolled back to 6.60.1 restoring services.
Customer Impact
Incident Duration: 02 Apr 2025, 14:49 - 23:14 (GMT+10)
Impact: some customers experienced site page 500 errors.
Impact times and service restoration times varied throughout the course of the incident duration.
The effect of this issue was limited to clients who changed asset statuses during a specific period of time which meant that impact was only felt by some users who would have been editing assets at the time of the incident.
Root cause Analysis
An asset property was removed in Matrix Version 6.61.0. This impacted assets that were placed into Safe Edit as it resulted in errors when Matrix attempted to serialise objects.
Resolution Actions
Squiz Support Team identified a trend in logs when investigating reports of problems. Product teams were engaged with and quickly isolated the cause.
Squiz developed, tested and deployed hot-patches, whilst in parallel assessing version Rollback vs Roll forward.
To fully resolve the issue a Matrix Version downgrade took place.
Follow-up Actions
Squiz has deployed monitoring enhancements to have the ability to detect/monitor for similar events including identification during testing - completed
Squiz has rolled out Matrix Version 6.61.1 successfully, which introduced a change to circumvent this issue. - completed