At 12:09 PM the primary database cluster for Allotrac experienced an issue that caused server number 4 to crash. The cluster automatically detected that a server had crashed in a manner that could lead to data sync issues and safely disabled access to the database. This caused affected customers to lose access to Allotrac while the issue was addressed. By 12:28 PM the Allotrac DevOps team had verified the data integrity of the remaining database cluster servers and selected servers 1 and 3 to initiate an automatic restoration. At 1:35 PM the automated recovery of the cluster for servers 0 and 2 completed and the Allotrac DevOps team began restoring access to affected customers. At 2:00 PM access was safely restored to all customers, and an automated rebuild of server 4 was initiated and is expected to be concluded at 3PM.
At this time service has been restored. Furthermore the Allotrac DevOps team has confirmed that no loss of data has occurred. The automated systems disabling the servers successfully protected all customer data. No ongoing issues from this incident are expected.
Posted Oct 28, 2024 - 14:05 AEDT
Update
Service restored We are now monitoring for any degradation in performance
Posted Oct 28, 2024 - 13:46 AEDT
Monitoring
Database Cluster Functional, Web application connections are being restored
Posted Oct 28, 2024 - 13:32 AEDT
Update
Database recovered, establishing connections
Posted Oct 28, 2024 - 13:29 AEDT
Update
Database cluster recovering
Posted Oct 28, 2024 - 12:59 AEDT
Update
We are continuing to work on a fix for this issue.
Posted Oct 28, 2024 - 12:43 AEDT
Update
We are continuing to work on a fix for this issue.