A sudden escalation in CPU in our production database cloud database provider led to a period of longer than normal load times. While our platform was under high load for Cyber Monday, our investigation leads us to believe this was a hardware-related failure and our operations team responded accordingly.
Additionally, the period of slow response times led to our Slack application automatically turning off as per Slack’s platform policy. Engineers manually turned the app back on numerous times, but unfortunately some messages sent via Slack threads were unable to be delivered during the windows when the Slack app was disabled.
Data was not lost as we were able to fail over to a read replica database.
We have upgraded database hardware to move away from potentially degraded hardware and performed routine system maintenance.