During routine development cycle, database provisioning for a new service within our cloud services provider inadvertently applied an authorization change to our production database cluster. This caused requests to the Avochato app and homepage to fail temporarily until the changes were reverted.
We immediately reverted the change when operations identified that it impacted production and the change rolled back out.
While Avochato was unavailable, queued automations and uploads entered our retry queues and successfully continued once we recovered. Incoming messages received during the period were not lost.
Moving forward, our team has adjusted our processes for provisioning services regardless of their proximity to production services.