Slow application response times
Incident Report for Avochato
Postmortem

Note This incident was unrelated to an outage relating to CDNs that may have impacted customers sending MMS images with public-facing URLs powered by Fastly (read more here).

What Happened

Avochato cloud services were temporarily unable to split traffic across our secure cloud databases, causing only one Avochato database to manage all load on the platform at once time.

This resulted in slower than average speeds when processing updates to records and when serving pages throughout the app on mobile and desktop, as well as requests to our API. This slowdown compounded during the middle of the day, causing delays in fetching and writing data as well as delays syncing data to our search nodes. It is also possible that the delays would have impacted any time-sensitive operations such as adding contacts to a broadcast before the broadcast’s scheduled date.

Resolution

The engineering team patched the issue once it was determined safe to do so. Traffic patterns returned to normal and our throughput returned to expected levels. We have also made proactive performance improvements to double the normal throughput for certain data pipelines and implemented monitoring to immediately detect this regression in traffic.

We apologize for the inconvenience to you and your team,

Christopher, CTO

Posted Jun 10, 2021 - 14:19 PDT

Resolved
This incident has been resolved.
Posted Jun 08, 2021 - 15:20 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 08, 2021 - 14:25 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted Jun 08, 2021 - 13:02 PDT
Investigating
Our team is investigating slower than average response times when using the app.
Posted Jun 08, 2021 - 11:00 PDT
This incident affected: avochato.com, API, and Mobile.