Platform Latency

Incident Report for Avochato

Postmortem

What Happened

At approximately 9AM PST, the Avochato cloud platform began receiving double the average volume of requests for that time period. This appeared to be due to high platform utilization at the top of the hour. Our system scaled to handle this load and no requests to our servers were dropped, however the response was delayed as our system queued the requests and handled them in order. This began to slowly impact the end-user experience, with latency peaking at roughly 9:30am PST, and subsequently began to decrease.

Messages that needed to be delivered to customers were queued and delivered during this time. However, accessing conversations suffered from 5-7 second delays. The root cause appeared to be a bottleneck in our cloud infrastructure preventing our servers from handling the sheer load, causing requests to be queued.

Latency persisted for some users until approximately 10:05am PST and is now has regressed to the mean.

Resolution

While our systems automatically adjusted to handle the load, the time to do so left users with a suboptimal experience for navigating Avochato inboxes. The Avochato operations team investigated the source of the volume and took action to limit the volume of requests that were being caused by outsized usage.

What will be done

Tonight we will be performing maintenance to address this bottleneck and continue increasing the threshold of concurrent volume that can impact the user experience.

We know how important it is to quickly respond to your customers and appreciate the trust you place in our team to deliver the best possible experience for those important to you.

Thanks again for choosing Avochato,

Christopher

Posted Aug 06, 2020 - 11:12 PDT

Resolved

This incident has been resolved -- we will be performing maintenance tonight to introduce preventative measures.
Posted Aug 06, 2020 - 11:00 PDT

Monitoring

Latency due to unprecedented platform usage has been identified, and our team is monitoring platform volume.
Posted Aug 06, 2020 - 10:04 PDT

Investigating

We are currently investigating this issue.
Posted Aug 06, 2020 - 09:37 PDT
This incident affected: avochato.com and Mobile.