Inboxes were receiving stale data that was sometimes 10-15 minutes out of date.
This meant that even when an inbox was updated to reflect an incoming call or new message, it would rapidly be replaced by stale data, which made it really hard to find new or active conversations. Messages and calls were being received in real-time, but the Avochato inbox was not being properly updated to reflect that, and if you missed a notification or refreshed your inbox, you would not always get the latest list of active conversations (though you could still view those conversations and respond to them if you still had a link to them from a notification).
This was exacerbated by ever-increasing daily platform usage such as syncing contacts and new conversations, which slowed our services in inboxes. It was particularly challenging for inboxes with many open conversations or conversations with very long message histories. This year, we analyzed and foresaw these growth challenges specifically with indexing, and have been making architectural changes to avoid these types of slowdowns. However, our growth rate outpaced our estimates of shipping the new version of our search indexing.
During this time, no data was lost, though it was in some cases difficult to respond to incoming calls as well as handle live chats, and certain automation like closing conversations or marking conversations as addressed would appear to the end user as incorrect status in the app.
First, the engineering team implemented a patch to the adapter handling indexing platform-wide, but that did not make a significant enough impact to eliminate the symptoms of this incident. After analyzing the results, we decided to edit a specific index to exclude certain expensive queries, namely full-text searching of incoming messages for all new records. Historical messages in conversations will still be searchable for the time being.
We are actively rearchitecting how we index messages within conversations (among other things) so that they are easily fully searchable from the inbox and our API, as well as increasing the speed and accuracy of fetching conversations in an inbox on mobile and desktop devices.
We thank you for your patience and for choosing Avochato as your business communications platform of choice,
Christopher Neale
CTO & CISO