Due to continued growth of activity on the platform, a fast-growing and critical database table exceeded the 32 bit memory requirement for inserting new primary keys, preventing additional tags from being created. The inability to create new rows in this table impacted a number of features including adding contacts to broadcasts, adding tags to contacts, and more. Until the column type could be upgraded from Integer to BigInteger, those features were impacted and other business logic within the Avochato application was subsequently impacted. Unfortunately, the upgrade process took significant time due to the size of the table itself, leading to a protracted incident window for the impacted feature set.
In order to resolve the issue, the engineering team placed the Avochato platform in maintenance mode during 4:15-5pm in order to attempt to resolve the issue, which directly impacted the ability to read conversations in the inbox during the maintenance window. A second process was applied overnight and into the following business day to fully migrate the database and resolve the issue without losing data until the new primary key column using the BigInteger type was populated, reindexed, and could be swapped for the old column.
The team performed an in-place migration to change the column type from Integer to BigInteger. This took quite a bit of time to perform on a table with over 2 billion rows, but once the migration was complete, platform functionality immediately returned to normal. Meanwhile, the design of the Avochato platform allowed failure attempts for things like tagging, broadcasting, and contact uploading to be queued for retry while the platform was impacted. Those tasks were then completed successfully in the order they were received as soon as the incident was resolved.
Moving forward, we are auditing our legacy database tables to ensure that primary or foreign keys across our data warehouse will scale decades into the future as we continue to experience growth on the platform.
We know how critical Avochato is for communicating with your teams, and appreciate your patience during this period.