Issue importing contacts with custom fields.
Incident Report for Avochato
Postmortem

What Happened

An underlying database table hit a primary key limit that had not properly been migrated to 8 bytes. Today, we hit the limit on the maximum primary key for that table.

This resulted in inboxes using Custom Fields feature see intermittent failures with various features that tried to update the value of custom fields on their contacts. This included creating contacts via API, uploading contacts to broadcast audience lists, editing contacts in the app, and handling answers to survey questions if the surveys stored data on custom fields.

Inboxes that did not use the Custom Fields feature were not impacted.

Sending messages with $custom_field embeded in the message were not impacted.

Contact uploads for that did not include custom field column names were not impacted even if they had custom fields, and should have finished accordingly.

Resolution

We have fully migrated the impacted table to use the correct primary key size limit and existing uploads continued as usual and reindex the results. We were able to do this without much disruption, though load times for inboxes with many contacts with custom fields may have been slower than usual during the day.

Contact uploads that failed were retried after our database migration completed, so if you had a broadcast audience upload that failed, please double check your audience, as it should have all the contacts (and their custom fields) accounted for.

Impacted surveys will have properly continued down their execution path.

Next Steps

All our new tables for the past few years already use 8 bytes to represent primary keys where appropriate. However, we plan on migrating all existing integer primary keys to use 8 bytes to avoid this issue in the future for any other tables.

Thanks for your patience and trusting Avochato with your contact data,
Christopher

Posted Oct 24, 2024 - 16:21 PDT

Resolved
This incident has been resolved.
Posted Oct 24, 2024 - 15:19 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 24, 2024 - 14:09 PDT
Identified
We are working on an issue with importing, saving, and creating new contacts in inboxes that use custom fields, as well as creating new fields in your inbox.
As a temporary workaround, removing custom field columns from CSV uploads and/or API requests will resolve the issue.
App load times may be slower than usual while we resolve the underlying issue.
Posted Oct 24, 2024 - 04:45 PDT
This incident affected: avochato.com, API, and Mobile.