An underlying database table hit a primary key limit that had not properly been migrated to 8 bytes. Today, we hit the limit on the maximum primary key for that table.
This resulted in inboxes using Custom Fields feature see intermittent failures with various features that tried to update the value of custom fields on their contacts. This included creating contacts via API, uploading contacts to broadcast audience lists, editing contacts in the app, and handling answers to survey questions if the surveys stored data on custom fields.
Inboxes that did not use the Custom Fields feature were not impacted.
Sending messages with $custom_field embeded in the message were not impacted.
Contact uploads for that did not include custom field column names were not impacted even if they had custom fields, and should have finished accordingly.
We have fully migrated the impacted table to use the correct primary key size limit and existing uploads continued as usual and reindex the results. We were able to do this without much disruption, though load times for inboxes with many contacts with custom fields may have been slower than usual during the day.
Contact uploads that failed were retried after our database migration completed, so if you had a broadcast audience upload that failed, please double check your audience, as it should have all the contacts (and their custom fields) accounted for.
Impacted surveys will have properly continued down their execution path.
All our new tables for the past few years already use 8 bytes to represent primary keys where appropriate. However, we plan on migrating all existing integer primary keys to use 8 bytes to avoid this issue in the future for any other tables.
Thanks for your patience and trusting Avochato with your contact data,
Christopher