Summary
On April 21, 2026, between 09:57 UTC and 10:24 UTC, Fin was unable to reliably respond to customer conversations due to a code change. Upon reverting the code change, normal Fin operations were restored. No conversations were lost during the period of the incident and no customer data was affected.
What happened
A code change deployed to our AI backend introduced a dependency on model files into the Fin response pipeline. These model files had not yet been pre-loaded into production infrastructure resulting in errors when Fin attempted to process new conversations.
Our engineering teams identified the issue shortly after the code deployment and started implementing a fix. This fix was introduced at 10:21 UTC and Fin was fully operational once more by 10:24 UTC.
Due to some gaps in our monitoring systems, an internal incident was not created in a timely manner for this issue, leading to a reduction in speed in communication with our customers about the issue.
Next Steps
• Adding pre-deployment checks to verify required assets exist in all regions before code goes live
• Updating affected code to fall back gracefully instead of failing and producing errors entirely in the event other dependences are missing
• Identifying and resolving gaps in our monitoring for these areas of the product to ensure our incident management and communication processes are always kicked off in a more timely manner
