Intercom

Write-up
Increased Inbox Latencies in US Region

Increased Inbox Latencies in US Region - October 22nd 2025

Between 10:20 and 10:42 UTC, the Intercom application experienced significant slowdowns that affected parts of the Inbox and other areas of the product. The underlying issue was traced to a surge in the number of customer requests to one of our asynchronous workers. The jobs called a database query which was not optimized for such a large workload. Enqueueing these requests consumed most of the CPU resources for one of our databases - the same database which serves core components of the Intercom Inbox which resulted in the degraded experience.

Once the on-call team were able to identify the issue they temporarily paused the worker, which stabilized application performance. Having detected the impacted table, we moved the affected table to a different keyspace in our database adding increased levels of resilience in the system. We are confident with this change, should we have a repeat of unexpectedly heavy load to this table, it should be completely bulkheaded from impacting the Inbox.