This issue should now be resolved and all services are restored to normal functionality.

Please don’t hesitate to reach out to team@intercom.io if you have any further questions or if you’re still seeing any unexpected behaviour.

We've seen a reoccurance of these errors between 1330 and 1339 UTC+1 - error rates have returned to normal, but we continue to monitor.

This issue is resolved and all services are working as normal.

Please don’t hesitate to reach out to team@intercom.io if you have any further questions or if you’re still seeing any unexpected behaviour

Intercom's REST API in the USA hosting region went down due to a bad deployment referencing newly built caching infrastructure that was not fully accessible to all fleets.

The web fleets serving the messenger, REST APIs and inbox in Europe also went down due to the same underlying cause, and processing capacity was exhausted as a result, causing a full outage of the endpoints there.

The Australia region endpoints had the same problem, but the problem only caused higher latency than usual.

The deployments were automatically rolled back.

The downtime on the EU endpoints was between 10:46-10:59 UTC

The downtime on the USA REST API endpoint was between 10:46-10:59 UTC, with some residual errors until 11:03 UTC