Write-up published
Resolved
This issue should now be resolved and all services are restored to normal functionality.
Please don’t hesitate to reach out to team@intercom.io if you have any further questions or if you’re still seeing any unexpected behaviour.
Monitoring
We've seen a reoccurance of these errors between 1330 and 1339 UTC+1 - error rates have returned to normal, but we continue to monitor.
Resolved
This issue is resolved and all services are working as normal.
Please don’t hesitate to reach out to team@intercom.io if you have any further questions or if you’re still seeing any unexpected behaviour
Monitoring
Intercom's REST API in the USA hosting region went down due to a bad deployment referencing newly built caching infrastructure that was not fully accessible to all fleets.
The web fleets serving the messenger, REST APIs and inbox in Europe also went down due to the same underlying cause, and processing capacity was exhausted as a result, causing a full outage of the endpoints there.
The Australia region endpoints had the same problem, but the problem only caused higher latency than usual.
The deployments were automatically rolled back.
The downtime on the EU endpoints was between 10:46-10:59 UTC
The downtime on the USA REST API endpoint was between 10:46-10:59 UTC, with some residual errors until 11:03 UTC
Investigating
We're seeing issues with our APIs in all regions. Our team is aware and investigating the issue. We’ll update you here as soon as we have more information.