Between 1545 UTC+1 and 1553 UTC+1, customers would have experienced elevated error rates and increased latencies while using Intercom. This has been resolved and all services are working as expected.

Our systems have been stable for over 24 hours and we have decided to close off this status page and will continue to monitor system health. Full root cause analysis is being investigated and will be written up and shared with our customers. We’ve provided a summary below of the incident. 

Problem: Since the 30th March, there have been intermittent errors and performance degradations across the system due to mainDB experiencing spikey outages.

Impact: Whilst many of the blips were not detectable, some customers may have experienced brief intermittent errors.

Causes: The problem has been linked to workload-related issues on our databases and underlying hardware/storage layer. Further investigation is on-going.

Steps to resolve: Measures have been implemented to mitigate the issue, including:

• Reducing binlog retention window on key databases

• Enabling changes to remove problematic transactions 

• Reduced deletions on heavy load database tables 

• Coordinated with our database partner, AWS,  on potential backend changes to be implemented 

  
  

Further steps to reduce database load are planned this week.

Maintenance has completed

Between 20:30 - 20:38 UTC Intercom was down due to a database issue. We failed over the database, and are carrying out a software upgrade later tonight. This issue is resolved and all services are restored to normal functionality.

Please don’t hesitate to reach out to team@intercom.io if you have any further questions or if you’re still seeing any unexpected behaviour

Intercom was down between 10:28 UTC - 10:34 UTC. The database has since recovered. We are continuing to work on stabilizing Intercom as our highest priority.

We are now monitoring this incident.

There was impact to production from 09:35 UTC to 09:41 UTC. All systems are operating normally.

This issue with our phone vendor is now resolved. Please don’t hesitate to reach out to team@intercom.io if you have any further questions or if you’re still seeing any unexpected behaviour

This issue should now be resolved and all services are restored to normal functionality.

Please don’t hesitate to reach out to team@intercom.io if you have any further questions or if you’re still seeing any unexpected behaviour.