Between 2019-01-18 1945 UTC and 2019-01-19 07:56 UTC, the following Intercom services were impacted due to a network event, and subsequent efforts to restore services:
* Message Delivery * Updates to People data, as well as Workspace Reporting & Insight data * Search Functionality across Inbox, Articles and Platform * Answerbot
All functionality has now been restored. Message delivery queues have been redriven to send any queued auto or manual messages. This issue is now resolved and Intercom is working as expected.
Posted Jan 19, 2019 - 08:21 UTC
Monitoring
Our People Data is fully restored and Message Delivery has been resumed.
We are currently catching up on the backlog for message deliveries. Our engineering teams are monitoring the backlog's process.
Posted Jan 19, 2019 - 05:35 UTC
Update
We have completed back filling Workspace Reporting & Insight data and both should now be fully restored.
People data is accurate as of 2019-01-18 at 04:00 UTC. The people data in our search index is accessible but up to 24 hours out of date and we are in the process of making this data current. We expect to be finished as of 2019-01-19 at 12:30 UTC.
Message Delivery continues to be impacted from the earlier networking event.
Our engineering teams are working on restoring all remaining functionality.
Posted Jan 19, 2019 - 02:59 UTC
Update
We have fully restored the following services for all workspaces: Reporting, Insights, Answer Bot, People List, and Help Center.
Workspace Reporting & Insight data is accurate as of 2019-01-18 at 00:00 UTC. We are in the process of backfilling this data.
People data is accurate as of 2019-01-18 at 04:00 UTC. The people data in our search index is accessible but up to 24 hours out of date and we are in the process of making this data current. We expect to be finished as of 2019-01-19 at 12:30 UTC.
Message Delivery continues to be impacted from the earlier networking event.
Our engineering teams are working on restoring all remaining functionality.
Posted Jan 19, 2019 - 00:45 UTC
Update
Message Delivery, Search functionality, (People search, Conversation search, and Help Center search), Reports, Answer Bot, and the People List continues to be impacted from the earlier networking event. Our engineering teams are working on restoring all functionality.
Posted Jan 18, 2019 - 21:47 UTC
Identified
We have identified the source of the network interruptions on our production servers, and have reverted a deploy to resolve those issues. At 20:07 UTC, Intercom services are recovering and primary services are resuming. Customers will continue to experience issues with user search functionality, message delivery, Answer Bot services, and conversation search functionality while we work to fully resolve this issue.
Posted Jan 18, 2019 - 20:33 UTC
Investigating
We are currently investigating reports of Intercom services being interrupted. Starting around 2019-01-18 1950 UTC.
Posted Jan 18, 2019 - 20:04 UTC
This incident affected: Intercom message delivery (Email, Chats and Posts, Mobile Push, Admin notifications), Intercom Messenger (Web Messenger, Mobile Messenger), and Intercom Web Application, Intercom APIs, Intercom Mobile APIs, Intercom Webhooks.