Intermittent Event Data issues
Incident Report for Intercom
Resolved
After reviewing all options around backfilling Events Data during this event - from approx 2020-11-25 14:03 UTC to 2020-11-26 07:00 UTC - we unfortunately are unable to backfill this data for some affected customers. The following activities were affected during the times noted above:

- updates to the User timeline
- updating of events using the Events API

We apologise for any inconvenience this may cause.
Posted Dec 03, 2020 - 11:04 UTC
Update
We are continuing to work on backfilling Event Data during the incident timeline, and now expect this work to be completed early next week. All Intercom services remain fully operational, and we'll update here when Event Data processing is complete.
Posted Nov 27, 2020 - 16:56 UTC
Update
Following up on the earlier restoration of services, we are now scheduling the backfill of missing Event Data from yesterday's event. This data will be backfilled over the next 24 hours. We'll update here when the backfill is complete. All other services continue to work as expected.
Posted Nov 26, 2020 - 17:30 UTC
Update
Events Data has been successfully processing for all updates since approx 2020-11-26 0610 UTC and this service is now functioning as normal.

We are currently investigating the status of Event Data processing failures during the upstream provider issue, between approximately 2020-11-25 1330 UTC and 2020-11-26 0610 UTC.
Posted Nov 26, 2020 - 13:04 UTC
Monitoring
We're continuing to monitor for upstream provider issues, and are seeing recovery. We are now working on bringing Event Data services back online.
Posted Nov 26, 2020 - 10:13 UTC
Update
We have implemented a fix for delayed User Data processing, and all updates are working as expected. Event Data continues to be impacted due to an upstream provider issue, and we continue to monitor the situation.
Posted Nov 25, 2020 - 21:30 UTC
Update
Due to an upstream provider issue, we are aware of impact to both Event Data and User Data processing services.

Impacted features that use these services include REST API requests, App Store apps (i.e. Slack app), filtered people or company list views, audience previews for outbound messages, people or company profile views, conversation search by user, segment membership updates, and series, campaign, and message matching.

We are continuing to monitor the situation.
Posted Nov 25, 2020 - 20:27 UTC
Identified
Due to an upstream provider issue, Event Data processing is impacted. We are continuing to monitor the situation.
Posted Nov 25, 2020 - 18:55 UTC
Update
We are continuing to investigate this issue.
Posted Nov 25, 2020 - 18:27 UTC
Update
We are continuing to investigate this issue.
Posted Nov 25, 2020 - 18:04 UTC
Investigating
We have identified an issue where user data event reads/updates to the API and from the Intercom workspace may fail intermittently. We are working to resolve this.
Posted Nov 25, 2020 - 16:05 UTC
This incident affected: Intercom Web Application and Intercom APIs.