Email blacklisting event
Incident Report for Intercom
Postmortem

On June 10th, some customers experienced email delivery issues after Intercom was flagged by a spam monitoring organization called Spamhaus. On June 25th, Intercom was fully removed from Spamhaus’s blacklist and normal email deliverability resumed.

We want to share what caused Intercom’s platform to be added to Spamhaus’s blacklist, what we did to restore service, and what we’re doing to prevent similar issues from happening in the future.

What caused the Spamhaus blacklist event:

On May 25th, we experienced a spike in email volume across our customer base in what we believe to be GDPR-related outreach to their users. While these communications were well-intentioned, a group of our customers sent email to invalid email addresses or spam traps. Organizations like Spamhaus flag when emails are being sent to invalid addresses. Sending emails to invalid addresses is how companies can be placed on an email blacklist. In addition, a number of fraudulent spammers started using Intercom after our internal spam detection system encountered issues from an unrelated software upgrade, which again resulted in an increase of emails being sent to spam traps.

The combination of these events led Intercom to be fully blacklisted by Spamhaus.

What we did to restore service:

Once we identified the root causes of the blacklisting, we immediately took the following actions:

1) Detected and removed the fraudulent spammers who signed up for Intercom

2) Temporarily restricted email sending for a small number of customers who were sending a high volume of emails being marked as spam

3) Helped customers who were inadvertently sending emails to spam traps improve their sending practices

As a result of these and other actions, we successfully reduced the the levels of spam and spam trap hits coming from our network. On June 21, Spamhaus started to remove us from their blacklist and our email deliverability began to return to normal.

We are now fully removed from Spamhaus’s blacklist.

What we’re doing to prevent similar issues:

We know this incident disrupted our customers’ businesses and are committed to putting better measures in place to lower the risk of high impact spam blacklist issues and the effect that they have on our customers. Here’s how we’re improving our spam detection systems:

Put better safeguards in place to prevent spammers from signing up for Intercom Implement monitors, metrics and incident alarms to track email patterns that could lead to a blacklisting event

We are also improving and updating our incident management process to allow for faster high impact resolution, management and communication if Intercom is affected by spam blacklists in the future. In addition, our customer support and sales teams will have better logs to help them pinpoint which customers are affected by spam or blacklist events. This will help us reach out to customers faster if an incident like this occurs in the future.

We appreciate that you entrust us to help you communicate with your customers and it’s not something we take lightly. These actions and others we’re taking should prevent similar incidents from happening again. If you have specific questions, do not hesitate to contact our Support team at team@intercom.io or via the Messenger.

Posted about 2 months ago. Jul 23, 2018 - 09:14 UTC

Resolved
What happened:

We recently experienced a spike in email volume across our customer base in what we believe to be GDPR-related outreach to users. While these communications were well-intentioned, a group of our customers sent email to inactive users, including spam trap addresses.

This led a spam monitoring organization called Spamhaus to flag Intercom in their blacklist. As a result, over the past two weeks, approximately 16% of our outbound emails were not delivered (14% to Microsoft domains, 1% to Apple domains, and the remainder to other domains). Emails going to any Google Apps, Gmail, Yahoo and AOL addresses were delivered as expected.

What we did about it:

Complete resolution of this issue has been the highest priority for our engineering teams for the last 2 weeks. We worked quickly to remove ourselves from the Spamhaus list and bolster our capabilities to detect irregularities and ensure email deliverability.

All of our sending IPs have now been delisted by Spamhaus as a result of extensive work on improving the quality of emails sent via Intercom. All emails to users and admins should be delivering as normal. In order to achieve this we've had to put more aggressive measures in place to ensure our customers are following best practices. We do not intend to keep these long term. This may have an effect on who you are able to email from Intercom at the moment. Please reach out if you are having issues.

To prevent incidents like this happening again, we’re investing heavily in spam monitoring technology and detection resources, so we can identify irregular behaviour before it becomes an issue. This way we can help our customers maintain their reach and stay in good standing.

We are going to significantly increase the amount of engineering capacity dedicated to email deliverability, including more comprehensive checks and balances to ensure proper use of Intercom.

We appreciate that you entrust us to help you communicate with your customers and it’s not something we take lightly. These actions and others we’re taking should prevent similar incidents from happening again. If you have specific questions, do not hesitate to contact our Support team at team@intercom.io or via the Messenger.
Posted 3 months ago. Jun 25, 2018 - 17:16 UTC
This incident affected: Intercom message delivery (Small message delivery, Medium message delivery, Large message delivery, Admin notifications).