Between May 17th and June 17th, some emails sent through Intercom to Gmail or GSuite accounts showed a warning regarding unsafe content or malicious links, and some were routed to spam. This happened because malware, attached to inbound emails, passed our upstream vendor’s spam and anti-virus filters causing reputation issues with Google.

This post mortem is to share what caused Intercom’s domains to be flagged by Google, what we did to restore service, and what we’re doing to prevent similar issues from happening in the future.

What caused the event

Since Intercom accepts inbound email, we also encounter inbound spam, including malicious files. When a file is sent into Intercom, it is hosted on our shared domain. This domain was also used in email headers, image hosting, and open and click tracking.

When Google detect enough malicious files on a domain, they list the domain as dangerous. Any email containing that domain in the headers or content then displays a warning about malicious links, and those messages are routed to spam.

What we did to restore service

Once the issue was identified, we took the following actions to limit the blast radius and to find any potentially malicious files still being hosted on our domains:

• Added a strict whitelist of accepted filetypes to the Intercom product. We now only accept images, videos and PDFs by default.
• Created a whitelisting settings page to allow customers to only accept the file types they need, rather than accepting all file types by default.
• Increased the frequency of our antivirus rescans to proactively find dangerous attachments.
• Moved attachments off one domain and split them across multiple domains, so that a future listing by Google would only impact ⅙ of our customers

All email delivery and attachment download functionality has since returned to normal.

What we’re doing to prevent similar issues

Moving forward, we are focused on improving the way we handle attachments. We will continue to finetune our whitelist and blacklists to make it easy to access good files, while preventing access to potentially dangerous uploads. Additionally, we’re investigating improvements to our inbound spam and antivirus filtering, to prevent malicious files from getting onto our network in the first place.

With every incident, we learn more and more about how to improve our incident management process and our communication about incidents to our customers. There will always be a risk of deliverability incidents like this in the future, but we will continue to improve our infrastructure and processes so we can successfully deliver your messages and limit impact to your sending. If you have specific questions, do not hesitate to contact our Support team at team@intercom.com or via the Messenger.