Emails to GSuite and Gmail email addresses being flagged as "malicious"
Incident Report for Intercom
Postmortem

Between May 17th and June 17th, some emails sent through Intercom to Gmail or GSuite accounts showed a warning regarding unsafe content or malicious links, and some were routed to spam. This happened because malware, attached to inbound emails, passed our upstream vendor’s spam and anti-virus filters causing reputation issues with Google.

This post mortem is to share what caused Intercom’s domains to be flagged by Google, what we did to restore service, and what we’re doing to prevent similar issues from happening in the future.

What caused the event

Since Intercom accepts inbound email, we also encounter inbound spam, including malicious files. When a file is sent into Intercom, it is hosted on our shared domain. This domain was also used in email headers, image hosting, and open and click tracking.

When Google detect enough malicious files on a domain, they list the domain as dangerous. Any email containing that domain in the headers or content then displays a warning about malicious links, and those messages are routed to spam.

What we did to restore service

Once the issue was identified, we took the following actions to limit the blast radius and to find any potentially malicious files still being hosted on our domains:

  • Added a strict whitelist of accepted filetypes to the Intercom product. We now only accept images, videos and PDFs by default.
  • Created a whitelisting settings page to allow customers to only accept the file types they need, rather than accepting all file types by default.
  • Increased the frequency of our antivirus rescans to proactively find dangerous attachments.
  • Moved attachments off one domain and split them across multiple domains, so that a future listing by Google would only impact ⅙ of our customers

All email delivery and attachment download functionality has since returned to normal.

What we’re doing to prevent similar issues

Moving forward, we are focused on improving the way we handle attachments. We will continue to finetune our whitelist and blacklists to make it easy to access good files, while preventing access to potentially dangerous uploads. Additionally, we’re investigating improvements to our inbound spam and antivirus filtering, to prevent malicious files from getting onto our network in the first place.

With every incident, we learn more and more about how to improve our incident management process and our communication about incidents to our customers. There will always be a risk of deliverability incidents like this in the future, but we will continue to improve our infrastructure and processes so we can successfully deliver your messages and limit impact to your sending. If you have specific questions, do not hesitate to contact our Support team at team@intercom.com or via the Messenger.

Posted Jul 24, 2019 - 14:57 UTC

Resolved
Our Engineering teams have now re-enabled the download of attachments created through emails sent to Intercom workspaces. Customers might find that links to attachments in older emails will no longer download directly, but these attachments will still be available through accessing the conversation in Intercom. This issue is now resolved and services are working as expected.
Posted Jun 10, 2019 - 12:35 UTC
Monitoring
Between approximately 2019-06-09 0000 UTC and 2019-06-10 0230 UTC, emails sent to Gmail or GSuite email addresses via Intercom would have appeared with a banner in GMail indicating that the email contained "dangerous" content. We have worked with Google to resolve the cause of this, and this banner is no longer appearing on emails sent via Intercom.

While we continue to work to resolve the root cause, we have temporarily prevented the download of attachments that were created through emails coming into Intercom workspaces which match any of the filetypes in the list below.

Attachment filetypes that are temporarily prevented include:
- doc, xslx, docm, docx, xlsm, exe
- html, htm, php, js
- zip, gz, 7z, rar, z, zls, tar
- arj, iso, ace, apk
- IMG, img, uue, cab

Workflows that rely on receiving inbound email attachments coming into Intercom will be temporarily disrupted.
Posted Jun 10, 2019 - 09:29 UTC
Investigating
We are investigating a reoccurrence of email delivery issues related to customer files sent via Intercom being flagged as malicious by Google.

Known Impacted Email Services include:
- Emails sent via Intercom to GSuite or Gmail addresses.

Email Services Not Impacted:
- Transactional emails sent from Intercom itself
- Conversation summaries from Inbox conversations
- Emails sent via Messages to non Gmail or GSuite addresses
Posted Jun 09, 2019 - 09:36 UTC
This incident affected: Intercom message delivery (Small message delivery, Medium message delivery, Large message delivery).