Simple e-mail service with Rails - Part 4: Receiving e-mails

Our app is now able to send e-mails with attachments. That’s a good first step, but you’d probably expect your e-mail service to be able to receive e-mails too. So let’s get that going. Luckily for us, Rails fairly recently introduced ActionMailbox API which makes it pretty easy.

TLDR: You can find the code in this GitHub repository. Every part has a corresponding commit.

As always, there’s setup required. For now, we won’t bother to configure ingress for any of the available services. We save that for later when we will be deploying our app.

bin/rails action_mailbox:install
bin/rails db:migrate

Similar to ActiveStorage, Rails use database table to store every incoming e-mail as an InboundEmail record. Attachments of these e-mails are conveniently stored using ActiveStorage. As a next step, we can generate first mailbox.

bin/rails g mailbox Inbox

This InboxMailbox will be responsible for receiving and parsing e-mails delivered to our application. E-mail routing is defined in ApplicationMailbox which we have to change slightly. By default, no e-mail is routed to our application, which makes sense, because there’s no default mailbox.

class ApplicationMailbox < ActionMailbox::Base
  routing all: :inbox
end

Now, that every e-mail will be routed to InboxMailbox, let’s write some tests and logic to parse incoming e-mails. We’ll start with basic tests for InboxMailbox

require 'rails_helper'

RSpec.describe InboxMailbox, type: :mailbox do
  it 'routes emails to correct mailbox' do
    expect(InboxMailbox).to receive_inbound_email(to: 'test@example.com')
  end

  it 'marks e-mail as delivered and creates new Message' do
    mail = Mail.new(
      from: 'from@example.com',
      to: 'test@example.com',
      subject: 'Test email',
      body: 'Test body',
    )

    mail_processed = process(mail)

    expect(mail_processed).to have_been_delivered
    expect(Message.count).to eq 1

    message = Message.last
    expect(message.from).to eq 'from@example.com'
    expect(message.to).to eq 'test@example.com'
    expect(message.subject).to eq 'Test email'
  end
end

Routing test should already pass, but the test ensuring new Message is being created fails as expected, because we haven’t written any code yet.

class InboxMailbox < ApplicationMailbox
  attr_reader :message

  def process
    @message = Message.create!(
      from: mail.from.join(', '),
      to: mail.to.join(', '),
      subject: mail.subject,
      content: process_email_content,
    )
  end

  private

  def process_email_content
    if mail.text_part.present?
      mail.text_part.body.decoded
    else
      mail.body.decoded
    end
  end
end

I’m keeping things simple for now. Most of modern e-mails are multipart e-mails. These parts can be text and HTML content of the e-mail and its attachments. For now, we just check whether received e-mail has text part and store it, or simply store its rare body if no text part is present.

While working on this app and using similar code in a real world, I discovered many pitfalls of processing and storing e-mail body. Many pages were written with e-mail specifications so I won’t go into much detail in this tutorial. But feel free to dig a little deeper!

To be able to receive a real e-mail and see how it really works, instead of just trusting our tests, you can visit this magic URL http://localhost:3000/rails/conductor/action_mailbox/inbound_emails Rails provide to work with inbound e-mails in development. There are two ways of creating InboundEmail. Manually filling the form or importing source. Importing source is really handy if you want to simulate complex multipart e-mail or try to debug real world e-mails (wonder why I know that huh). But for now, fill up the form and send yourself a simple e-mail. Note: I encountered a Rails bug when I first tried to submit the form. You might need to update Rails (bundle update rails) too!

If there are no error messages, you should see a new Message created if you open up messages list.

Attachments yet again

We are able to send e-mail with attachments, we should be able to also receive e-mail with attachments.

  it 'saves attachments' do
    mail = Mail.new(
      from: 'from@example.com',
      to: 'test@example.com',
      subject: 'Test email',
      body: 'Test body',
    )

    mail.attachments['test.csv'] = { mime_type: 'text/csv', content: 'a,b,c' }
    mail.attachments['test.png'] = { mime_type: 'image/png', content: 'Not PNG' }

    mail_processed = process(mail)

    expect(Message.count).to eq 1
    message = Message.last
    expect(message.attachments.count).to eq 2
  end

Writing a test first again. We create another e-mail object, but this time, we’re going to add attachments. For this test to pass, you have to create ActiveStorage::Blob for every attachment and include it while creating new Message record. E-mail attachment is technically its data and type. And its data can be represented as a string. ActiveStorage has a handy method create_and_upload!, which accepts IO object (string representing an attachment) and it’s type. I’ve added these two private methods to the InboxMailbox code for reading and storing attachments.

  def parsed_attachments
    @parsed_attachments ||= mail.attachments.map do |attachment|
      blob = blob_from_attachment(attachment)

      {
        attachment: attachment,
        blob: blob
      }
    end
  end

  def blob_from_attachment(attachment)
    ActiveStorage::Blob.create_and_upload!(
      io: StringIO.new(attachment.decoded),
      filename: attachment.filename,
      content_type: attachment.content_type,
    )
  end

Notice how parsed_attachments method is using instance variable of same name. Leveraging ||= conditional assignment operator, method will only run its code once and cache the result. It’s an easy way to ensure we won’t accidentally create Blobs from e-mail attachments multiple times.

Last thing to do is to actually use parsed_attachments and save attachments with new Message.

  def process
    @message = Message.create!(
      from: mail.from.join(', '),
      to: mail.to.join(', '),
      subject: mail.subject,
      attachments: parsed_attachments.map { |attachment| attachment[:blob] }, # new code
      content: process_email_content,
    )
  end

If you fill the Inbound e-mail magic form and include attachments this time, you should see a new Message with whatever attachment you included.

Inline attachments such as images, unfortunately, are a topic that’s quite complicated. I’m not yet comfortable sharing my results. This however, might be included in future parts of this tutorial, so stay tuned!

HTML e-mails

Last thing missing for having a complete e-mail experience is leveraging HTML e-mails. Which, let’s be fair, will be almost every e-mail send by any modern e-mail client.

To create a Mail object with both text and HTML part, we can look for an inspiration in Ruby Mail Gem examples.

  it 'uses HTML body' do
    mail = Mail.new(
      from: 'from@example.com',
      to: 'test@example.com',
      subject: 'Test email',
    )

    text_part = Mail::Part.new do
      body 'This is plain text'
    end

    html_part = Mail::Part.new do
      content_type 'text/html; charset=UTF-8'
      body '<h1>This is HTML</h1>'
    end

    mail.text_part = text_part
    mail.html_part = html_part

    mail_processed = process(mail)
    message = Message.last

    expect(message.content.to_plain_text).to eq 'This is HTML'
  end

HTML e-mail can include a lot of crap. Unbelivable amount of crap. While using this code in a real world application, I’m stripping away many things. Different e-mail clients use different formatting, add signature blocks differently and generally don’t bother too much with quality of generated HTML. Basic approach is to at least only use body element of the HTML content, instead of just simply saving whatever HTML we receive. Main reason being that it can include styles which could easily break formatting of our application.

To be able to extract just a body element, we have to parse the HTML code. Luckily for us, there’s this small gem called Nokogiri, which can do that for us. Nokogiri is already part of our application’s bundle as it’s dependency of many gems we already use. ActionText to name one. No installaction required here!

  def process_email_content
    if mail.html_part.present?
      document = Nokogiri::HTML(mail.html_part.body.decoded)
      document.at_css("body").inner_html.encode('utf-8')
    elsif mail.text_part.present?
      mail.text_part.body.decoded
    else
      mail.body.decoded
    end
  end

We read HTML part of the e-mail, parse it using Nokogiri, find the body DOM element and voilà, new test is passing!

As a homework, you can try and figure out how to simulate HTML Inbound e-mail and test it. Only way working for me is importing real email. Most e-mail clients allow you to download or display original in EML format, so you can experiment with that.

In next part, we’ll take a look at replies and forming Topics of related Messages. Stay tuned!