Processing emails with Postfix and Rails

Posted on 10 Aug 2011

This is a short manual on how to setup postfix and rails application to receive and process email messages.

Stack:

  • Debian / Ubuntu Server
  • Postix
  • Ruby 1.9
  • Rails 3.0

Overview

You have an application where users get email notifications. And you want to allow them to reply directly to the email. In order to do so, each email should have an unique (depends on situation) reply-to address. Usually its something like that:

P946d272cf7da4dd6b0cb613605bced65@yourdomain.com

This means that the mailserver you use should support dynamic/virtual email addresses and forwarding.

Postfix Configuration

First, you'll need to install postfix (in not installed):

apt-get install postfix

Configuration should look like this:

# See /usr/share/postfix/main.cf.dist for a commented, more complete version

default_privs = apps

# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = ap

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=no
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

myhostname = YOUR_APP_HOSTNAME
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydomain = YOUR_APP_DOMAIN
mydestination = YOUR_APP_DOMAIN, localhost
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
relay_domain = localhost
recipient_canonical_maps = regexp:/etc/postfix/recipient_canonical

Last option recipient_canonical_maps allows you to define a dynamic email addresses and forward them to the specific system mailbox for processing.

Create a file /etc/postfix/recipient_canonical:

/^P[0-9abcdef]{1,}(M[0-9]{1,})?/ apps

This will add a virtual recipient addresses and forward messages to user apps.

NOTE: Regular expressions should be in POSIX format. For test you can use regextester.com

Email Aliases

After you added support for virtual addresses all mail will be delivered to the system user mailbox (apps). But, we need to drive all that traffic into our app. In order to do so we will have to setup mail piping directly into your application script.

Edit /etc/aliases:

apps: "| /home/apps/APP_NAME/current/script/email_receiver_script"

And rebuild the aliases db by running:

newaliases

Do not forget to restart postfix:

/etc/init.d/postfix restart

You can test out the email delivery. For errors check /var/log/mail.info

Mail Receiver Script

Since all mail will be forwarded directly to our mail receiver script via piping there are few things to consider:

  • Email receiver should consume as less memory as possible.
  • Email receiver should not load the whole application (because of item above).
  • Email receiver should only validate and preprocess incoming messages and leave actual processing to another subsystem via queue.

Configuration

There are few ruby libraries that are well suited for this case:

  • mail - Email processing, ruby 1.9.2 compatible (comparing to tmail which is not)
  • redis - Simple key-value in-memory database.
  • resque - Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them.

Install gems:

gem install mail redis resque

Here is an example email receiver script:

#!/usr/bin/env ruby

require 'rubygems'
require 'mail'
require 'redis'
require 'resque'

class EmailReply
  @queue = :email_replies
  
  def initialize(content)
    mail    = Mail.read_from_string(content)
    from    = mail.from.first
    to      = mail.to.first
    
    if mail.multipart?
      part = mail.parts.select { |p| p.content_type =~ /text\/plain/ }.first rescue nil
      unless part.nil?
        message = part.body.decoded
      end
    else
      message = part.body.decoded
    end
    
    unless message.nil?
      Resque.enqueue(EmailReply, from, to, message)
    end
  end
end

EmailReply.new($stdin.read)

This script receives the mail message then tries to extract the plaintext body. If the email message is valid it adds it to the queue for future processing.

Mail Queue processing

After we put emails into the queue we'll need to create a worker.

If you need to extract a reply from the body, use mail_extract:

gem install mail_extract

Simple worker (resque job worker), extracted from one of the projects. (RAILS_ROOT/lib/email_reply.rb):

class InvalidReplyUUID    < StandardError ; end
class InvalidReplyUser    < StandardError ; end
class InvalidReplyProject < StandardError ; end
class InvalidReplyMessage < StandardError ; end

class EmailReply
  @queue = :email_replies
  
  def self.parse_email_uuid(str)
    if str =~ /^P[0-9abcdef]+(M[\d]+)?@/i
      parts = str.scan(/^P([0-9abcdef]+)(M([\d]+))?/).flatten
      project_uuid = parts.first
      message_id = parts.size == 3 ? parts.last : nil

      result = {:project_uuid => project_uuid}
      result[:message_id] = message_id unless message_id.nil?
      result
    else
      raise InvalidReplyUUID, "Invalid UUID: #{str}"
    end
  end
  
  def self.perform(from, to, body)
    user = User.find_by_email(from)
    if user.nil?
      raise InvalidReplyUser, "User with email = #{from} is not a member of the app."
    end
    
    info = parse_email_uuid(to)
    
    project = Project.find_by_uuid(info[:project_uuid])
    if project.nil?
      raise InvalidReplyProject, "Project with UUID = #{info[:project_uuid]} was not found."
    end
    
    if info.key?(:message_id)
      message = project.messages.find_by_id(info[:message_id])
      if message.nil?
        raise InvalidReplyMessage, "Message with ID = #{info[:message_id]} was not found on project '#{project.name}'"
      end
    end
    
    params = {
      :project  => project,
      :body     => MailExtract.new(body).body,
      :markup   => 'plain',
      :sent_via => 'email'
    }
    params[:message] = message unless message.nil?
    
    message = user.messages.new(params)
    unless message.save
      raise RuntimeError, "Unable to save message. Errors: #{message.errors.inspect}"
    end
  end
end

NOTE: Its important that both mail receiver and worker are using the same queue.

Create a resque.rake in RAILS_ROOT/lib/tasks:

require 'resque/tasks'
task "resque:setup" => :environment

And fire it up:

rake resque:work QUEUE=email_replies