There’s a really solid, tried and true, Internet wide messaging queue. It’s been around for decades and has first class support in Ruby. What is it? Email. That’s right, email.

Before you run screaming from the room, think about it. Email works. It’s robust. It has fallback and retry strategies. It’s standardized and well understood.

In this two-parter, we’ll look at how you might use email to get data into your app. Let’s start with a story:

A million years ago, a friend on mine created a better version of the UNIX “from” command. It added a few useful features:

  • Show only unread messages.
  • Filter based on From.
  • Filter message based on To/Cc (Handy for showing only messages to me).

Handy, if not earth shattering. I used it forever, I liked to quickly see what new mail I had, without getting sucked inside a mail client. I especially liked using it to so only new mail to me. I had an alias:

1
alias         me   subj -T spike

However, subj only knows about mbox format and can only do it’s thing if you are sitting on the server where your mail is spooled. In the Modern Era, my mail, like yours, live on a distant IMAP server I rarely log into. “subj” time has passed.

But say we wanted to reproduce it’s functionality (~500 lines of C) with Ruby? What would it take?

Not much it turns out, Ruby ships with a full-featured IMAP library. It’s straight forward.

First, we need to connect to an IMAP server:

1
2
require 'net/imap'
imap = Net::IMAP.new('mail.example.com', ssl: true)

The (miss-named) ssl turns on TLS encryption. Don’t think, just use it.

Next, we need to log in:

1
2
3
4
5
begin
  imap.authenticate('PLAIN', 'spike', password)
rescue
  abort 'Authentication failed'
end

The authenticate method raise an error if it fails, hence the rescue. There are a handful of IMAP authentication methods, but typically PLAIN or LOGIN is what you want.

Once we are connected, we need to select the IMAP folder we are interested in:

1
imap.examine('INBOX')

Note that #examine opens the folder read-only, which is good for our purposes. If you are going to make changes, use #select instead.

Next we need a list of message, but first an aside about what it means to delete a message in IMAP. There are three possibilities:

  • Remove the message immediately.
  • Move it to “Trash” folder.
  • Flag it as deleted, hide it in the current folder, and show it in a virtual “Trash” folder.

The first two options are relatively expensive on the server side. And the first is user unfriendly, there’s no undo.

As a result, most implement the third option. This means that deleted messages remain in the folder until “the trash is emptied” which, under the hood, means that the IMAP server is told to permanently delete them.

Given that our INBOX almost certainly contains deleted messages, we will need to filter them out. The simplest way is to get our list via a search.

1
ids = imap.search(['NOT','DELETED'])

(Searching for ['ALL'] would get everything, including the deleted messages.)

Another option for our purposes is

1
ids = imap.search(['UNSEEN'])

which effectively means “return any message that has not been marked as read”. (Note, if you read up on IMAP, you might be tempted to try “RECENT”, which means the message has not yet been access by any client, this is not what you want.)

Now we have a list of message ids, let’s get the messages or, more precisely their headers. We ask the server for all of the “envelopes” for our list of messages.

1
message_data = imap.fetch(ids, 'ENVELOPE')

The response is an array of Net::IMAP::FetchData, which is just a wrapper around a Hash containing the envelope we ask for:

1
2
message = message_data.first
envelope = message.attr['ENVELOPE']

The envelope it self is an Net::IMAP::Envelope, a Struct that allows us to get at standard headers like To:, From:, and Subject:.

Headers that contain email address are always arrays, even when there can be only one, like From, and are another, rather unhelpful type, a Net::IMAP::Address. All other headers are simply Strings.

So, the subject is easy, it’s just envelope.subject. The from is more of a pain. Net::IMAP::Address doesn’t actually expose the full email address, we have to compose it:

1
2
3
from = envelope.from.first
name = from.name
email = "#{from.mailbox}@#{from.host}"

For the purposes of subj I’m just going to display the name or the email if there is no name:

1
name = from.name || "#{from.mailbox}@#{from.host}"

Let’s put it all together:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#!/usr/bin/env ruby

require 'net/imap'
require 'io/console'

imap = Net::IMAP.new('mail.example.com', ssl: true)

STDERR.print 'Password: '
password = STDIN.noecho(&:gets).chomp!
puts

begin
  imap.authenticate('PLAIN', 'spike', password)
rescue
  abort 'Authentication failed'
end

imap.examine('INBOX')
ids = imap.search(["UNSEEN"])

message_data = imap.fetch(ids, 'ENVELOPE').each do |message|
  envelope = message.attr['ENVELOPE']
  from = envelope.from.first
  name = from.name || "#{from.mailbox}@#{from.host}"
  puts "from: #{name.slice(0,15).ljust(15)} #{envelope.subject.slice(0,57)}"
end

Note that I’m displaying only new, UNSEEN, message. Display all messages or, better still adding a command line flag to toggle between all and new, is left as an exercise for the reader.

Next time we’ll take this new found knowledge and build a job queue.

Comments