Building Advanced Email Features With IMAP and PHP

In this tutorial, I'll walk you through some real world examples of how you can use PHP and IMAP to build new features for managing your email—features that the big email providers haven't built for us.

My interest in this began in 2010 when I wrote Twelve Gmail Ideas to Revolutionize Email (Again), but mostly the ideas I wished for have remained out of reach. For as important as email is, innovation of email as an application has been quite slow.

We’re drowning in email, and managing our inboxes remains a heavy burden. Mail services and clients have done very little to help us with this. Most of the email we receive is sent by machines, not people, and yet we’re the ones who have to individually process all of it.

Analysis of my own email showed I was receiving email from more than 230 automated senders, far fewer actual people. I was tired of constructing filters in Gmail and filling in a myriad of unsubscribe forms. I wanted to have more control over managing my email and simplifying my life.

Finally, this past year, I decided to build the features I needed. The result is Simplify Email (SE), a small web app you can host yourself which offers a variety of cool new email features all of which you can check out on the project website.

The coolest thing about SE is that it's a platform for reading, analyzing, routing and managing your email—the possibilities abound. Simplify Email is essentially a programmable playground for "hacking" your own email.

I'll walk you through the code of three examples from SE that use PHP, IMAP, and MySQL to work with email:

Checking your inbox and filtering messages

Implementing a Whitelist challenge to unknown senders

Reporting unanswered email

This tutorial will definitely give you a head start at writing IMAP code in PHP. But you can also work directly with the Simplify Email codebase. You can purchase the code for as little as $10, and there is an older open source version (which lacks some of the features we're describing below). Installation guides are provided for typical Linux configurations. I also offer pre-installed images at Digital Ocean for $25 as well as a handheld valet install. SE is written in PHP, in the Yii Framework.

Note that you won't be able to access most email servers via your local development machine unless you compile a secure IMAP library for PHP. This is one of the reasons I encourage people to run Simplify Email in droplets at Digital Ocean. There are also a few tricks to getting Google account security to let you in via IMAP.

Working with IMAP

How Simplify Email Works

With SE, you can continue using your email client of choice on both the Web and your mobile devices. You don't have to change any apps or personal habits. SE accesses your email accounts behind the scenes via IMAP; acting as a smart personal assistant, SE pre-processes your email, moving messages to the appropriate places based on everything you've told it.

When a message arrives from a familiar sender, SE moves it to the folder you've specified. When one arrives from an unknown sender for the first time, it moves it to the review folder.

Every couple of hours (or at a frequency you choose), SE will send you a summary of where it moved your messages and which messages are in review. Note, links for training senders are included for the review folder, making it quite easy to train SE over time.

At any time, you can browse your review folder—you don't have to wait for the digest to arrive. But the advantage of SE is that you no longer have to browse your folders; you can just read your digest to get a view of the email that you've received and train new senders.

1. Checking Your Inbox and Filtering Messages

SE uses several cron tasks to operate in the background on your server. Each is called from DaemonController.php.

The first, processInbox, is called frequently and needs to operate quickly—its job is to screen email and move it out of the Inbox as quickly as possible and into the triage folder, called the filtering folder.

The second, processFiltering, is more process-intensive and performs deeper operations on email, ultimately moving messages to their final destination.

Here's an adapted version of publicly available IMAP header parsing code which gathers the additional information that SE needs for a variety of tasks. Basically, it uses imap_rfc822_parse_adrlist to determine recipient information, message-id, subject and timestamps (or sender information when scanning the sent folder):

Next, we determine if the user might have dragged a message from another folder back into the inbox—intending to train it via drag and drop. If so, we set the training for this sender to the inbox. In other words, next time, we'd want to just route messages from this sender to the inbox:

If not, we'll prepare to move the message to the Filtering folder for further processing. First, we may send notifications to the user's phone if there is a sender match or keyword match for notifications (and it's not quiet hours):

The primary processing loop for each message in the filtering folder is quite detailed. First we look at the recipient address, as SE allows people to train folders by recipient address, e.g. messages to the happyvegetarian.com domain go to the veggie folder:

During quiet hours, messages are primarily kept in the filtering folder.

Every couple of hours, a different process will build the message digest using the message table records to determine which emails were recently received and filtered and how they were routed.

2. Implementing a Whitelist Challenge to Unknown Senders

The goal of the whitelist challenge is to keep any message from an unknown sender, e.g. possibly a marketing bot or spammer, out of your inbox. SE places mail from unknown senders in the review folder. However, if you turn whitelisting on, we send out a challenge email that gives the sender a chance to verify that they're human. If they respond, we'll move the message to your inbox. If the email turns out to be unwanted, you can zap the message from the digest or drag it to any folder you wish to train it to.

The user can turn whitelisting on and off in settings:

To implement whitelisting, we send out email challenges whenever mail arrives from a new sender:

Furthermore, when messages are sent in reply to other messages, they have an in_reply_to field which links back to the original message_id.

So, we use a SQL query to find all the received messages that don't have a corresponding reply message referencing their message_id. For this, we use a LEFT OUTER JOIN where there is no in_reply_to id:

We use the $subject_compare mode to differentiate between our sent messages that haven't been answered and our sent replies to a thread that haven't been answered. Here's the unanswered message report in your account:

SE also offers this information as an optional digest, called the unanswered email digest. You can receive it every day, every few days or every week.

We also use similar SQL tabulation with Google Charts to provide reports of how frequently certain people email you:

Next Steps

I hope you've found Simplify Email intriguing enough to try your hand at PHP IMAP programming. There are so many cool features you can build without any need for the big email providers to do anything new.