iContact has a development team (we call ourselves “Q-Branch” as an homage to James Bond’s Research and Development division) that spends a lot of time behind the scenes, building parts of our application you rely on, yet rarely see. Recently, the Q-Branch significantly improved how iContact processes opens and link clicks for each email.

During iContact’s growth the past few years, we’ve been monitoring how efficiently we handle opens and clicks. A few months back, we noticed we were running out of runway – we would need to make changes to how we process opens and clicks, or risk potential performance issues. Performance issues would be unacceptable, so we set out to rewrite this part of our application.

On April 10th, 2010 (at 3:30AM, even), we released our changes with no fanfare, which was exactly what we were aiming to do. Still, we’re proud of the work we did, and wanted to share a little bit of what’s going on behind the scenes.

Then and Now: Our Changes

First, let’s spend a moment looking at how we had been handling clicks and opens, and what could have happened if we didn’t come up with a new solution. Figure 1, which represents the old process, shows a user trying to login to iContact but being unable to, because the database was handling lots of simultaneous opening of messages. Each time a contact opened a message, this caused the database to use one of a limited number of “connections,” and someone logging in would also need to use a connection to the database. Without an available connection, the iContact user would be temporarily unable to access their account.

Figure 1 – If we made no changes, at some point our users may have had a hard time accessing their account.

Sometimes (in the middle of the afternoon, for instance), lots of contacts may be opening messages. At other times (when it’s night in the United States), far fewer contacts are opening messages. Most of the time, we would have had very little chance of experiencing performance issues; however, as our customer base grew, we were getting closer to having intermittent issues. This is what we wanted to avoid.

Our goal was to design and develop (and test, test, test) a way to make it so our customers would never have performance issues caused by our system processing lots of opens and clicks. The solution was to insert a new component in front of the database.

Figure 2 shows the new component, called the “Message Queue”, that intercepts each open or click, and then inserts them into the database. When we’re at peak load, the Message Queue will “hold on to” opens and clicks to keep from overwhelming our databases. They would then get processed as soon as the database is ready.

Figure 2 – The solution adds a new component in front of the database.

The Impact to You

When we started this project, we knew we’d need to make sure these changes would have little to no effect on you and your use of iContact. Granted, during times of high open and click load, it is possible that you won’t see an open or click register immediately – but our testing shows that, when there is a delay, it is usually no more than a couple of seconds.

In Conclusion

Q-Branch (and all of iContact) is really excited about the changes we’ve made to how we process opens and clicks. As always, we’re dedicated to ensuring that you always have access to iContact, so that you can continue to create, send and track the email your business relies on.