shahine.com/omar/

Catchy name for a feature huh? Boomerang is one of the cooler and innovate things we've done at Hotmail, and chances are you have no idea what it is or that we've even using it.

I'm going to be upfront and tell you that I had nothing to do with Boomerang. However, I'm writing about it because it restores dependability to email. It's specifically aimed at keeping the spammers from ruining mail.

Background

Let’s go back to when I started using email. That was around 1994, my first year in college. Back then you could send and reply to email as much as you wanted and it generally got delivered. The only thing that prevented email getting from user A to user B was some kind of routing/smtp failure or bug in the mail client (like a crash). There weren't any spammers out there to muck this up. Furthermore, if you made a typo or something when writing some one's email address you typically got a Non Delivery Receipt (NDR) notifying you of the failure. Generally, things were good.

Today Hotmail blocks what we estimate to be about 3.2 billion messages. I say estimate because much of our anti-spam technology will simply prevent us from picking up the phone when you try and call. In other words, we drop you at the router, before we even spend any cycles figuring out if you are sending spam using software such as SmartScreen. We've invested a tremendous amount of time and energy into systems that try and figure out the behavior of your IP address, and the content of your messages. These systems are by no means fool-proof and unfortunately, it’s possible that you have experienced some false positives (messages that are not spam in your Junk folder), or, more likely, false negatives, which are messages in your Inbox that are spam. What's worse though are the messages you never even received. To give you an example, I didn't find out my old roommate was getting married till pretty late because his "save the date" email never made it to my Inbox. That sucks!

The Solution

Well it's impossible to really fix things perfectly. But there is one critical scenario that we did solve, and we did it using Boomerang. If you send mail to some one using your Hotmail account, replies to that message will go straight to your Inbox. In other words, messages you send your friends and family, will have a straight shot back to you if they chose to reply. There is also another additional benefit. Over time spammers started to do things like spoofing your email address or send spam to other people that looks like it came from you (“spoofing” you).  Some of that doesn’t reach its victim and gets failed back to you as an NDR, diluting the value of NDRs. The problem with spoofing is that anyone can pretend to be you and send mail using your email address. The result is that many mail providers will send NDRs back to you (not the spammer) telling you that you tried to send mail to a non-existent mailbox. Annoying. Well Boomerang allows us to nuke any NDR messages that were not a result of YOU sending mail. How did we do all this? With smart people and simple technology. It's my favorite kind of work. No standards bodies, no politics, just an idea and some code.

Boomerang takes advantage of two very simple things: 1) The Message-ID header, 2) A hash, and 3) the In-Reply-To header. Basically, when you send a mail at Hotmail, we create a Message-ID that is unique to you. The hash is a one way hash, and it's pretty much only possible for our servers to generate them. I say pretty much because with any simple technology, it's entirely possible that some motivated person will find a way around this. However, it's really not worth their time to do so given that each instance of these things are time limited, and unique to an individual.

When another mail client receives the message from Hotmail, it contains this special Message-ID. Most mail clients that respect our Message-ID will use that Message-ID as the basis for the In-Reply-To header. This behavior is specified in RFC 822—the basis of the format of internet email.  The In-Reply-To is another message header that allows mail clients to uniquely tie a reply to another email (the one you sent). Each subsequent response to that thread maintains the Message-ID in the form of the References header. Whenever we receive a piece of mail that has this special Boomerang identifier in either of those two headers, we place it in your inbox ensuring that it bypasses any content filtering. Now some mail clients do not respect the Message-ID that we generate, or they may not generate a In-Reply-To header. Clients like Hotmail, Outlook, and OE do, and I suspect the list will continue to grow.

On NDRs. The format is such that the original message is almost always attached (to facilitate resending).  In this case, we can know that the proper Boomerang identifier is present or not, and not only bypass filtering if it is, but automatically junk it if not.  This is a very important scenario for users who can get confused by NDRs of messages they probably didn’t send from someone else that received a virus claiming to be from them, detected, cleaned, and refused the message, causing someone two degrees away from the actual infected computer to get worried, call helpdesks, reformat their machine, etc.

So, to summarize. We have some code that tags your outbound messages. If the email recipients reply to your message, and they are using a modern mail client, that message will find its way back to you; just like a boomerang. If the message reaches an address that does not exist you will receive notification of that Non Delivery, and you will not get spam messages disguised as NDRs, or NDRs that are a result of some one hijacking your email address.

We've had this feature in Hotmail for a few months now and it's working great. You probably never noticed, but you're probably getting all the replies to emails you sent when people chose to reply. Making email more reliable, it's just one things we're bringing you this year.

You can thank Eliot Gillum, Aditya Bansod and Pablo Stern for this work. They are smart guys, and I'm glad I get to work with them.

Disclaimer: this work is Patent Pending.