shahine.com/omar/

Catchy name for a feature huh? Boomerang is one of the cooler and innovate things we've done at Hotmail, and chances are you have no idea what it is or that we've even using it.

I'm going to be upfront and tell you that I had nothing to do with Boomerang. However, I'm writing about it because it restores dependability to email. It's specifically aimed at keeping the spammers from ruining mail.

Background

Let’s go back to when I started using email. That was around 1994, my first year in college. Back then you could send and reply to email as much as you wanted and it generally got delivered. The only thing that prevented email getting from user A to user B was some kind of routing/smtp failure or bug in the mail client (like a crash). There weren't any spammers out there to muck this up. Furthermore, if you made a typo or something when writing some one's email address you typically got a Non Delivery Receipt (NDR) notifying you of the failure. Generally, things were good.

Today Hotmail blocks what we estimate to be about 3.2 billion messages. I say estimate because much of our anti-spam technology will simply prevent us from picking up the phone when you try and call. In other words, we drop you at the router, before we even spend any cycles figuring out if you are sending spam using software such as SmartScreen. We've invested a tremendous amount of time and energy into systems that try and figure out the behavior of your IP address, and the content of your messages. These systems are by no means fool-proof and unfortunately, it’s possible that you have experienced some false positives (messages that are not spam in your Junk folder), or, more likely, false negatives, which are messages in your Inbox that are spam. What's worse though are the messages you never even received. To give you an example, I didn't find out my old roommate was getting married till pretty late because his "save the date" email never made it to my Inbox. That sucks!

The Solution

Well it's impossible to really fix things perfectly. But there is one critical scenario that we did solve, and we did it using Boomerang. If you send mail to some one using your Hotmail account, replies to that message will go straight to your Inbox. In other words, messages you send your friends and family, will have a straight shot back to you if they chose to reply. There is also another additional benefit. Over time spammers started to do things like spoofing your email address or send spam to other people that looks like it came from you (“spoofing” you).  Some of that doesn’t reach its victim and gets failed back to you as an NDR, diluting the value of NDRs. The problem with spoofing is that anyone can pretend to be you and send mail using your email address. The result is that many mail providers will send NDRs back to you (not the spammer) telling you that you tried to send mail to a non-existent mailbox. Annoying. Well Boomerang allows us to nuke any NDR messages that were not a result of YOU sending mail. How did we do all this? With smart people and simple technology. It's my favorite kind of work. No standards bodies, no politics, just an idea and some code.

Boomerang takes advantage of two very simple things: 1) The Message-ID header, 2) A hash, and 3) the In-Reply-To header. Basically, when you send a mail at Hotmail, we create a Message-ID that is unique to you. The hash is a one way hash, and it's pretty much only possible for our servers to generate them. I say pretty much because with any simple technology, it's entirely possible that some motivated person will find a way around this. However, it's really not worth their time to do so given that each instance of these things are time limited, and unique to an individual.

When another mail client receives the message from Hotmail, it contains this special Message-ID. Most mail clients that respect our Message-ID will use that Message-ID as the basis for the In-Reply-To header. This behavior is specified in RFC 822—the basis of the format of internet email.  The In-Reply-To is another message header that allows mail clients to uniquely tie a reply to another email (the one you sent). Each subsequent response to that thread maintains the Message-ID in the form of the References header. Whenever we receive a piece of mail that has this special Boomerang identifier in either of those two headers, we place it in your inbox ensuring that it bypasses any content filtering. Now some mail clients do not respect the Message-ID that we generate, or they may not generate a In-Reply-To header. Clients like Hotmail, Outlook, and OE do, and I suspect the list will continue to grow.

On NDRs. The format is such that the original message is almost always attached (to facilitate resending).  In this case, we can know that the proper Boomerang identifier is present or not, and not only bypass filtering if it is, but automatically junk it if not.  This is a very important scenario for users who can get confused by NDRs of messages they probably didn’t send from someone else that received a virus claiming to be from them, detected, cleaned, and refused the message, causing someone two degrees away from the actual infected computer to get worried, call helpdesks, reformat their machine, etc.

So, to summarize. We have some code that tags your outbound messages. If the email recipients reply to your message, and they are using a modern mail client, that message will find its way back to you; just like a boomerang. If the message reaches an address that does not exist you will receive notification of that Non Delivery, and you will not get spam messages disguised as NDRs, or NDRs that are a result of some one hijacking your email address.

We've had this feature in Hotmail for a few months now and it's working great. You probably never noticed, but you're probably getting all the replies to emails you sent when people chose to reply. Making email more reliable, it's just one things we're bringing you this year.

You can thank Eliot Gillum, Aditya Bansod and Pablo Stern for this work. They are smart guys, and I'm glad I get to work with them.

Disclaimer: this work is Patent Pending.

 

Last week my parents were visiting. And like a good son, I spent a few hours getting my dad set up on his new Sony S Series Laptop to replace his Fujitsu Tablet (my Dad was jealous by all the Apple PowerBooks his friends have and he wanted a sexy wide screen laptop that didn't weigh 50lbs). The laptop is awesome, thinking of one for myself. However... I counted > 30 items preinstalled in the Add/Remove programs. I spent most of this time just removing all this nonsense. Things have actually gotten worse over the past few years. In terms of bad offenders I'd rank:

1. Sony
2. Toshiba
3. Gateway
4. HP
5. Fujitsu
6. Dell

With a Fujitsu or a Dell you barely get anything installed, which is great. Toshiba/Sony laptops come with dozens of utilities, processes, crap I don't want or need. Who thinks this is a good idea? Do I really need a process running for every hotkey and widget? Do I really need AOL, RealPlayer, and every AntiSpyware, Virus, Trojan Horse protector, 2 Wifi thingies, and Media Home Server sharing software? 25 icons on my desktop?

To make matters worse, Sony did not include any software to restore the laptop to factory state. Nada. Good thing I can boot XP from a CD on that thing... at least they dumped the Jog Dial.

You probably don't care, and should not care, but after many months of living with this bug, we have released a fix for it.

Unlike Jonathan, I had a repro (it was called using my computer for 2 days w/o rebooting), and was basically told to pound salt. Then magically it was prioritized the "right way" and now we have a fix. In about one day my tcserver.exe process could go up to 300 MB of working set. Then an exception would occur, and the process would recycle. Not the end of the world, but pretty annoying.

I actually don't care much for the extra overhead of all the processes that come with my toshiba m200 tablet. I figure you pay at least a 256 MB memory stick in overhead. But my biggest problem is that the screens on these things are blurry, dim, and not very sexy when compared to a widescreen 4lb laptop with a DVD drive and an XBrite Screen. My next laptop is going to be a laptop. I've gone so far as to disable tabtip.exe from starting up on my machine. When I click the button near the start menu it comes up anyway.

Yes, I get to be grumpy every once in a while .

I've been reading a lot about how no one is going to the movies any more. Well I'm one of them. Going to the movies pretty much sucks these days. With the $20 ticket fee for two, the $8 soda, the screaming annoying teenagers, and the freaking ads! Why pay good money for that when I can pay $20 a month to have DVDs delivered to my door (via Netflix), and pop them into my Media Center and watch on my 42" Plasma with 5.1 Surround? At least I can press pause, and eat my own food.

"Just as home entertainment systems — including plasma screens and surround sound — have become increasingly lavish, the overall moviegoing experience has become a shell of its former self. Even as theaters offer stadium seats and martinis, moviegoers are being bombarded with countless advertisements and coming attractions.

"Going to a movie theater used to be a unique way of seeing a movie and carried with it a romantic notion — it was a special forum you shared with a group of people," said Terry Press, the head of marketing for DreamWorks. Theater advertising is annoying and ruins the value of movie previews, which are a studio's most powerful marketing tool, Press said. "At least at my house I have the ability to fast-forward through the commercials," she said.

Said Richard Zanuck, a producer of the forthcoming "Charlie and the Chocolate Factory": "I don't like [commercials] at all. People come to see the movie. The movie experience is supposed to be that…. They've come to see the film and not to be sold something else."

In fact, 73% of adults prefer watching movies at home, according to an Associated Press-AOL poll released last week. A quarter of those polled said they had not been to a theater in the last year."

[La Times]

I suspect that many folks who have purchased the latest in display technology (Plasma, DLP, Projector, etc) are finding themselves unwilling to pay ~$30 to see a movie in the theater. However, listen up Hollywood. If you want to get my butt in a theater, take notice of Batman Begins on IMAX. I happily paid $15 for the ticket, $9 for parking, and $4.00 for a small soda. But it was worth every penny. Like Rick, I loved it.

To be honest, the main reason I switched my parents from Mac to a PC many years back was the built in Remote Assistance. I can't tell you how many times this has saved my parents a plane ticket to fly me home. I probably use it twice a month. However, there are two big problems with Remote Assistance.

1) MSN Messenger needs a way for me to Offer Remote Assistance. Each time I need to help my parents I spend about 10 minutes telling them how they need to ask me for Remote Assistance. That involves launching MSN Messenger (to which they say, I'm already using MSN), finding my contact, opening it up, going to the Actions menu, Start Activity, then Ask for Remote Assistance. For the love of Pete give me a way to offer them Remote Assistance.

2) Why oh why can't Windows XP stop drawing pretty things when I'm connected. It's freaking slow as molasses as I sit there and wait for all the animations, drop shadows and pretty colors to draw. I wish it could down sample the UI like Remote Desktop can. This would save me a lot of time.

I wish some one would fix this.

I've always wondered what the difference was between:

• %USERPROFILE%\Application Data\
• %USERPROFILE%\Local Settings\Application Data\

Well Raymond gives us the answer.

Basically, %USERPROFILE%\Local Settings\Application Data\ is for local cache type data. Stuff you don't want to roam in your Roaming Profile. I used Romaing Profiles a few years ago, and after a few days it so badly screwed up my PCees, I promptly flattened my boxes and went back to the basics.

99% of the appointments on my calendar say "Updated: <original appointment name>". I don't find this particularly useful, especially when I look at my calendar on a small screen like the Treo 650, or even smaller Windows Mobile Device. Half the time I have no idea what was Updated anyway, and I just blindly accept the meeting request.

In fact, most meetings go like this:

1. You get a Meeting Request for a meeting
2. 2 min later you get another meeting request because the person:
   • Forgot to make it recurring
   • Could not get the conference room
   • Added some attendees
   • Moved it to a new time because person foo could not make it
3. Some time later, due to issues with step 2 you get another appointment.

If you are lucky enough to get to Step 3, you will often find that Outlook will display a message that says "Object moved" when you try and delete, accept or do anything with the third meeting request. I have no idea why.

Anyway, to make the calendar more useful, i wrote a little command line app that will simply remove "Updated: " and "FW: " from the beginning of any appointments on your calendar.

You can download it here, or in the RSS enclosure.

Usual disclaimer applies. This works on my machine, did not do bad, and I am not responsible for what it does on your machine. You must have the Outlook .NET Programability Installed for Outlook 2003.

The code looks like this:

static void Main()
{
    string[] prefixes = new string[] {"Updated: ", "FW: "};
    try
    {
        Application application = new Application();
        MAPIFolder calendar = application.GetNamespace("MAPI").GetDefaultFolder(OlDefaultFolders.olFolderCalendar);

        int counter = 0;

        foreach (object o in calendar.Items)
        {
            if (o is AppointmentItem)
            {
                AppointmentItem appointmentItem = (AppointmentItem)o;

                foreach (string s in prefixes)
                {
                    if (appointmentItem.Subject.ToLower().StartsWith(s.ToLower()))
                    {
                        string currentSubject = appointmentItem.Subject;
                        string cleanSubject = currentSubject.Remove(0, s.Length);
                        appointmentItem.Subject = cleanSubject;
                        appointmentItem.Save();
                
                        Console.WriteLine("subject: {0}", currentSubject);
                        Console.WriteLine(" new subject: {0}", cleanSubject);
                
                        counter++;
                    }
                }
            }
        }
    
        Console.WriteLine("Cleaned {0} of {1} Appointments", counter, calendar.Items.Count);
        Console.WriteLine("Press any key to exit.");
        Console.ReadLine();
    }
    catch (System.Exception ex)
    {
        Console.WriteLine(ex.Message);
    }
}

Getting Things Done teaches you that you should have zero messages in the inbox. I was up in Redmond this past week and was thinking about my zen like experience with mail now, and that realistically I get to zero messages one or two times this week. But I do that each week.

At Microsoft we have a term, Zero Bug Bounce (ZBB) which is the milestone at which you get to Zero bugs older than n days. Some times it's a week, sometimes 24 hours. But you do that for a few weeks till you cut your Release Candidate build. It seems I wasn't the first to consider the term ZEB for email, but I've adopted it as a term for the goal that I try and attain each work day.

In case you didn't notice, American Car companies are having a hard time these days. Here is some great insight into why, courtesy of Dennis and Avis.

So this past thursday, I was headed home from SeaTac and was greated by a Boeing 737-400 with a repair man, one of those rental crane things, a dirty rag and some WD-40 I guess fixing our tail rudder. This is like the 10th mechanical error I've had this year. It doesn't even phase Alaska (or their customers). The other San Francisco flight was delated from 1 pm to 5:40 pm.

I think Alaska should change their departure times to:

• Morning (depart by 9 am)
• Afternoon (depart by 3 pm)
• Evening (depart by 9 pm)

Just so they can meet the only expectations they can reasonably attain as an airline.

Remember when you thought having a sound play when you got new mail was cool? Remember when you though that little toast that showed up when a contact signed into IM service was cool, or that little toast that Outlook draws when you get new mail? I have turned all that nonsense off, and now for the new tired. Remember when sorting by conversation was cool, and a major improvement over sort by date? You probably don't because you are using either of these in your Inbox right now.

Well I was thinking the same thing when I read this fortune interview of Ray Ozzie and Bil Gates and I smiled for the same reasons the folks at ClearContext did.

This Fortune article has a great interview with Bill Gates and new Microsoft CTO Ray Ozzie on how they see email tools evolving.  It's not hard to guess why we find Ray Ozzie's comment so intriguing:

"Funneling messages in chronological order into an in-box is not necessarily the best model for dealing with different projects, different teams, different issues, while other unrelated things get intermixed with those. You have no sense of the priorities."

You're preaching to the choir here, Ray.  Read the rest of the article for Bill's explanation on how he manages his email account.  Good stuff.

Now I think, imagine if I had to look at my inbox in a manner that was not optimized for what's really important. It's not the attributes of the messages like date and subject that matter. It's how important the people and conversation is to me, and when I need to deal with it.

BTW - I got to meet Ray Ozzie last week and it was one of the highlights of my career at Microsoft. Such an interesting person. I think he will do great things for Microsoft. Very few people can come into the company and have such a great amount of respect from the technical rank and file.

Aditya just wrote something that struck me on his blog post about Rent.

"The trailer for Rent (via Trina) is online. Segements [sic] of Rent were shot on 6th Street, between Mission and Market, which is exactly where I live. For a week or two, movie busses, lights, and extras flooded the streets every night to shoot a couple of scenes. It you look in the trailer, all the New Year's celebration scenes are from my neighbourhood. It's pretty nutty. Club 6 became the Cat Scratch Club (?), 7th Street Haircutters became a mid 80s haircut place, some pawn shop becaome [sic] Crazy Eddie's. It's funny to think that to find grimy mid-80s NYC, the best location they found was the 6th area."

Actually, it's not very funny that they found San Francisco to be a perfect city to represent the grimy mid-80s NYC. The area of the shot, is actually grimy and much like many parts of NYC in the 80s. San Francisco is a great city, and I love it to death, but I grew up in NYC in the 80s, and saw an INCREDIBLE transformation of that city in the 90s. Most of this transformation can be attributed to the Broken Windows theory outlined by sociologists James Q. Wilson and George L. Kelling.

"Their approach worked in New York City’s subways, where felonies have fallen by 75% in the 1990s, and all across New York City as former Police Chief William Bratton implemented many of Kelling’s and Coles’ policy recommendations."

The premise of the theory is that if some one breaks a window, and no one bothers to clean up the glass or prevent the crime from occuring in the first place, criminals, or those that may be inclided to commit crime will be more willing to do so since they think no one cares. This results in a problem that is often hard to fix with a band-aid. A good example of this is my street in SF where once a week at least one car is broken into. There is broken glass all over the place, and it remains that way till the owner shows up. Next week another car is victimized. In NYC in the 80s, my father's car was broken into the only 3 times he ever parked it on the street at night. And mind you, this is on the Upper East Side, one of the nicer parts of the city. When I was a boy walking home from school one day, two guys stole a car radio in broad daylight on 73rd and 5 avenue, and 10 feet in front of me as I walked by. Since the mid 90s I haven't heard of a en a single car broken into on my family's block. As a kid I was mugged twice in NYC. Once in Central Park and once 3 blocks from our apartment. This was the reality of NYC in the 80s.

Some other people think that the same approach that worked for NYC would not work for San Francisco. I have four things for you. 1) 6th street, 2) tenderloin, 3) the mission (mostly mission between 16th and 21st), 4) market street between the Castro and 6th street.

As a child of the 80s and 90s, and seeing how far NYC is come in terms of being a tourist friendly and generally clean city, I am often shocked when going around San Francisco by foot or by car. There are many times I've felt ashamed or embarrassed by the condition of the street, or the people in the street.

This is one of the areas that SF really falls short of NYC. There is just too much graffiti on buildings, broken car glass, trash, and displaced folks roaming around.

Generally speaking, SFO is a fantastic airport. It's 15 minutes from the city, is no where near as crowded as LAX, and has a good selection of flights from the big airlines, and many of the smaller ones. American (my preferred airline of choice has many flights there). There are two problems with SFO though.

1. The Skybridge does not exist for Terminal 1. However, it does exist for Terminal 2 and 3.
2. The AirTrain thingy does not go to long term parking.

Lets look at #1. The Skybridges are these things that connect the AirTrain to the terminals. There are 4 terminals at SFO. The old terminals, 1, 2 & 3 and the new international terminals. So to make it easy to get from the AirTrain to terminals 1, 2 & 3, SFO set out to build these bridges that bypass the necessity to go to the short term parking lot, take an elevator 5 floors down, cross some underground tunnel to the basement of the terminal and take escalators or stairs to check. The AirTrain goes right into the International terminals so they do not necessitate any AirTrain.

Now, Terminal 3 serves United and American Airlines. Terminal 2 is the old International terminal and is currently a ghost town. Terminal one is for everything else. Now get this. SFO build a Skybridge for Terminal 1 & 2, and they ran out of money to build one for 1. I would love to meet the person who decided to build a Skybridge for a Terminal that services NO passengers.

I would also like to meet the person who built out enough rail for the AirTrain to get to the beginning of the Long Term Parking lot, but not close enough to place a station there.

I bet this is just purely a result of the inefficiencies of Bureaucracy.

If you are a user of Cingular (or AT&T Wireless). MSN Mobile Messaging, which I wrote about in the past, is a really cool feature where you can have a two way conversation between yourself and a mobile phone user.

As we speak, I am sending an IM to my wife who is in a hospital where she is not signed into messenger. Rather than use my phone to SMS her I can just do so from the comfort of my computer, and when she replies it just pops back into the conversation window. If I'm not signed into messenger when she replies, it gets routed to Hotmail, and then the next time messenger signs in, all the "offline" messages that are queued for me get delivered to messenger again.

There is nothing like a persistent annoying issue to get Visual Studio fired up to put an end to a problem. Some really annoying person/bot has been leaving me 4-6 Trackback spams per day. This only happens on a handful of posts, and to make matters worse, they point back to a site that has no info.

Well NO MORE! dasBlog can now fight back against Trackback spam. After some time this weekend my Event log now says:

Info TrackbackBlocked:
Trackback Referral blocked for Tablet PCs improve employee productivity at Microsoft from http://bardak.com.ru/ originating at IP Address 211.138.91.30 because the server did not return a valid response

Haha! Coming soon to dasBlog will be:

1. Trackbacks/Referrals can be deleted from posts
2. dasBlog will check an incoming trackback for a link to your site (preventing stupid spammers from getting links on your posts) and serve them a 404 if it does not find one.
3. dasBlog will check if the incoming url to see if it is even alive and server them a 404 if it is not.
4. dasBlog will check the trackback URL against the blocked referral blacklist (Scott checked this in but it's currently not working for me for some reason).
5. A number of other features and bugfixes as time allows (believe it or now, I'm dammed busy with work and life right now).