I’ve blogged numerous times about XMPP, SMTP, and communications evolution on the web.  I’ve suggested what I want ultimately and snippets of how we might get there.  Here, I am going to outline just briefly what I consider “next steps”.  The big ones.  Get these done, and you will have made a *huge* stride in online messaging:

1. Allow offline messages (type normal or chat) to be collected as “email”.  Gmail sort-of does this by presenting unseen offline messages in the web interface inbox.  I want IMAP access to these in the inbox and their archive.  Heck, store them in a Unix mailspool (have to store them somewhere anyway) and existing IMAP servers will just work for you!
2. SMTP messages are type=normal.  If you store offline messages in a mailspool and run an SMTP server on that spool, you’re mostly done.  Might be good to offer real-time deliver of those messages to the user of XMPP as well though.

That’s it! Sure, more can be done, but if you get the first one done I will be your biggest fan.  Do both and you’re well on your way to an evolution in how we deal with email (both from a user and a protocol perspective).  Yes, I’ve tried to build this.  I want to do it as an ejabberd module, but ejabberd is barely documented.  I’ll try again sometime if no one else does – maybe with ejabberd, maybe with someone else.

Today I received an email from Boxbe support telling me they had finally given users the option to turn off their “coutesy notification” system.  I couldn’t be happier!  I thought I’d take this post to share about my SPAM problems, and my solution.

The Problem

GMail SPAM filtering is nice.  I may not have it forever, and don’t like to count on it, but it works very well.  Unfortunately I made the choice when I registered this domain name to set up a catch-all.  At first that was fine, but after over a year *@singpolyma.net was receiving so much SPAM, so fast, that even the GMail SPAM filter couldn’t keep up.  I began to receive over 40 SPAM (sometimes over 200) per day, sometimes all at once!  I didn’t want to disable the catch-all though… that felt like the wrong solution.

The Right Solution

I decided the right solution was whitelisting.  Since most of the people I know don’t use PGP (yet) there is no way to guarentee the sender of the messages, but from a cursory glance over my SPAM box I decided that trusting the From: header would work for 99% of today’s SPAM.

I can’t set up a forwarder from a catch-all with Dreamhost, so I set it to be delivered into a mailbox.  I then created a “dummy” Gmail account to fetch this mail via POP3.  Bonus #1, Gmail filters all this mail as it comes in, catching a huge amount of the illigitimate messages (just not enough of them).  Set Gmail to forward all email to singpolyma@boxbe.com (more on that in a bit) and delete.  Using Gmail as an email pipe/filter really.

Then Boxbe.  Boxbe gives you a you@boxbe.com email address that you can forward mail to, it checks it against a whitelist, and sends it on if it matches.  Previously, if it did not match, they would reply with a “challenge” email.  This is annoying, broken, and sometimes embarassing, so I am very pleased that they have now given people the option I wanted all along.  Disable all “courtesy notifications” and turn on the report of the queue, daily.  If I receive any mail from people not on my whitelist, I get an email from Boxbe once a day summarizing who tried to contact me.  I go and let through any legitimate new people.  Perfect.

Boxbe uses the password anti-pattern (although they’re working on fixing that, they say) to import your address book.  They have a CSV importer though.  Export from Gmail, import to Boxbe.  Set up some trusted domains (like *@uwaterloo.ca) and go.

I haven’t seen SPAM since, and have only once or twice had to go over and let through a message that got stopped.

I’ve been interested in different forms of communication for some time.  It’s part of what makes social networking so interesting to me.  I’ve been reading about others’ experiences too.  Of course there’s Tantek’s CommunicationProtocols page, which inspired my own Communication Protocols section on my main page (and, probably, @seanbonner’s PreferedMeansOfContact).  Trevor Creech recently challenged me on my Twitter usage, calling it “Twitterfail” (in reference to efail).  I would like to discuss some of they ways I’ve started thinking about communication.

First, a tip from my own main page: If you find a solution, from me or elsewhere, blog it. Someone else may benefit.  I have come to think of my Twitter and Ma.gnolia accounts as blogs (especially since they began to manifest their updates in the actionstream on my main page).  If I find an interesting tidbit, or have a potentially interesting thought, I tweet it.  This lead, one day, to >20 tweets in 24 hours, the condition which Trevor complained about.  In response, I have tried to consider first if something is really useful at all before I tweet (I don’t want to be the cause of a signal/noise problem) – but I have also started including better context in my tweets.  Interesting links go to Ma.gnolia.

Searchability has become key for me. This is one of the reasons I love my IM setup – everything anyone says to me, whether I’m online or off, at my computer or not, is archived in Gmail for easy search.  My tweets and blog posts are also searchable – in fact, if you just say “singpolyma” or link to me in a blog post, there’s a huge chance that I’ll see it.

When it comes to factors like immediacy, lifespan,  audience, bandwidth, and sychronity they are all important, but are different for different messages.  If I’m setting up a meeting or working on a project, immediacy and bandwidth are hugely important (thus, face to face or IM are best).  If I’m discussing something of interest to me, asychronity, lifespan, and audience are the most important factors (thus, mailing lists, forums, Pibb, and IRC are best).  There is no “perfect” communication form – all have their place.

I have requested that people use post/page comments for debugging/feature requests on my projects.  This is because a comment is almost as good as blogging something (it can be used by others who may benefit) and is searchable.  It also reduces the chances that I get asked for the same thing a bunch of times – others can see what is being discussed (which, incidentally, in the same reason I love GetSatisfaction).  Pages + comments are almost as good as (in fact, in many cases, I feel are better than) a wiki.

Pempeth is the result of my work based on my previous private messaging TEP.  The protocol draft has matured and there are now implementations! (see the page).  Most notable is a WordPress plugin, active on this site.

The development of that plugin also sparked an XRDS plugin, which I have also released (despite its somewhat cryptic interface).

Those looking at the main page may also have noticed changes. Yes, that’s a mini-feed based on my online activity.  Yes, it’s a plugin.  The interface, however, only allows for adding sources (not editing or removing) and is somewhat cryptic, so I have not released it yet.

You can also now log into my blog with your Facebook account (see link in header)!  This uses the API, so I don’t got your Facebook password or anything like that.  Also an as-yet-unreleased plugin.

Fun days!  I’m going to be working at AideRSS as a co-op in the coming months, should be fun and more freeing than school!

This is a repost of the Pranketh avoid article.

The Problem

If Pranketh’s existence proves anything, it is that email is not the safest medium around. It has always been relatively easy to send an email that says it came from someone it did not, similar to the way one can write any return address on an envelope when sending a letter. So, now that Pranketh has made this problem very obvious, how can one determine if an email is what we call ‘spoofed‘?

Some email providers and programs show warnings on messages that may be spoofed, but the problem is that detecting spoofing is more art than science. A legitimate email may be spoofed (for example, if you write Pranketh and we write you back, we are actually writing you from our GMail accounts, but it will appear as though it came from Pranketh, which, in reality, it did) or a spoofed email may not be detected (because it also spoofs whatever the automatic detection system uses).

Message Headers

First of all, you’ll want to view what we call the ‘message headers’. Some of them (From, To, Subject) are always visible. Depending on your program, different ones will usually be hidden. The option to view them all may be called ‘View Message Headers’, ‘All Message Headers’, ‘Original Message’ or something similar. Below are some screenshots for two popular email services (more will be added as time goes on) :

View Headers in GMail

View Headers in Evolution

View Headers in Eudora

View Headers in Outlook

View Headers in Outlook Express

Now that the headers are visible, there are a few key things to check for. The first is a special header added by Pranketh to all emails it sends. If this header is there, we can be sure the email was sent using Pranketh! The line will likely be near the bottom and will look like this :

X-Joke: This email is not from whom it appears to be from. It was sent from pranketh.com.

What if someone is spoofing you without using Pranketh? Thankfully, there are other things you can check. You should see if there is a Return-Path header, similar to the following :

Return-Path: <singpolyma@sunkist.dreamhost.com>

The email-address-like part of that should be similar to who it says it is from (it does not have to be an exact match, but should be similar). If it is not similar at all (i.e., the above is on an email that says it is from bill@microsoft.com) then the email may be spoofed (see the next section for more on that ‘may’).

Another header to check for is ‘mailed-by’. For example, if an email claims to be from a GMail address it may have a header like the following :

mailed-by: gmail.com

That’s pretty simple.

If none of the above is present, or if it all checks out, you may want to checked the ‘Received’ section. It will look something like the following :

Received: from smarty.dreamhost.com (d06184b1.dreamhost.com [208.97.132.177])
by spaceymail-mx3.g.dreamhost.com (Postfix) with ESMTP id 8EF98188FC7
for <feedback@pranketh.com>; Wed, 16 May 2007 16:29:36 -0700 (PDT)
Received: from sunkist.dreamhost.com (sunkist.dreamhost.com [208.97.175.14])

by smarty.dreamhost.com (Postfix) with ESMTP id 7E510EE2C4
for <feedback@pranketh.com>; Wed, 16 May 2007 16:29:36 -0700 (PDT)
Received: by sunkist.dreamhost.com (Postfix, from userid 1429516)
id 81E63402A3; Wed, 16 May 2007 16:29:36 -0700 (PDT)

Notice how there are many references to dreamhost.com. That is because this email was sent from an address that lives there (actually, it was sent by Pranketh). A GMail email will have gmail.com, google.com, or googlemail.com there instead. A Hotmail email should have hotmail.com, etc.

Maybe Spoofed

Why in the above paragraphs did we say that if any of that was true the email ‘may’ be spoofed? Well, remember, detecting spoofing is more art than science. My email addresses all live on dreamhost.com and I send most of my email through GMail, but my email addresses are all at singpolyma.net. So how can you tell the difference between an email that’s spoofed on purpose by the person that owns it, or an email that is not from who it says it is? The best way is to check emails that you know are really from them. If they are spoofed in a similar way, then the email is likely legit. If they do not normally spoof their emails, or if the spoofing looks a lot different than normal, be very suspicious.

If you are not sure an email is really from someone, write them and ask if they sent it. That way, you can be absolutely sure.

Spread the Word

A lot of people trust email every day. It is our responsibility as people who know how to detect spoofing to spread the word. Link to this article, post it on your site, email it to friends, review it, translate it, anything that you think will get the word out faster!

We only ask that you give us credit and link back to this page according to the terms of a
Creative Commons Attribution-ShareAlike license.

How to Avoid Getting Pranketh’d, Scam’ed, or Phish’ed by Pranketh is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.