Today I received an email from Boxbe support telling me they had finally given users the option to turn off their “coutesy notification” system.  I couldn’t be happier!  I thought I’d take this post to share about my SPAM problems, and my solution.

The Problem

GMail SPAM filtering is nice.  I may not have it forever, and don’t like to count on it, but it works very well.  Unfortunately I made the choice when I registered this domain name to set up a catch-all.  At first that was fine, but after over a year *@singpolyma.net was receiving so much SPAM, so fast, that even the GMail SPAM filter couldn’t keep up.  I began to receive over 40 SPAM (sometimes over 200) per day, sometimes all at once!  I didn’t want to disable the catch-all though… that felt like the wrong solution.

The Right Solution

I decided the right solution was whitelisting.  Since most of the people I know don’t use PGP (yet) there is no way to guarentee the sender of the messages, but from a cursory glance over my SPAM box I decided that trusting the From: header would work for 99% of today’s SPAM.

I can’t set up a forwarder from a catch-all with Dreamhost, so I set it to be delivered into a mailbox.  I then created a “dummy” Gmail account to fetch this mail via POP3.  Bonus #1, Gmail filters all this mail as it comes in, catching a huge amount of the illigitimate messages (just not enough of them).  Set Gmail to forward all email to singpolyma@boxbe.com (more on that in a bit) and delete.  Using Gmail as an email pipe/filter really.

Then Boxbe.  Boxbe gives you a you@boxbe.com email address that you can forward mail to, it checks it against a whitelist, and sends it on if it matches.  Previously, if it did not match, they would reply with a “challenge” email.  This is annoying, broken, and sometimes embarassing, so I am very pleased that they have now given people the option I wanted all along.  Disable all “courtesy notifications” and turn on the report of the queue, daily.  If I receive any mail from people not on my whitelist, I get an email from Boxbe once a day summarizing who tried to contact me.  I go and let through any legitimate new people.  Perfect.

Boxbe uses the password anti-pattern (although they’re working on fixing that, they say) to import your address book.  They have a CSV importer though.  Export from Gmail, import to Boxbe.  Set up some trusted domains (like *@uwaterloo.ca) and go.

I haven’t seen SPAM since, and have only once or twice had to go over and let through a message that got stopped.

There a fair amount of buzz around messaging sysems, be it microblogging or direct messages.  There is also discussion about broadcast social media (share this with all your friends!).  One use case keeps cropping up for me: sharing content with individuals or ad hoc groups.  I will focus here on sharing links, but much of this applies to any media richer than one raw text blob.

If I want to keep a URL for later – I use bookmarks.  This was de facto for a long time.  Then, one day, someone decided it might be cool if not only they could read that page later, but everyone else could too!  Thus, the birth of social bookmarking.  Today, if I want to share a link with all my contacts I simply bookmark it on my Ma.gnolia, and if they care, they’ll see it.

Then, groups.  If I want to share a URL about copyright issues with the Waterloo Students for the Information Commons, we have a Ma.gnolia group.  Interested parties subscribe, and the stream is also syndicated to the main page of our wiki for general interest.  (Aside: if a discussion with the group is to take place around a link posted there it sometimes happens on our mailing list… I’ve recently begun experimenting with Friendfeed rooms for this.  While commenting on FF in generally seems dumb, in this case many of the shared links have no comments themselves and the commentary would only be interesting to other group members anyway.)

One extension of groups really : ad-hoc groups.  I don’t want to create a new group somewhere and invite everyone who might be interested every time a topic comes up breifly.  It needs to be easy (like, one step, no more than three short fields) and not require people to sign up for anything to contribute/subscribe.  Then it can die out later naturally.  Stronger (more organized) than hashtags, but less formal and permanent than groups.  This is analogous to the cc-everyone chains that develop because people are too lazy to make a small, temporary mailing list.

Alright, now to the big one: point-to-point.  While 1:1 communication is usually not the answer (and this has partially sparked my ideas about ad-hoc groups) – sometimes you just read a page and go “so-and-so would be interested in this”.  This has, in the past, caused me to email URLs to people.  This feels like the wrong solution.  Even Twitter dm doesn’t seem quite suited to this.  First I will describe my ultimate UX, then I will describe what seems to exist today.

I want a button in Firefox (or whatever browser I end up using in the future – Firefox for now) that opens a dialog allowing me to simultaneously save the link into my bookmarks (on Ma.gnolia or wherever), share with an arbitrary number of groups, and with an arbitrary number of contacts.  You can take a peek at my mockup if you like.  This is very different from how, say, Ma.gnolia or Pownce does link sharing.  Note that all of these (my bookmarks, some groups, some contacts) should be optional – I may not want to use all of them each time.  When people send me links this way I want an RSS feed of the links.  If they get emailed to me it is not much better than the original solution.  If they are delivered into some “private message” box we have YAI, and that’s worse.

Tie in to DiSo: wouldn’t it be extra neat if I could type not just, say, Ma.gnolia or Pownce usernames, but could type URLs?  System asks their provider how they prefer to recieve links and then sends it that way.  I really don’t want to make people sign up for whatever service I happen to use.

So what can we use today?  Well, there are a few options.

1. Emaling/dming/@heyyouing URLs can work – but it’s not ideal for one key reason: there is no simple way to get a “list of recent links”.  I don’t want to go through every recent email or tweet to find a URL.  Some people prefer this because it facilitates discussion around the link somewhat.
2. Pownce.  Using, say, http://pownce.com/send/link/?url='%20+%20encodeURIComponent(window.location.href)%20%20+%20'&note_body='%20+%20encodeURIComponent(document.title)%20));">a bookmarklet, one can add links to Pownce and send them to contacts or even “sets” (not-quite-ad-hoc-groups).  The key issues here are that if I also want to bookmark the link (I usually do) I must do that separately with a separate form and bookmarklet.  I must also re-post to Pownce for each contact/set I want to send it to.  There is also the issue that people would have to sign up for yet another social media account in order for me to share links with them – Pownce doesn’t have OpenID support just now.
3. del.icio.us for: tags.  This is not too bad of a solution if all your contacts are on del.icio.us… and if you use it yourself.  I really need to get that Ma.gnolia-to-del.icio.us bridge project finished.
4. Ma.gnolia groups.  This is a hack really, but it’s working for myself and a contact of mine.  We have set up Ma.gnolia groups whose sole purpose is for others to share links with us.  Anyone with an OpenID can just log in and start sharing links with us, which we then get from the groups’ RSS feed.  The problems here are: it’s a hack and sharing with more than one group at a time is still a pain.

Enough from me for now.  Think about it.

Older readers will remember my Blogger Recent Comments service that was hosted on Ning.  It has be obsolete for some time now (since Blogger now has provided their own comment feeds since v3) and has long since been shut down by Ning for being a horrible bandwidth hog.  I have thus decided to delete the app (since it’s just taking up space on my Ning account) and release the code under an MIT-style license on my devjavu repository.  Maybe someone will find it useful.

Now to go see if I can use Ning for my next project… Google App Engine not having a decent cron/fakecron is not at all useful… and my Python sucks.

I’ve recently been moving from the slow, bulky AJAX of the Gmail interface to the nice, lean, familiar keybinding of mutt.  Below is a simple ruby script I wrote to convert the Gmail CSV of your contacts to useful mutt aluases:

#!/usr/bin/ruby
require 'iconv'

names = []

Iconv.iconv(‘UTF32//IGNORE’,’UTF-8′,File.open(ARGV[0]).read)[0].gsub(/\000/,”).sub(/^../,”).each(“\n”) do |line|
f = line.chomp.split(/”?,”?/)
f[1] = f[1].split(‘@’)[0].sub(‘%’,’@’) if f[1] =~ /%/
f[0] = f[1] if f[0].to_s == “”
next if f[0] == ‘Name’
puts “alias \”#{f[1]}\” \”#{f[0]}\” <#{f[1]}>”
next unless names.index(f[0]).nil?
puts “alias \”#{f[0].gsub(‘ ‘,”)}\” \”#{f[0]}\” <#{f[1]}>”
names << f[0]
end

I haven’t written about the XOXO microformat in some time, but some recent discussions caused me to dig into my archives to source a new post.  Microformats tend to follow the rule of only formalizing the most common of existing publishing patterns (the 80-20), meaning that some more “edge case” data cannot be represented.  Does this mean that this data is useless?  Not at all: but it is outside the realm of microformats, at least for now.  So we either need to invent something new, or extend what we have.

A Page from Recent History

This is not a new problem.  Every formalised standard is going to face those who feel that their bit of metadata should be included.  Take, as an example, the RSS 2.0 spec.  Core essentials of news feeds are present: title, description, date, etc.  Lots of metadata is missing though: author name, comment counts, comment feed URLs, ane more.  People solved this problem in two very different ways: some extended, and some invented something new.

Extending RSS (or any XML format) is easy: create a namespace, add your elements, publish.  If a particular piece of metadata is popular it gets standardised in a spec’ed extension (dc:creator, slash:comments, wfw:commentRss, etc).  The benefit of this approach is that all existing parsers can still read your content.  If a parser doesn’t need your extra metadata, it can safely ignore it and present just the core content.  No new code needs to be written, and no new formats need to be learned for 80% of the applications.

There was another group interested in solving this problem: the ATOM group.  They threw away all the existing formats (RSS 2.0 and RSS 1.0/RDF) and built something brand new from scratch to accomodate their data needs.  What was the result?  Feed aggregators everywhere had to write all-new code to handle this new, incompatible, and often more complicated case.  Time and effort was wasted both in code and user education (unlearn “What is RSS” learn “What is ATOM” / “What are feeds”).  Once the standard hit a spec’ed form, what happened?  People began to use namespaces in ATOM as well, because for all the “better” it was, for some edge cases it just wasn’t “better” enough.

Back to XOXO

It seems the key is to be easily extendable, not to think of everything up front.  If microformats are going to make their way into lots of APIs and not just be used for better page scraping (Ma.gnolia does a good job of this), then extensability is necessary.  Fortunately, XOXO provides an easy solution.  Check this out:

<ul>
<li class="vcard">
<dl>
<dt>fn</dt>
<dd class="fn">Martha</dd>
<dt>Anniversary</dt>
<dd>2005-02-04</dd>
</dl>
</li>
</ul>

An hCard parser can read that.  For a normal use case, no new code is needed.  An XOXO parser can read that, and if it knows about hCard will likely know what “fn” means.  The other data is there, though.  The parser has that data.  Minimal new code, and all the data can be used.  Cool or what?