It’s really great that we’re slowly seeing multiple microblogging platforms crop up. One of the problems with federating all our data across the web, however, is that there’s no way to really track what’s going on. No way to search. If all you care about is Twitter you can use search.twitter.com, and identi.ca and others similarly have their own search for one site, but where is the central search option?

Well, it turns out that it’s really not that hard to build one. Using the powerful libre Xapian full text index solution and PubSubHubBub, a microblog search engine doesn’t even need a crawler.

I’ve launched an early version of this over at µsearch.singpolyma.net or the easier to type musearch.singpolyma.net. To seed content there is a script polling identi.ca and rstat.us periodically, until they implement a PSHB firehose, but any microblog site can get itself added by just adding PSHB feeds with the box on the homepage.

I make no promises at this point about the quality or longevity of the data in the search engine. Please report any issues or features you would like to see.

Of course the source is available under the ISC license. Feel free to submit patches!

Query Syntax

Just typing words will search all the metadata and content of posts for those words. Searching in quotes will search for a phrase. You can also prefix words or phrases with a field name to search only one bit of metadata. The currently indexed fields are:

• content
• category (like hashtags)
• in_reply_to
• bookmark (permalink)
• author
• to (mentioned users)

Don’t get me wrong, I love Google’s Social Graph API, it’s a great way to speed up the discovery of XFN data by using Google’s cache.  What does not make sense to me, however, is their ‘NodeMapper’ concept that is built in to the API.  It maps multiple URLs from a site on to, not a single URL, but a SGAPI-only URI scheme.  It maps using URL patterns that are known about the site, so it doesn’t even work on the web in general.  When it does work, what is it useful for?  URL consolidation.  The problem is, that the only thing you can do with a nodemapped URI is (1) use it as a unique key or (2) turn it back into a URL to get data.

I don’t get it guys.  How is this better?  Is there even a reason to consolidate things like FOAF files backwards to the main page, since most people will enter the main page itself as input anyway?  Even if it was useful, shouldn’t it actually map to the main page and not to some proprietary URI scheme?

Thoughts?  Anyone see a use for this that I’m missing?  Or is this misfeature just adding a layer of data that someone might use and that we’ll have to hack around again later?

I knew others were talking about distributed social networking, but Messina is working on code, and even code for WordPress! The Kilroy concept (an “I was here” plugin) looks excellent. Avatars looks like exactly what I’ve been saying I should do since I installed Gravatars on this blog. The contactlist plugin looks like basically what I have here (his code, I believe), but I’ve hacked mine some.  His delegation looks a lot like what I want to add to my XRDS plugin.  Pempeth would go great with his contact form concept.

Other cool stuff there too — overall just excited that things seem to be picking up a bit in this field — I’m writing some bookmarklets just now, and really should clean and release some of the remaining WordPress code.

Pempeth is the result of my work based on my previous private messaging TEP.  The protocol draft has matured and there are now implementations! (see the page).  Most notable is a WordPress plugin, active on this site.

The development of that plugin also sparked an XRDS plugin, which I have also released (despite its somewhat cryptic interface).

Those looking at the main page may also have noticed changes. Yes, that’s a mini-feed based on my online activity.  Yes, it’s a plugin.  The interface, however, only allows for adding sources (not editing or removing) and is somewhat cryptic, so I have not released it yet.

You can also now log into my blog with your Facebook account (see link in header)!  This uses the API, so I don’t got your Facebook password or anything like that.  Also an as-yet-unreleased plugin.

Fun days!  I’m going to be working at AideRSS as a co-op in the coming months, should be fun and more freeing than school!

I have updated this draft and released some code.

Regular readers of my blog know that I am obsessed with one thing: decentralisation. I hate having all my eggs in one basket. IMHO, if anyone has the ultimate power, then it’s not true Web 2.0. In my recent thinking this has extended to my view of social networks such as Facebook and MySpace.

One of the much-used features of these is personal messages. This can be public (ala Facebook wall) or private (ala Facebook message). I here intend to provide a TEP, which I will implement, suggesting a way to do this in a decentralised and RESTful manner.

Detecting the endpoint

I love XRDS! This format comes to us from the OpenID world but really provides a standard way to associate RESTful endpoints with a URL [example].

I think it makes most sense to have one type URL for this TEP and declare a namespace for extending XRDS to (optionally) tag endpoints with a ‘sub-type’. This could look like the following:

<?xml version="1.0" encoding="UTF-8"?><xrds:XRDS xmlns:xrds="xri://$xrds" xmlns="xri://$xrd*($v*2.0)" xmlns:dsn="http://singpolyma.net/dsn/xmlns">

<XRD>

<Service priority="50">

<Type>http://singpolyma.net/dsn/pmt

<URI>http://example.com/msg

<dsn:msgtype>private</dsn:msgtype>

</Service>

</XRD>

</xrds:XRDS>

(I will create these pages in a bit.) I would propose the following dsn:msgtype for now: private (only to you) and public (like Facebook wall). As this is a TEP, nothing is remotely final. I’m not sure I like msgtype as the tag name, for one thing.

Sending Data to the Endpoint

This part is easy. To support maximum compatibility with existing systems, either GET or POST data is to be accepted. All fields passed must be processed. The simplest system will simply take all data passed via both GET and POST and put the field name as a header and the value under/beside it in the message body. No data may be ignored (unless there is a particular value, such as a session ID, specific to your server which you want removed). The following is a list of fields which make sense to use in a special way. If you are going to do something special for one of these values you should use the name given here, but really any data can have anything done with it as long as it is included somewhere in the message.

subject, from (contact of sending user), cc, url (of sending user), name (of sending user), body

XHTML is allowed. Special data (such as calendar events) should be marked up with microformats where possible.

A way to contact the user back SHOULD be supplied. If either from or url is a URL, XRDS discovery can be performed on it in order to detect an endpoint for this TEP to use in replying.

I am going to write some code to implement this both for WordPress and independently soon. I may also create a simple service targeted at adding this capability to a blogger blog (or other locked-hosting website).