It’s really great that we’re slowly seeing multiple microblogging platforms crop up. One of the problems with federating all our data across the web, however, is that there’s no way to really track what’s going on. No way to search. If all you care about is Twitter you can use search.twitter.com, and identi.ca and others similarly have their own search for one site, but where is the central search option?

Well, it turns out that it’s really not that hard to build one. Using the powerful libre Xapian full text index solution and PubSubHubBub, a microblog search engine doesn’t even need a crawler.

I’ve launched an early version of this over at µsearch.singpolyma.net or the easier to type musearch.singpolyma.net. To seed content there is a script polling identi.ca and rstat.us periodically, until they implement a PSHB firehose, but any microblog site can get itself added by just adding PSHB feeds with the box on the homepage.

I make no promises at this point about the quality or longevity of the data in the search engine. Please report any issues or features you would like to see.

Of course the source is available under the ISC license. Feel free to submit patches!

Query Syntax

Just typing words will search all the metadata and content of posts for those words. Searching in quotes will search for a phrase. You can also prefix words or phrases with a field name to search only one bit of metadata. The currently indexed fields are:

• content
• category (like hashtags)
• in_reply_to
• bookmark (permalink)
• author
• to (mentioned users)

Version 1.0 of the DiSo Actionstream plugin for WordPress has finally been released! The upgrade for this version may be a little rough, because the entire data storage model has changed since the last release. Let me know if you have any trouble!

For Debian users, there is a wordpress-diso-actionstream package in my APT repository.

I am writing my own stuff, borrowing heavily from @tantek.com

1. Store wp-diso-actionstream content as WordPress posts (DONE, commited to SVN)
2. Tweak theme to display actionstream items nice for single.php (DONE)
3. Hack local WordPress core to display posts of all types when you visit the permalink (DONE, see https://singpolyma.net/2010/01/the-freeallmusic-com-non-us-landing-page-is-great-short-and-to-the-point/)
4. Make reversible URL shortener. (DONE, http://sngpl.ma/t433)
5. Update Picoformats (Microsyntax) plugin to support URL autolinking and generally be better (IN PROGRESS DONE, borrowing from @tantek.com’s auto_link)
6. Implement Twitter API for wordpress.org blogs (IN PROGRESS DONE)
7. Write XMPP bot to talk to any Twitter-compatible API (NOT STARTED found one)
8. Write plugin to push new local tweets out to Twitter-compatible APIs (NOT STARTED)

Bonus points:

• Write plugin to pull tweets from Twitter-compatible APIs locally for viewing (“following”)(DONE)
• Following-type plugin based on wp-diso-actionstream (to follow arbitrary public streams)
• PuSH outbound for blog feeds and wp-diso-actionstream feeds (DONE, using http://wordpress.org/extend/plugins/pubsubhubbub/)
• PuSH inbound for wp-diso-actionstream

So, finally someone talking about the future of distributed social networking. The tech and the connecty bits we want have really been mostly there for some time now, the problem is, no one has been very clear on what the next step is. Chris Messina has been a bit distracted with the Activity Streams project, and no one else has really been saying much about DiSo.

The next step, however, is really coherent UI. I’ve been talking about it off and on as my “ultimate aggregator”, Marc Canter is calling it “dashboards”.

One of the things he talks about in the presentation is “distributed friending”. This is something I’ve brought up before. IMHO, the best way to go about this is to have magical buttons that, when clicked, take the user to their “dashboard” with the target’s URI (or one of them, anyway) already filled in. At that point, you have an asynchronous friending model. The local software can then do different things (like permissions, autofilling searches, pulling in content, just making the list available to other services than then do these things, whatever) based on this data, but no magical “protocol” or anything is needed, because with an asynchronous model all you’re really doing is making a note of the relationship in a data model and letting the software use that list for whatever.

Past integrating the posting/following/aggregation UI a bit more, I’m not really sure there’s anything left, conceptually. I’d like to dig up some code and make OAuth+AtomPub work for sure with the newest version (so that any aggregator can talk to my WP blog 🙂 ), and code can always be improved, but really, what is a social network? It’s an aggregator of sorts, a posting mechanism of sorts, and email. We’ve had the later two for ages, which is why so much work has been dancing around the first one.

First off, why would one want to do this? Well, cryptographic security is useful in communications medium other than email, and sometimes one may not have an email address for the person one is contacting. Also, a public key got from someone’s profile page is more likely to be their current key than the one got off a keyserver. Finally, if the discovery is done over TLS (or upcoming XRD signing techinques) then one can use the PKI to verify that the public key is, at very least, the one the owner of the URL claims. Which, for pseudonymous communications, may often be enough.

I will here propose three different ways to make this discovery work. Consumers must try all three. Publishers may publish more than one.

Content Negotiation

A public key represents a person. If a URL represents a person (such as on a profile page), then were that page’s data to be represented in the OpenPGP key format, one would get the user’s OpenPGP public key.

Send the header Accept: application/pgp-keys along with an HTTP GET request. If the Content-Type on the response is application/pgp-keys then the body is the user’s OpenPGP public key.

Links

If a GET or HEAD request is performed on the URL and in the headers is a Link header with rel=me and type=application/pgp-keys, then the URL of that link is the user’s OpenPGP public key.

If the Content-Type header of the GET request is text/html or application/xhtml+xml, then look in the page for <a> and <link> tags with rel=me and type=application/pgp-keys. If there is such a tag, then its href attribute is the URL to the user’s OpenPGP key.

LRDD+XRD

If LRDD discovery is performed on an endpoint, leading to the discovery of an XRD document containing a section like the following:

<Link>
<Rel>http://www.iana.org/assignments/relation/me;
<MediaType>application/pgp-keys</MediaType>
<URI>...</URI>
</Link>


Then the URI is the URI of the user’s OpenPGP key.

Security Considerations

The URLs used in all methods above should be either HTTPS URIs secured using TLS and a certificate issued by a CA known to the client, or data URIs.

Application to Other Crytography Schemes

Everything in this document applies equally well to public keys for any cryptography scheme, as long as the MIME types are changed appropriately.