ralphm's blog

Sunday, 19 July 2009

Twisted at Mediamatic

How Georgious is Twisted?

Even before I got to work for Mediamatic Lab, Mediamatic was using Twisted. My friend Andy Smith used it for a bunch of projects around physical objects, usually involving some kind of RF tags. Examples include the Symbolic Table and the Friend Drinking Station. From this grew fizzjik, a Twisted based library that implements support for several kinds of RFID readers, network monitoring and access to online services like Flickr and of course anyMeta.

On the other hand, I have dabbled in Twisted for quite a while now, mostly contributing XMPP support in Twisted Words and through the playground that is known as Wokkel. But why go through all that effort, while there are a several different Python-based XMPP implementations out there? And why does Mediamatic use Twisted? Why do I believe Twisted is awesome?

First of all, we like Python. It is a great little language with extensive library support (batteries included), where everything is an object. Much like in anyMeta. It is a language for learning to program, to code small utility scripts, but also for entire applications.

But going beyond that, building applications that interact with different network protocols and many connections all at the same time is a different story. Many approach such a challenge by using preemtive threading. Threads are hard. Really hard. And Python has the GIL, allowing the interpreter to only execute byte codes in one thread at a time.

So in comes Twisted. Twisted is a framework for building networked applications in Python, through a concept known as cooperative multitasking. It uses an event loop that hands off processing of events (like incoming data on a socket or a timer going off) to non-blocking functions. Events loops are mostly known from GUI toolkits like GTK, and so Twisted goes even beyond networking by working with such toolkits' event loops, too. As most network protocol implementations only have a synchronous interface (i.e. one that blocks), Twisted includes asynchronous implementations of a long list of network protocols. For the blocking interfaces that come from C libraries, like databases, Twisted provides a way to work with their threads, while keeping all your controlling code in the main thread. Asynchronous programming does take some getting used to, hence Twisted's name.

So how do we use Twisted? Well, a recent application is our recent RFID polling system. It allows people to use their ikTag (or any card or other object with a Mifare tag), tied to their user account on an anyMeta site, to take part in a poll by having their tag read at an RFID reader corresponding to a possible answer. The implementation involves:

  • Communication with one or more RFID readers (that also have output capabilities for hooking up lamps, for example).

  • Communication over HTTP to access anyMeta's API to store and process votes.

  • Network availability monitoring (can we access the network and specifically our anyMeta site?)

  • Power management monitoring (do we still have power?)

  • Device monitoring. Are RFID readers plugged in or not? Which device handle are they tied to? Also watch coming or leaving devices.

  • A (GTK-based) diagnostic GUI

Additionally, we also want to show polling results, so we have a browser talking to a local HTTP server and a listener for XMPP publish-subscribe notifications.

This is quite a list of tasks for something as seemingly simple as a polling stations. But wait: there can be multiple readers tied to a particular poll answer, likely physically apart, a polling question can have maybe 50 answers (depending on the type of poll, like choosing from a collection of keywords) or there could be a lot of questions at one event.

So, back to Twisted. Twisted has HTTP and XMPP protocol support (both client and server-side), can talk to serial devices (like your Arduino board) and DBus (for watching NetworkManager and device events) and provides event loop integration with GTK to also process GUI events and manipulate widgets based on events. Together with Wokkel, it powers the exchange of information in our (and your?) federating social networking sites. In Python. No threads and associated locking. In rediculously small amounts of code. That's why.

Not yet convinced? Add a Manhole to your application server, SSH into it, and get an interactive, syntax highlighted Python prompt with live objects from your application. Yes, really.

XMPP Summit #7 and OSCON 2009

Two great flavors...

I am attending XMPP Summit #7 and part of OSCON 2009, with which it is co-located due the kind folks at O'Reilly. Much like last year, only this time in San José, California. Unlike the European version of the summit last February, we hope to focus more on doing than talking, although there will be plenty of that, of course.

Suggestions were made to do some interoperability testing, along with general hacking sessions. I am bringing my implementation of server-to-server dialback, and a bunch of other protocol implementations in Wokkel to the table. While there are a bunch of other protocol implementations in Python, I think the Twisted approach is so different that I want people to know about the ideas behind it. By introducting them to Twisted through Wokkel should give them at least a glimpse of why I believe Twisted is awesome.

So, nearing the summit I prepared a bunch of examples around the XMPP Ping protocol, as I mentioned before. Additionally I prepared an example echo bot on steroids, which is basically a stand-alone XMPP server that connects to other servers using the server-to-server protocol. It will accept presence subscriptions to any potential account at the configured domain, sending presence and echoing all incoming messages.

Besides the hacking sessions, I planning to discuss publish-subscribe delete-with-redirect, node collections, publish-subscribe in multi-user chats and service discovery meta data. Oh, and we might go on a field trip to discuss Google Wave XMPP-based federation protocols. Then, after the summit, I will hanging out at OSCON until Thursday, for hallway meet-ups on federating social networks with protocols like OpenID, OAuth and technologies like webfinger and pubsubhubbub. I also brought an RFID reader to play with.

Wednesday, 8 July 2009

Wokkel releases

Making crispy sounds...

Today's Wokkel 0.6.2 release is to show case some of the features in the previous 0.6.0 release. Most of the work was part of the things we have been building at Mediamatic Lab as part of a restructuring of how we federate our social networking sites using publish-subscribe.

First of all, I added a preliminary, but functional, implementation of server-to-server support, using the dialback protocol. This complements the router code that went into 0.5.0 and Twisted Words 8.2.0 to make a fully stand-alone XMPP server. Note that it does not implement any client-to-server functionality yet, but this can be added as separate server-side components now.

To show this off, I have created a bunch of examples around the XMPP Ping protocol, for which the protocol implementation itself is also a nice example of how to write XMPP protocol implementations using Twisted Words and Wokkel. Be sure to check out these examples.

The other feature I want to mention is publish-subscribe Resources. They provide an abstraction of (part of) a publish-subscribe service. The protocol parts are handled by Wokkel. This should make it easier to do node-as-code scenarios, by just filling in the blanks of the various methods that are called upon receiving requests from pubsub clients. I'll create some examples for this shortly.