A Couple of Python Tools for Tornado or AMQP Users

Hi all,

Not lots of time to blog since starting my new gig as Senior Operations Developer at MyYearbook.com (which is pretty awesome, btw), but wanted to let anyone who’s interested know about a couple of tools I’ve released on github:

bunny is a CLI tool for doing quick tests and simple operations on an AMQP server (I use RabbitMQ for testing, but it should work with any AMQP server). It’ll do your basic create/delete of exchanges, queues, and bindings. It’s only capable of doing things that are part of AMQP, and I still need to finish the feature that tests functionality via a ’round trip’ test, where it sends a message, then gets it, then sends the body to stdout. If you find time before me to finish that, send a pull request or patch.

curtain is a mixin for use with the Tornado web server that provides HTTP Digest Authentication. I actually started with this recipe and altered/updated it in various ways. It’s still not complete according to RFC 2617 — I need to code for a few edge cases like getting a stale nonce, but what’s there does work, so if you (like me) were wondering what the performance comparison would look like between tornado doing digest auth and having a front-end proxy do it, have at it :)

Way Better Python Indentation In Vim

Just wanted to share this tip I just stumbled across:

If you code in Python, and in your ~/.vimrc file you have ‘set smartindent’, change that to ‘set nosmartindent’ and add the following two lines (I’m told it can be combined into a one-liner, but I use two lines, so…)

filetype plugin on
filetype indent on

The indentation support for Python is much better. What I need to do now is write a little vim-fu that will tell vim to decrease the line indent if I enter two newlines or something, since there are no block delimeters (like braces or brackets) to tell vim about the block level.

Python, Creating XML, Recursion, and Order

I love being challenged every day. Today I ran across a challenge that has several solutions. However, most of them are hacks, and I never feel like I really solved the problem when I implement a hack. There’s also that eerie feeling that this hack is going to bite me later.

I have an API I need to talk to. It’s pure XML-over-HTTP (henceforth XHTTP). Actually, the first issue I had was just dealing with XHTTP. All of the APIs I’ve dealt with in the past were either XMLRPC, SOAP, which have ready-made libraries ready to use, or they used GET parameters and the like, which are pretty simple to deal with. I’ve never had to actually construct an HTTP request and just POST raw XML.

It’s as easy as it should be, really. I code in Python, so once you actually have an XML message ready to send, you can use urllib2 to send it in about 2 lines of code.

The more interesting part is putting the request together. I thought I had this beat. I decided to use the xml.dom.minidom module and create my Document object by going through the DOMImplementation object, because it was the only thing I found that allowed me to actually specify a DTD programmatically instead of hackishly tacking on hard-coded text. No big deal.

Now that I had a document, I needed to add elements to it. Some of these XML queries I’m sending can get too long to write a function that manually creates all the elements, then adds them to the document, then creates the text nodes, then adds them to the proper element… it’s amazingly tedious to do. I really wish Python had something like PHP’s XMLWriter, which lets you create an element and its text in one line of code.

Tedium drives me nuts, so rather than code this all out, I decided to create a dictionary that mirrored the structure of my query, with the data for the text nodes filled in by variables.

query_params = {'super_special_query':
                   {
                      'credentials': {'username': user, 'password': password, 'data_realm': realm},
                      'result_params': {'num_results': setsize, 'order_by': order_by},
                       query_type: query_dict
                    }
                }

def makeDoc():
    impl = getDOMImplementation()
    dt = impl.createDocumentType("super_special", None, 'super_special.dtd')
    doc = impl.createDocument(None, "super_special", dt)
    return doc

def makeQuery(doc, query_params, tag=None):
    """
        @doc is an xml.minidom.Document object
        @query_params is a dictionary structure that mirrors the structure of the xml.
        @tag used in recursion to keep track of the node to append things to next time through.

    """

    if tag is None:
        root = doc.documentElement
    else:
        root = tag

    for key, value in query_params.iteritems():
        tag = doc.createElement(key)
        root.appendChild(tag)
        if isinstance(value, dict):
            makeQuery(doc, value, tag)
        else:
            root.appendChild(tag)
            tag_txt = doc.createTextNode(value)
            tag.appendChild(tag_txt)

    return doc.toxml()

doc = makeDoc()
qxml = makeQuery(doc, query_params)

This is simplistic, really. I don’t need to deal with attributes in my queries, for example. But it is generic enough that if I need to send different types of queries, all that’s required is creating another dictionary to represent it, and passing that through the same makeQuery function to create the query.

Initial testing indicated success, but that’s why you can’t rely on only simple initial tests. Switching things up immediately exposed a problem: The API server validated my query against a DTD that enforced strict ordering of the elements, and Python dictionaries do not have the same notion of “order” that you and I do.

So there’s the code. If nothing else, it’s a less-contrived example of what you might actually use recursion for. Tomorrow I have to figure out how to enforce the ordering. One idea is to have a separate list to consult for the ordering of the elements. It requires an extra outer loop to go through the list, get the name of the next tag, then use that value to ask the dictionary for any related values. Seemed like a good way to go, but I had a bit of difficulty figuring out how to make that work at all. Maybe with fresh eyes in the AM it’ll be more obvious — that happens a lot, and is always a nice surprise.

Ideas, as always, are hereby solicited!

It’s so much easier now

The information superhighway’s construction has benefited me in ways I’ve only been made conscious of this weekend.

I’ve had a lot of time to myself this weekend, and decided to spend a few solid 1-hour blocks playing guitar. Oh, I still play around the house here and there, but usually if I can get in 20 minutes I feel privileged. Learning more kids songs is helping :)

But this weekend I wanted to learn a couple of songs in particular. I wanted to learn “Why Georgia” and “My Stupid Mouth” from John Mayer’s “Room for Squares” album. Although since that album’s release Mr. Mayer has gone on to do more produced and (to a guitarist) less interesting things on his commercial albums, Room for Squares contained really wonderful textures and sounds while somehow keeping the guitar front-and-center.

John also uses a lot of cool chords and fingerpicking techniques that I knew I’d have fun with if I ever had the time to sit down with the songs and my guitar at the same time. I’ve been playing for almost 25 years, and knew a lot of the chords and techniques, but it’d be great to be able to use them in songs that people under 30 recognize rather than having to explain (for example) who James Taylor is.

How I Learn Songs: First Phase

I learn songs by first listening to them either with headphones on, or at a relatively high volume, about 20 times. I want to become intimate, at a subconscious level, with every detail of the song. I want to feel the tension build, recognize cues from all of the instruments, and understand how they’re playing off each other. This is wildly important to a one-man musician, because often I try to embellish the final piece to try to put hints of the entire song into the final piece — not just the guitar part.

Of course, I *start* with the guitar part, because it’s a bridge to the rest of the song. I learn it as close to note-for-note as possible (where there are solos, the same goes for the solo. I’m sick that way).

Once I can play the song through along with the recording and not hear anything going awry, I practice it that way several times. Maybe 10 times.

Then the fun begins.

Having Fun Learning: Second Phase

When I was a kid, my mom and her sisters liked to go to flea markets. I was not a big fan of these huge collections of what looked like garage sales, until I came across a group of tables with what must’ve been 1000 cassette tapes (that was the popular audio medium of the day). When I scanned the collection, I noticed an unusually large collection of Jimi Hendrix tapes. I figured they were compilations, but after picking one up, I realized it was actually a random collection of live and unreleased recordings.

Over the course of months I collected as many of these as I could, because Hendrix was an artist that always sent me running to the guitar. All told I had about 6-8 versions of Red House, a classic 12-bar blues jam. I had 3-4 versions of Foxy Lady, another 3 of Purple Haze, and lots more. Going through and picking out differences between the live and recorded versions was a great primer on how you can alter and embellish while staying within the same basic framework of a song without totally destroying it.

From then on, whenever I learned new songs or started studying a particular guitarist, I tried to get as many different versions of different songs as I could, to see how the guitarist thought about the piece. Guitarists on the road forced to play the same songs night after night get bored, and often try to make things interesting by altering different parts of the song, or even by changing the guitar’s role in different parts of the song. Fascinating stuff.

Making it Your Own: Third Phase

I would learn lots of this stuff, again, note-for-note to the best of my ability. I was blessed with a fantastic ear and good sense of rhythm, and that along with practice and constant exercising of those talents made me able to pick out and play even some rather complex stuff in no time. By the time I was 14, I was able to play simple things like Led Zeppelin’s “The Lemon Song” in about an hour (not counting the solo), and most of the stuff on the radio I could begin playing along with accurately before the song was even over.

Even better, there were plenty of “solo acoustic” variants of songs, or songs where the live recording happened to pick up the guitar and drums more than the bass and keyboards, or it emphasized the bass and drums more than the guitar. These imperfections are absolute gems for learning more about what’s going in the song… or what *could* be going on when you play it!

Once I learn a song and work in some of the original artist’s own embellishments, some of my own embellishments, and some of my musical personality and influences, it more or less becomes “my own”. I can still play a variant faithful to the recording, but it seems boring when you’ve heard all that can be done with it.

What’s Better Now?

Well, it’s not like I used to pick a song, spend a weekend with it, and all of a sudden it was my own. While I could easily learn decently solid renditions of songs in a weekend, it would take me months to find different recordings of a particular song, to read magazine articles and sometimes books about guitarists, scouring the words for descriptions of how they played things and thought about them. It would be an ongoing process that sometimes also involved figuring out the players’ influences and tracking down *their* recordings… ugh.

Today?

*Literally* today, I decided to learn “Why Georgia” by John Mayer (I learned a decently solid version of “My Stupid Mouth” yesterday). While I was listening to the song in iTunes, I also checked out YouTube and searched for the song there. Boom! 5000 hits. In about an hour, I had heard (AND SEEN!!!) about 10 different performances by John Mayer himself, and I spent about another 20 minutes browsing videos others have posted either covering the song, or providing a tutorial on how to play the song!

Of course, some of the covers and tutorials are horribly wrong, but it’s easier to pick out stuff that’s not going to help you on YouTube than it is actually having to play through badly written tablature to figure out it’s wrong (and the internet, I promise you, is absolutely flooded with bad tablature).

The point is, I was able to learn this song, and work in Mayer’s embellishments as I went. Watching him play it in live performances, I was able to rely in visual cues and his body language to help me figure out how he thought about the song as he was playing it. Seeing him play in solo acoustic settings vs. live with a whole band was enlightening. In the end, I have learned a version of the piece, in one day, that would probably have taken me months to put together in the 80′s.

Thanks, Internet!