Category Archives: Web Services

Sending Alerts With Graphite Graphs From Nagios

Disclaimer

The way I’m doing this relies on a feature I wrote for Graphite that was only recently merged to trunk, so at time of writing that feature isn’t in a stable release. Hopefully it’ll be in 0.9.10. Until then, you can at least test this setup using Graphite’s trunk version.

Oh yeah, the new feature is the ability to send graph images (not links) via email. I surfaced this feature in Graphite through the graph menus that pop up when you click on a graph in Graphite, but implemented it such that it’s pretty easy to call from a script (which I also wrote – you’ll see if you read the post).

Also, note that I assume you already know Nagios, how to install new command scripts, and all that. It’s really easy to figure this stuff out in Nagios, and it’s well-documented elsewhere, so I don’t cover anything here but the configuration of this new feature.

The Idea

I’m not a huge fan of Nagios, to be honest. As far as I know, nobody really is. We all just use it because it’s there, and the alternatives are either overkill, unstable, too complex, or just don’t provide much value for all the extra overhead that comes with them (whether that’s config overhead, administrative overhead, processing overhead, or whatever depends on the specific alternative you’re looking at). So… Nagios it is.

One thing that *is* pretty nice about Nagios is that configuration is really dead simple. Another thing is that you can do pretty much whatever you want with it, and write code in any language you want to get things done. We’ll take advantage of these two features to actually do a couple of things:

  • Monitor a metric by polling Graphite for it directly
  • Tell Nagios to fire off a script that’ll go get the graph for the problematic metric, and send email with the graph embedded in it to the configured contacts.
  • Record that we sent the alert back in Graphite, so we can overlay those events on the corresponding metric graph and verify that alerts are going out when they should, that the outgoing alerts are hitting your phone without delay, etc.

The Candy

Just to be clear, we’re going to set things up so you can get alert messages from Nagios that look like this (click to enlarge):

And you’ll also be able to track those alert events in Graphite in graphs that look like this (click to enlarge, and note the vertical lines – those are the alert events.):

Defining Contacts

In production, it’s possible that the proper contacts and contact groups already exist. For testing (and maybe production) you might find that you want to limit who receives graphite graphs in email notifications. To test things out, I defined:

  • A new contact template that’s configured specifically to receive the graphite graphs. Without this, no graphs.
  • A new contact that uses the template
  • A new contact group containing said contact.

For testing, you can create a test contact in templates.cfg:

define contact{
        name                            graphite-contact 
        service_notification_period     24x7            
        host_notification_period        24x7 
        service_notification_options    w,u,c,r,f,s 
        host_notification_options       d,u,r,f,s  
        service_notification_commands   notify-svcgraph-by-email
        host_notification_commands      notify-host-by-email
        register                        0
        }

You’ll notice a few things here:

  • This is not a contact, only a template.
  • Any contact defined using this template will be notified of service issues with the command ‘notify-svcgraph-by-email’, which we’ll define in a moment.

In contacts.cfg, you can now define an individual contact that uses the graphite-contact template we just assembled:

define contact{
        contact_name    graphiteuser
        use             graphite-contact 
        alias           Graphite User
        email           someone@example.com 
        }

Of course, you’ll want to change the ‘email’ attribute here, even for testing.

Once done, you also want to have a contact group set up that contains this new ‘graphiteuser’, so that you can add users to the group to expand the testing, or evolve things into production. This is also done in contacts.cfg:

define contactgroup{
        contactgroup_name       graphiteadmins
        alias                   Graphite Administrators
        members                 graphiteuser
        }

Defining a Service

Also for testing, you can set up a test service, necessary in this case to bypass default settings that seek to not bombard contacts by sending an email for every single aberrant check. Since the end result of this test is to see an email, we want to get an email for every check where the values are in any way out of bounds. In templates.cfg put this:

define service{
    name                        test-service
    use                         generic-service
    passive_checks_enabled      0
    contact_groups              graphiteadmins
    check_interval              20
    retry_interval              2
    notification_options        w,u,c,r,f
    notification_interval       30
    first_notification_delay    0
    flap_detection_enabled      1
    max_check_attempts          2
    register                    0
    }

Again, the key point here is to insure that no notifications are ever silenced, deferred, or delayed by nagios in any way, for any reason. You probably don’t want this in production. The other point is that when you set up an alert for a service that uses ‘test-service’ in its definition, the alerts will go to our previously defined ‘graphiteadmins’.

To make use of this service, I’ve defined a service in ‘localhost.cfg’ that will require further explanation, but first let’s just look at the definition:

define service{
        use                             test-service 
        host_name                       localhost
        service_description             Some Important Metric
        check_command                   check_graphite_data!24!36
        notifications_enabled           1
        }

There are two new things we need to understand when looking at this definition:

  • What is ‘check_graphite_data’?
  • What is ‘_GRAPHURL’?

These questions are answered in the following section.

In addition, you should know that the value for _GRAPHURL is intended to come straight from the Graphite dashboard. Go to your dashboard, pick a graph of a single metric, grab the URL for the graph, and paste it in (and double-quote it).

Defining the ‘check_graphite_data’ Command

This command relies on a small script written by the folks at Etsy, which can be found on github: https://github.com/etsy/nagios_tools/blob/master/check_graphite_data

Here’s the commands.cfg definition for the command:

# 'check_graphite_data' command definition
define command{
        command_name    check_graphite_data
        command_line    $USER1$/check_graphite_data -u $_SERVICEGRAPHURL$ -w $ARG1$ -c $ARG2$
        }

The ‘command_line’ attribute calls the check_graphite_data script we got on github earlier. The ‘-u’ flag is a URL, and this is actually using the custom object attribute ‘_GRAPHURL’ from our service definition. You can see more about custom object variables here: http://nagios.sourceforge.net/docs/3_0/customobjectvars.html - the short story is that, since we defined _GRAPHURL in a service definition, it gets prepended with ‘SERVICE’, and the underscore in ‘_GRAPHURL’ moves to the front, giving you ‘$_SERVICEGRAPHURL’. More on how that works at the link provided.

The ‘-w’ and ‘-c’ flags to check_graphte_data are ‘warning’ and ‘critical’ thresholds, respectively, and they correlate to the positions of the service definition’s ‘check_command’ arguments (so, check_graphite_data!24!36 maps to ‘check_graphite_data -u <url> -w 24 -c 36′)

Defining the ‘notify-svcgraph-by-email’ Command

This command relies on a script that I wrote in Python called ‘sendgraph.py’, which also lives in github: https://gist.github.com/1902478

The script does two things:

  • It emails the graph that corresponds to the metric being checked by Nagios, and
  • It pings back to graphite to record the alert itself as an event, so you can define a graph for, say, ‘Apache Load’, and if you use this script to alert on that metric, you can also overlay the alert events on top of the ‘Apache Load’ graph, and vet that alerts are going out when you expect. It’s also a good test to see that you’re actually getting the alerts this script tries to send, and that they’re not being dropped or seriously delayed.

To make use of the script in nagios, lets define the command that actually sends the alert:

define command{
    command_name    notify-svcgraph-by-email
    command_line    /path/to/sendgraph.py -u "$_SERVICEGRAPHURL$" -t $CONTACTEMAIL$ -n "$SERVICEDESC$" -s $SERVICESTATE$
    }

A couple of quick notes:

  • Notice that you need to double-quote any variables in the ‘command_line’ that might contain spaces.
  • For a definition of the command line flags, see sendgraph.py’s –help output.
  • Just to close the loop, note that notify-svcgraph-by-email is the ‘service_notification_commands’ value in our initial contact template (the very first listing in this post)

Fire It Up

Fire up your Nagios daemon to take it for a spin. For testing, make sure you set the check_graphite_data thresholds to numbers that are pretty much guaranteed to trigger an alert when Graphite is polled. Hope this helps! If you have questions, first make sure you’re using Graphite’s ‘trunk’ branch, and not 0.9.9, and then give me a shout in the comments.

Thoughts on Python and Python Cookbook Recipes to Whet Your Appetite

Dave Beazley and myself are, at this point, waist deep into producing Python Cookbook 3rd Edition. We haven’t really taken the approach of going chapter by chapter, in order. Rather, we’ve hopped around to tackle chapters one or the other finds interesting or in-line with what either of us happens to be working with a lot currently.

For me, it’s testing (chapter 8, for those following along with the 2nd edition), and for Dave, well, I secretly think Dave touches every aspect of Python at least every two weeks whether he needs to or not. He’s just diabolical that way. He’s working on processes and threads at the moment, though (chapter 9 as luck would have it).

In both chapters (also a complete coincidence), we’ve decided to toss every scrap of content and start from scratch.

Why on Earth Would You Do That?

Consider this: when the last edition (2nd ed) of the Python Cookbook was released, it went up to Python 2.4. Here’s a woefully incomplete list of the superamazing awesomeness that didn’t even exist when the 2nd Edition was released:

  • Modules:
    • ElementTree
    • ctypes
    • sqlite3
    • functools
    • cProfile
    • spwd
    • uuid
    • hashlib
    • wsgiref
    • json
    • multiprocessing
    • fractions
    • plistlib
    • argparse
    • importlib
    • sysconfig
  • Other Stuff
    • The ‘with’ statement and context managers*
    • The ‘any’ and ‘all’ built-in functions
    • collections.defaultdict
    • advanced string formatting (the ‘format()’ method)
    • class decorators
    • collections.OrderedDict
    • collections.Counter
    • collections.namedtuple()
    • the ability to send data *into* a generator (yield as an expression)
    • heapq.merge()
    • itertools.combinations
    • itertools.permutations
    • operator.methodcaller()

* If memory serves, the ‘with’ statement was available in 2.4 via future import.

Again, woefully incomplete, and that’s only the stuff that’s in the 2.x version! I don’t even mention 3.x-only things like concurrent.futures. From this list alone, though, you can probably discern that the way we think about solving problems in Python, and what our code looks like these days, is fundamentally altered forever in comparison to the 2.4 days.

To give a little more perspective: Python core development moved from CVS to Subversion well after the 2nd edition of the book hit the shelves. They’re now on Mercurial. We skipped the entire Subversion era of Python development.

The addition of any() and all() to the language by themselves made at least 3-4 recipes in chapter 1 (strings) one-liners. I had to throw at least one recipe away because people just don’t need three recipes on how to use any() and all(). The idea that you have a chapter covering processes and threads without a multiprocessing module is just weird to think about these days. The with statement, context managers, class decorators, and enhanced generators have fundamentally changed how we think about certain operations.

Also something to consider: I haven’t mentioned a single third-party module! Mock, tox, and nosetests all support Python 3. At least Mock and tox didn’t exist in the old days (I don’t know about nose off-hand). Virtualenv and pip didn’t exist (both also support Python 3). So, not only has our code changed, but how we code, test, deploy, and generally do our jobs with Python has also changed.

Event-based frameworks aside from Twisted are not covered in the 2nd edition if they existed at all, and Twisted does not support Python 3.

WSGI, and all it brought with it, did not exist to my knowledge in the 2.4 days.

We need a Mindset List for Python programmers!

So, What’s Your Point

My point is that I suspect some people have been put off of submitting Python 3 recipes, because they don’t program in Python 3, and if you’re one of them, you need to know that there’s a lot of ground to cover between the 2nd and 3rd editions of the book. If you have a recipe that happens to be written in Python 2.6 using features of the language that didn’t exist in Python 2.4, submit it. You don’t even have to port it to Python 3 if you don’t want to or don’t have the time or aren’t interested or whatever.

Are You Desperate for Recipes or Something?

Well, not really. I mean, if you all want to wait around while Dave and I crank out recipe after recipe, the book will still kick ass, but it’ll take longer, and the book’s world view will be pretty much limited to how Dave and I see things. I think everyone loses if that’s the case. Having been an editor of a couple of different technical publications, I can say that my two favorite things about tech magazines are A) The timeliness of the articles (if Python Magazine were still around, we would’ve covered tox by now), and B) The broad perspective it offers by harvesting the wisdom and experiences of a vast sea of craftspeople.

What Other Areas Are In Need?

Network programming and system administration. For whatever reason, the 2nd edition’s view of system administration is stuff like checking your Windows sound system and spawning an editor from a script. I guess you can argue that these are tasks for a sysadmin, but it’s just not the meat of what sysadmins do for a living. I’ll admit to being frustrated by this because I spent some time searching around for Python 3-compatible modules for SNMP and LDAP and came up dry, but there’s still all of that sar data sitting around that nobody ever seems to use and is amazing, and is easy to parse with Python. There are also terminal logging scripts that would be good.

Web programming and fat client GUIs also need some love. The GUI recipes that don’t use tkinter mostly use wxPython, which isn’t Python 3-compatible. Web programming is CGI in the 2nd edition, along with RSS feed aggregation, Nevow, etc. I’d love to see someone write a stdlib-only recipe for posting an image to a web server, and then maybe more than one recipe on how to easily implement a server that accepts them.

Obviously, any recipes that solve a problem that others are likely to have that use any of the aforementioned modules & stuff that didn’t exist in the last edition would really rock.

How Do I Submit?

  1. Post the code and an explanation of the problem it solves somewhere on the internet, or send it (or a link to it) via email to PythonCookbook@oreilly.com or to @bkjones on Twitter.
  2. That’s it.

We’ll take care of the rest. “The rest” is basically us pinging O’Reilly, who will contact you to sign something that says it’s cool if we use your code in the book. You’ll be listed in the credits for that recipe, following the same pattern as previous editions. If it goes in relatively untouched, you’ll be the only name in the credits (also following the pattern of previous editions).

What Makes a Good Recipe?

A perfect recipe that is almost sure to make it into the cookbook would ideally meet most of the criteria set out in my earlier blog post on that very topic. Keep in mind that the ability to illustrate a language feature in code takes precedence over the eloquence of any surrounding prose.

What If…

I sort of doubt this will come up, but if we’ve already covered whatever is in your recipe, we’ll weigh that out based on the merits of the recipes. I want to say we’ll give new authors an edge in the decision, but for an authoritative work, a meritocracy seems the only valid methodology.

If you think you’re not a good writer, then write the code, and a 2-line description of the problem it solves, and a 2-line description of how it works. We’ll flesh out the text if need be.

If you just can’t think of a good recipe, grep your code tree(s) for just the import statements, and look for ideas by answering questions on Stackoverflow or the various mailing lists.

If you think whatever you’re doing with the language isn’t very cool, then stop thinking that a cookbook is about being cool. It’s about being practical, and showing programmers possibly less senior than yourself an approach to a problem that isn’t completely insane or covered in warts, even if the problem is relatively simple.

Slides, an App, a Meetup, and More On the Way

I’ve been busy. Seriously. Here’s a short dump of what I’ve been up to with links and stuff. Hopefully it’ll do until I can get back to my regular blogging routine.

PICC ’11 Slides Posted

I gave a Python talk at PICC ’11. If you were there, then you have a suboptimal version of the slides, both because I caught a few bugs, and also because they’re in a flattened, lifeless PDF file, which sort of mangles anything even slightly fancy. I’m not sure how much value you’ll get out of these because my presentation slides tend to present code that I then explain, and you won’t have the explanation, but people are asking, so here they are in all their glory. Enjoy!

I Made a Webapp Designed To Fail

No really, I did. WebStatusCodes is the product of necessity. I’m writing a Python module that provides an easy way for people to talk to a web API. I test my code, and for some of the tests I want to make sure my code reacts properly to certain HTTP errors (or in some cases, to *any* HTTP status code that’s not 200). In unit tests this isn’t hard, but when you’re starting to test the network layers and beyond, you need something on the network to provide the errors. That’s what WebStatusCodes does. It’s also a simple-but-handy reference for HTTP status codes, though it is incomplete (418 I’m a teapot is not supported). Still, worth checking out.

Interesting to note, this is my first AppEngine application, and I believe it took me 20 minutes to download the SDK, get something working, and get it deployed. It was like one of those ‘build a blog in -15 minutes’ moments. Empowering the speed at which you can create things on AppEngine, though I’d be slow to consider it for anything much more complex.

Systems and Devops People, Hack With Me!

I like systems-land, and a while back I was stuck writing some reporting code, which I really don’t like, so I started a side project to see just how much cool stuff I could do using the /proc filesystem and nothing but pure Python. I didn’t get too far because the reporting project ended and I jumped back into all kinds of other goodness, but there’s a github project called pyproc that’s just a single file with a few functions in it right now, and I’d like to see it grow, so fork it and send me pull requests. If you know Linux systems pretty well but are relatively new to Python, I’ll lend you a hand where I can, though time will be a little limited until the book is done (see further down).

The other projects I’m working on are sort of in pursuit of larger fish in the Devops waters, too, so be sure to check out the other projects I mention later in this post, and follow me on github.

Python Meetup Group in Princeton NJ

I started a Meetup group for Pythonistas that probably work in NYC or PA, but live in NJ. I work in PA, and before this group existed, the closest group was in Philly, an hour from home. I put my feelers out on Twitter, found some interest, put up a quick Meetup site, and we had 13 people at the first meetup (more than had RSVP’d). It’s a great group of folks, but more is always better, so check it out if you’re in the area. We hold meetings at the beautiful Princeton Public Library (who found us on twitter and now sponsors the group!), which is just a block or so from Triumph, the local microbrewery. I’m hoping to have a post-meeting impromptu happy hour there at some point.

Python Cookbook Progress

The Python Cookbook continues its march toward production. Lots of work has been done, lots of lessons have been learned, lots of teeth have been gnashed. The book is gonna rock, though. I had the great pleasure of porting all of the existing recipes that are likely to be kept over to Python 3. Great fun. It’s really amazing to see just how it happens that a 20-line recipe is completely obviated by the addition of a single, simple language feature. It’s happened in almost every chapter I’ve looked at so far.

If you have a recipe, or stumble upon a good example of some language feature, module, or other useful tidbit, whether it runs in Python 3 or not, let me know (see ‘Contact Me’). The book is 100% Python 3, but I’ve gotten fairly adept at porting things over by now :) Send me your links, your code, or whatever. If we use the recipe, the author will be credited in the book, of course.

PyRabbit is Coming

In the next few days I’ll be releasing a Python module on github that will let you easily work with RabbitMQ servers using that product’s HTTP management API. It’s not nearly complete, which is why I’m releasing it. It does some cool stuff already, but I need another helper or two to add new features and help do some research into how RabbitMQ broker configuration affects JSON responses from the API. Follow me on github if you want to be the first to know when I get it released. You probably also want to follow myYearbook on github since that’s where I work, and I might release it through the myYearbook github organization (where we also release lots of other cool open source stuff).

Python Asynchronous AMQP Consumer Module

I’m also about 1/3 of the way through a project that lets you write AMQP consumers using the same basic model as you’d write a Tornado application: write your handler, import the server, link the two (like, one line of code), and call consume(). In fact, it uses the Tornado IOLoop, as well as Pika, which is an asynchronous AMQP module in Python (maintained by none other than my boss and myYearbook CTO,  @crad), which also happens to support the Tornado IOLoop directly.

Why Open Shop In California?

DISCLAIMER: I live on the East Coast, so these are perceptions and opinions that I don’t put forth as facts. I’m more asking a question to start a dialog than professing knowledge.

So, I just heard a report claiming that there are more IT jobs than techs to fill them in Southern California. Anyone who ever reads a tech job board and/or TechCrunch has also no doubt taken note that a vast majority of startups seem to be starting up there, and that there are just a metric asston of jobs there anyway.

This boggles my mind. This is a place with an extremely high cost of living, making labor more expensive. At the same time, aren’t there rolling power outages in CA? Does that not effect corporations or something? Do they just move their datacenters across the border to another state?

Between what I would think is an amazingly high labor cost and what I would think is an unfavorable place in terms of simple things like availability of power, I would think more places would look elsewhere for expansion or startups.

I live within spitting distance of at least 5 universities with engineering departments that I think would rate at the very least “solid”, many would rate better. I would guess that I could get to any Ivy League school in 6 hours or less, driving (3 are within an hour of my NJ home). MIT and Stevens are very good non-Ivy schools, and lots of other ones like Rutgers, NJIT, Penn State, NYU, and lots more are here, and those are just a few of the ones between NYC and Philadelphia, which are less than 2 hours apart. So…. there’s a labor pool here.

Is it tax breaks? Some aspect of the political atmosphere? Transportation? Is San Francisco such a clean, safe, friendly city that you just deal with the nonsense to live there?

What’s your take on this?

Head first into javascript (and jQuery)

So, I had to take a break from doing the Code Katas just as I was getting to the really cool one about Bloom Filters. The reason for the unexpected break from kata-ing was that I had a project thrown into my lap. I say “project” not because it was some huge thing that needed doing — lots of people reading this could probably have done it in a matter of a few hours — but because it involved two things I’ve never done any real work with: javascript, and jQuery.

My task? Well, first I had to recreate a page from a graphic designer’s mockup. So, take a JPEG image and create the CSS and stuff to make that happen. Already I’m out of my comfort zone, because historically I’m a back-end developer more comfortable with threading than CSS (95% of my code is written in Python and is daemonized/threaded/multiprocess/all-of-the-above or worse), but at least I’ve done enough CSS to get by.

Once the CSS was done, I was informed that I’d now need to take the tabular reporting table I just created and make it sort on any of the columns, get the data via AJAX calls to a flat file that would store the JSON for the dummy data, create nice drop-down date pickers so date ranges in the report could be chosen by the end user, page the data using a flickr-style pager so only 20 lines would show up on a page, and alternate the row colors in the table (and make sure that doesn’t break when you sort!).

How to learn javascript and/or jQuery REALLY fast

How exactly do you learn enough javascript and jQuery to get this done in a matter of a few days (counting from *after* the CSS part was done)? Here are some links you should keep handy if you have a situation like this arise:

  • If Douglas Crockford says it, listen. I’d advise you start here (part I of a 4-part intro to javascript). That site also has his ‘Advanced Javascript’ series. He also wrote a book, which is small enough to read quickly, and well done.
  • Packt has a lot of decent resources for jQuery. Specifically, this article helped me organize what I was doing in my head. The code itself had some rather glaring issues — you’re not going to cut-n-paste this code and deploy it to production, but coming from scorched earth, I really learned a lot.
  • After the project was already over I found this nice writeup that covers quick code snippets and demos illustrating some niceties like sliding panels and disappearing table rows and how to do them with jQuery.
  • jQuery itself has some pretty decent documentation for those times when your cut-n-pasted code looks a little suspect or you’re just sure there’s a better way. Easy to read and concise.

Why I Wrote My Own Sorting/Paging in jQuery

Inevitably, someone out there is wondering why I didn’t just use tablesorter and tablesorter.pager, or Flexigrid, or something like that. The answer, in a nutshell, is paging. Sorting and paging operations, I learned both by experience and in my reading, *NEED* to know about each other. If they don’t, you’ll get lots of weird results, like sorting on just one page (or, sorting on just one page until you click to another page, which will look as expected, and then click back), or pages with rows on them that are just plain wrong, or… the list goes on. This is precisely the problem that the integrated “all-sorting-all-paging” tools like tablesorter try to solve. The issue is that I could not find a SINGLE ONE that did not have a narrow definition of what a pager was, and what it looked like.

I wanted (well, I was required to mimic the mockup, so “needed”) a flickr-style pager — modified. I needed to have each page of the report represented at the bottom of the report table by a block with the proper number in the block. The block would be clickable, and clicking it would show the corresponding page of data. This is more or less what Flickr does, but I didn’t need the “previous” and “next” buttons, and I didn’t need the “…” they use (rather effectively) to cut down on the number of required pager elements. So… just some blocks with page numbers. That’s it.

I started out using tablesorter for jQuery, and it worked great — it does the sorting for you, manages the alternating row colors, and is a pretty flexible sorter. Then I got to the paging part, and things went South pretty fast. While tablesorter.pager has a ‘moveToPage’ function, it’s not exposed so you can bind it to a CSS selector like the ‘moveToPrevious’, ‘moveToLast’, ‘moveToNext’ and other functions are. So, I tried to hack it into the pager code myself. I got weird results (though I feel more confident about approaching that now than I did even three days ago). There wasn’t any obvious way to do anything but give the user *only* first/last/previous/next buttons to control the paging. I moved on. I googled, I asked on jQuery IRC, I even wrote the developer of tablesorter. I got nothing.

I looked at 4 or 5 different tools, and was shocked to find the same thing! I didn’t go digging into the code of all of them, but their documentation all seemed to be in some kind of weird denial about the existence of flickr-style paging altogether!

So, I wrote my own. It wasn’t all that difficult, really. The code that worked was only slightly different from the code I’d fought with early on in the process. It just took some reading to get some of the basic tricks of the trade under my belt, and I got a tip or two from one of the gurus at work as well, and I was off to the races!

Lessons Learned

So, one thing I have to say for my boss is that he knows better than to throw *all* of those things at me at once. Had he come to me and said he wanted an uber-ajaxian reporting interface from outer space from the get-go, I might not have responded even as positively as I did (and I would rate my response as ‘tepid, but attempting a positive outlook’) . It’s best to draw me in slowly, a task at a time, so I can get some sense of accomplishment and some feedback along the way instead of feeling like I still have this mountain to climb before it’s over.

I certainly learned that this javascript and jQuery (and AJAX) stuff isn’t really black magic. Once you get your hands dirty with it it’s kinda fun. I still don’t ever want to become a front end developer on a full-time basis (browser testing is where I *really* have zero patience, either for myself or the browsers), but this experience will serve me well in making my own projects a little prettier and slicker, and nicer to use. It’ll also help me understand more about what the front end folks are dealing with, since there’s tons of javascript at myYearbook.

So, I hope this post keeps some back end scalability engineer’s face from turning white when they’re given a front-end AJAX project to do. If you’ve ever had a similar situation happen to you (not necessarily related to javascript, but other technologies you didn’t know until you were thrown into a project), let’s hear the war stories!!

Python, Creating XML, Recursion, and Order

I love being challenged every day. Today I ran across a challenge that has several solutions. However, most of them are hacks, and I never feel like I really solved the problem when I implement a hack. There’s also that eerie feeling that this hack is going to bite me later.

I have an API I need to talk to. It’s pure XML-over-HTTP (henceforth XHTTP). Actually, the first issue I had was just dealing with XHTTP. All of the APIs I’ve dealt with in the past were either XMLRPC, SOAP, which have ready-made libraries ready to use, or they used GET parameters and the like, which are pretty simple to deal with. I’ve never had to actually construct an HTTP request and just POST raw XML.

It’s as easy as it should be, really. I code in Python, so once you actually have an XML message ready to send, you can use urllib2 to send it in about 2 lines of code.

The more interesting part is putting the request together. I thought I had this beat. I decided to use the xml.dom.minidom module and create my Document object by going through the DOMImplementation object, because it was the only thing I found that allowed me to actually specify a DTD programmatically instead of hackishly tacking on hard-coded text. No big deal.

Now that I had a document, I needed to add elements to it. Some of these XML queries I’m sending can get too long to write a function that manually creates all the elements, then adds them to the document, then creates the text nodes, then adds them to the proper element… it’s amazingly tedious to do. I really wish Python had something like PHP’s XMLWriter, which lets you create an element and its text in one line of code.

Tedium drives me nuts, so rather than code this all out, I decided to create a dictionary that mirrored the structure of my query, with the data for the text nodes filled in by variables.

query_params = {'super_special_query':
                   {
                      'credentials': {'username': user, 'password': password, 'data_realm': realm},
                      'result_params': {'num_results': setsize, 'order_by': order_by},
                       query_type: query_dict
                    }
                }

def makeDoc():
    impl = getDOMImplementation()
    dt = impl.createDocumentType("super_special", None, 'super_special.dtd')
    doc = impl.createDocument(None, "super_special", dt)
    return doc

def makeQuery(doc, query_params, tag=None):
    """
        @doc is an xml.minidom.Document object
        @query_params is a dictionary structure that mirrors the structure of the xml.
        @tag used in recursion to keep track of the node to append things to next time through.

    """

    if tag is None:
        root = doc.documentElement
    else:
        root = tag

    for key, value in query_params.iteritems():
        tag = doc.createElement(key)
        root.appendChild(tag)
        if isinstance(value, dict):
            makeQuery(doc, value, tag)
        else:
            root.appendChild(tag)
            tag_txt = doc.createTextNode(value)
            tag.appendChild(tag_txt)

    return doc.toxml()

doc = makeDoc()
qxml = makeQuery(doc, query_params)

This is simplistic, really. I don’t need to deal with attributes in my queries, for example. But it is generic enough that if I need to send different types of queries, all that’s required is creating another dictionary to represent it, and passing that through the same makeQuery function to create the query.

Initial testing indicated success, but that’s why you can’t rely on only simple initial tests. Switching things up immediately exposed a problem: The API server validated my query against a DTD that enforced strict ordering of the elements, and Python dictionaries do not have the same notion of “order” that you and I do.

So there’s the code. If nothing else, it’s a less-contrived example of what you might actually use recursion for. Tomorrow I have to figure out how to enforce the ordering. One idea is to have a separate list to consult for the ordering of the elements. It requires an extra outer loop to go through the list, get the name of the next tag, then use that value to ask the dictionary for any related values. Seemed like a good way to go, but I had a bit of difficulty figuring out how to make that work at all. Maybe with fresh eyes in the AM it’ll be more obvious — that happens a lot, and is always a nice surprise.

Ideas, as always, are hereby solicited!

Lessons Learned While Creating a Generic Taxonomy App for Django

So, when I first picked up a guitar, the first song I sat down to learn, by ear, was Stairway to Heaven, not “Twinkle, Twinkle, Little Star”. So goes my experience with Django :)

The Background

I was humming along on my recreation of LinuxLaboratory.org. I got a simple blog in place in just a couple of days, a code-sharing app in place a few days later (if that), and a very simple CMS I threw together using flatpages. A good bit of the base code I used came from the 2nd edition of “Practical Django Projects”, but I soon veered off in other directions, and started analyzing the work I’d already done a bit more closely.

One of the things that was glaringly obvious to me was that my method of classifying content was a little schizophrenic. I had three separate apps to represent different types of content, which is great, but each separate app had its own “Category” model. Yuck. On top of that, I was using django-tagging to enable tagging in addition to the categorization each app supported.

The Problem

So… for one type of classification (Categories), it’s built into the specific application, and for the other (tagging), it’s not built in, but it’s pretty tightly coupled. There are a few fundamental drawbacks to this approach:

First, you have to make a pretty big commitment to these things. The easiest way to implement them in your app is to add support for them at the outset, because adding them in later is going to be a bit of a headache. Categories aren’t quite so bad — I implemented them the way the book does, which is with a ManyToMany field. In Django, when you create a ManyToMany field in a model, there’s no corresponding field in that model’s table in the database. Instead, Django creates a lookup table for you, which is nice, because it means you *could* add categories at a later time without *too* much trouble. Tags use the django-tagging app, which implements tags as a multi-valued field in the database table representing the model that will use the tags. So adding this in later is a little bit more of a hassle.

The second issue is that this approach doesn’t treat classification in a consistent manner. One is in the app, the other is a separate app, one is a field, the other is a model, one affects the model’s table, the other doesn’t, etc. One place where this inconsistency becomes obvious is in your templates, where you’re likely to want to give users the ability to browse by category, or browse by tag. Browsing by category across all the different content types is going to be pretty tough if they all have their own implementation. Tags are a little easier, but it’s still a little cumbersome.

The third issue is specific to Categories, and has to do with maintenance: if I come up with some fantastic idea for the Category implementation (like, I dunno, subcategories?), I have to implement it separately in all of the apps that are using categories. No Bueno™.

The Dream

Wouldn’t it be nice if you could just say “give me a list of the taxonomy types used to classify this piece of content, whatever it is, whichever app it comes from, and also a list of the taxonomy terms involved”? Wouldn’t it be nice if you could just add support for categories and tags using a single app that doesn’t add anything to your existing tables? Wouldn’t it be nice to be able to come up with your own taxonomies and, perhaps, hierarchical taxonomies and more complex relationships? Wouldn’t it be nice to be able to say “here’s a category name, show me all of the content associated with it, sorted by content type” and then change your mind and say “no wait, show it to me sorted by title, with a content type indicator over here”, and then change your mind again and say “er, how about showing the taxonomy type, then the term, then all of the content objects under that type:term pairing”?

The Solution

I think it would, so I started creating this beast that I just call “taxonomy”. Right now it’s pretty simplistic, and it’ll likely change slightly based on some things I’ve learned, but surprisingly, I think my first shot at it is really darn close! I’ve stopped being surprised at how quickly I’m able to prototype in Django: getting this together, including the creation of the models, getting it into the admin interface, and getting it linked with any random content type from any app that wants to use it within the admin interface (to add taxonomies and labels to a piece of content in the ‘edit’ interface) took probably 4 hours, including time to read documentation and fall down a few times.

The admin interface for taxonomy lets you create a taxonomy, so if none of your apps currently don’t support the notion of a “Category”, you can go create a taxonomy called “Category”. Once that’s there, you can create a “taxonomy term”, where you’d select the “type” for this term (your new Category), and then a term. So if your term was “Django”, then you would have just created a category called “Django” that could be used by any other app/model in your project. The same, of course, would go for tags, and whatever other classification devices you want.

There’s support for parent-child relationships at the taxonomy term level (so you can have subcategories, or even subtags if you want, etc. I guess you could even categorize tags, and tag categories! They’re coming to take me away!!!). I haven’t given much thought to having hierarchical relationships for the taxonomies themselves. That would be a little overboard, no? I’m interested to hear realistic use cases for that :)

Once you’ve created a taxonomy and a term, the next thing to do is figure out how to associate your actual content to it. So, if our taxonomy is “Category” and the category name (the “term”) is “Django”, the way I’ve implemented it is that you’d go into the edit interface for the article, and a form for associating it with your category appears. This was created using a GenericInlineModelAdmin, which was a gem of a find in the documentation. Inlines let you easily create a form to update a piece of content using concepts and attributes from other models, and even other applications. If you don’t know much about Django, this sounds like a big mess, but in reality, it’s fairly elegant.

I’ve done some testing to see that I can pull things out of the database and associate things properly in the presentation layer, but I’d like to work on making it smoother before I go releasing code or anything like that…. which reminds me that I *did* look and ask around about an app that maybe already did this and came up dry. If anyone sees this and says “why not just use x”, let me know, because it’s not really a goal to write code for the sake of coding. I actually thought this was an interesting feature and couldn’t find it.

Lessons Learned #1: URLConf is a Choice, Not a Requirement

First, I learned that it’s completely possible to create an application that doesn’t have a URLConf at all. Currently, taxonomies actually work in testing, and there’s no URLConf. There actually *will* be one when I figure out how I want the data to be used on my own site, and how to enable users to do whatever they want with it as well. One thought, for example, is that it would be really awesome to be able to go to “/categories/django” and have my app somehow “just know” that “categories”, when singularized, is “category”, which is a taxonomy. From there, the taxonomy app takes over, and magic happens. I have faith that I can make this happen without having the word “taxonomy” in the url. We’ll see.

Anyway, the point is that you don’t have to have a URLconf, and that hadn’t really occurred to me. For the record, django-tagging also doesn’t have a URLconf.

Lessons Learned #2: ContentTypes Let Your Models Be All-Knowing

The second thing I learned was that, using the ContentTypes framework within Django, it’s possible to create a model that will deal with data, and relationships to data, in a dynamic way, such that you don’t have to know what type of data your models will be working with at the time you create them.

For example, my taxonomy app can be used with my blog’s “Entry” model, my code-sharing app’s “Snippet” model, and my CMSs “Page” model. If I pass the app to you, you can use it for your news site’s “Story” model, your ad network’s “Ad” model, and your Twitter clone’s “Tweet” model. No problem. This is in the docs, but here’s what I’ve done:

class Taxonomy(models.Model):
 """A facility for creating custom content classification types"""
 type = models.CharField(max_length=50, unique=True)

class TaxonomyTerm(models.Model):
 """Terms are associated with a specific Taxonomy, and should be generically usable with any contenttype"""
 type = models.ForeignKey(Taxonomy)
 term = models.CharField(max_length=50)
 parent = models.ForeignKey('self', null=True,blank=True)

class TaxonomyMap(models.Model):
 """Mappings between content and any taxonomy types/terms used to classify it"""
 term        = models.ForeignKey(TaxonomyTerm, db_index=True)
 type        = models.ForeignKey(Taxonomy, db_index=True)
 content_type = models.ForeignKey(ContentType, verbose_name='content type', db_index=True)
 object_id      = models.PositiveIntegerField(db_index=True)   
 object         = generic.GenericForeignKey('content_type', 'object_id')

Note that I’ve removed some stuff from the model defs — what you see here are just the fields, which are the relevant bit for what I’m explaining.

The TaxonomyMap model (a model is a class definition, by the way) has foreign keys to map to a human readable ‘term’ and ‘type’ in the other models. TaxonomyMap is just to store mappings between content objects and taxonomies (lower-level details of this might change to make it cleaner/more efficient – I know it’s not perfect). So, how does my app know that I’m storing a mapping to an “Entry” from my blog app? How does it get the id for that Entry? What’s going on?

Well, Django stores a list of every content type used by Django and any installed apps, and I’ve made a foreign key to ContentType so I can access the content type of the object that’s being dealt with and get its ID. I also have a “GenericForeignKey” field, which essentially creates a “dynamic” foreign key to the table that represents the object that’s being dealt with, so if I’m dealing with an “Entry” object from “monk” (which is the name of my blog app), then the foreign key will point to “monk_entry”, which is the table that stores my blog entries. When you create a taxonomy, and a term, and associate them to a piece of content, the resulting rows in the affected tables look like this:

mysql> select * from taxonomy_taxonomy;
+----+--------------+
| id | type         |
+----+--------------+
|  1 | TestCategory |
+----+--------------+
1 row in set (0.01 sec)

mysql> select * from taxonomy_taxonomyterm;
+----+---------+----------+-----------+
| id | type_id | term     | parent_id |
+----+---------+----------+-----------+
|  1 |       1 | TestTerm |      NULL |
+----+---------+----------+-----------+
1 row in set (0.00 sec)

mysql> select * from taxonomy_taxonomymap;
+----+---------+---------+-----------------+-----------+
| id | term_id | type_id | content_type_id | object_id |
+----+---------+---------+-----------------+-----------+
|  1 |       1 |       1 |              10 |         2 |
+----+---------+---------+-----------------+-----------+
1 row in set (0.00 sec)

Note that the table for the model that’s using the taxonomy app is untouched. Only taxonomy tables are used.

Seeing this, you might think that it’d be hard to put a form in the admin interface for arbitrary content types to classify them with taxonomies. Not so — which brings me to more lessons I learned.

Lessons Learned #3: Collecting Data About a Model Without Extending the Model and Creating Database Badness

If you have a model (we’ll use “Entry” again), and it has a core set of attributes, but you want to associate data with instances of this model not represented in the model definition (like, say, a taxonomy, for an arbitrary example), you can add a form to the admin interface for that model in about 5 minutes. This rocks, for those who didn’t know, because the alternative would either involve really ugly code, or really ugly data (you’d have to store the taxonomies in the table for the model, creating either tons of duplicate data, or multi-valued fields… and you’d still have duplicate data).

Typically, it seems that the normal use case for this is to relate models in the admin interface that are part of the same application and are explicitly related through a direct foreign key reference. This might even be enforced in the case of “InlineModelAdmin” objects, but I haven’t dealt with those personally. However, while reading about “GenericInlineModelAdmin” objects, it occurred to me that it shouldn’t matter that the related items are from different apps. I tried it, and it worked. Here’s what I did:

from django.contrib import admin
from django.contrib.contenttypes import generic
from monk.models import Entry
from taxonomy.models import TaxonomyMap

class TaxonomyMapInline(generic.GenericTabularInline):
   model = TaxonomyMap

class EntryAdmin(admin.ModelAdmin):
   prepopulated_fields = { 'slug': ['title'] }
   inlines = [ TaxonomyMapInline, ]

admin.site.register(Entry, EntryAdmin)

Again, I’ve edited out the irrelevant bits. The above comes from my blog app’s admin.py file. What I did was created an “inline” called “TaxonomyMapInline”, and then associated that inline with the “EntryAdmin” ModelAdmin object using ModelAdmin’s ‘inlines’ attribute, which takes a Python list, which means you can keep adding more inlines all day long if you like.

The result is that, when I go to edit a blog entry, there’s now a form at the bottom that lets the user select a taxonomy type and term (i.e. “Category” “Django”), and associate it with the post. When I added the inline to the admin.py file, it was a test to see what would happen. Since TaxonomyMap doesn’t hold anything but numeric IDs, I assumed I would have to go back and manually map the IDs to human readable values. Not true. Apparently, if the field being presented in the admin form maps to a ForeignKey field, Django automagically does the lookup for you and presents the human-readable text! And, when you save, it converts everything back to numeric IDs before going to the database, so everything “just works”. So the work I thought I’d be doing myself was already done for me!

LinuxLaboratory => Django – Now with Comments, Akismet, and 404.html

I stayed up a little late last night, because I wanted to get comments working on Linuxlaboratory.org. Comments *are* working, and I *do* get email notification of new comments, which is nice. Akismet is installed, and is no longer throwing errors, but to be honest, I’m not quite sure how to test it. I tried creating a comment that looked like spam, but it got posted publicly anyway. I’ll have to figure out how to log the Akismet processing operations and results.

I also set up email notification not just for comments, but for errors. And once I did that, I started getting slammed with emails. There’s a setting you can use to avoid being emailed for 404 (File Not Found) errors, but the problem I had was that, for every 404 error, there was also a TemplateNotFound error, because there was no 404.html file anywhere. So I created a 404.html file. It extends my base template, so it looks like the rest of the site, complete with the sidebar and stuff, and just has a note letting the user know what happened. This is nice because it’s an opportunity to rescue the user’s session by providing lots of links back to the home page, or to other content on the site. I haven’t gotten to it in the book yet (for those just tuning in, the site is being built in part using the 2nd Edition of “Practical Django Projects”).

Next up is RSS feeds. Once that works I might break from the book, because in the course of looking around and setting things up, I’m discovering that there’s a good bit of functionality I’d like to use, more apps I’d like to plug in, and a couple I’d like to write. I’d also like to put a horizontal navigation bar and make an “Articles” section of the site using a CMS, and have it be separate from the “Blog” part of the site. Then I’ll make a “Tools” section and start coding my own versions of tools like subnet calculators and a “CLI Hacks” application which will let me post one-liners I’ve found useful (kinda like commandlinefu.com).

Who knows what else — but first things first. RSS!

LinuxLaboratory woes, Drupal -> Django?

Ugh…

So, today I tried browsing to one of my sites, linuxlaboratory.org, and found a 403 “Forbidden” error. Calling support, they said it was a “billing issue”. Well, I pay my bills, and I haven’t received any new credit cards, so I’m not sure what that’s about. Further, they haven’t contacted me in any way shape or form at all in a very long time, and I’ve had the same email addresses for years now. Last time they failed to contact me, it was because they were sending all of the mail to “root@localhost” on the web server.

What’s more, the tech support guy, having determined that this wasn’t a technical but an administrative problem, transferred me to a sales person who was not there. I left a message. That was 3 hours ago. So I took matters into my own hands and changed the name server records to my webfaction account, and linuxlaboratory.org now points to an old test version of the site that uses Drupal.

It’s Over Between Us…

Drupal holds the record for the CMS that has run LinuxLaboratory the longest. Since its launch in 2001, LinuxLaboratory has used all of the major, and some of the minor open source PHP CMSes. Drupal gave me something very close to what I wanted, out of the box. Nowadays, Drupal is even nicer since they redid some of the back end APIs and attracted theme and module developers to the project. I’ve even done some coding in Drupal myself, and have to say that it really is a breeze.

But the problem is this: I’m a consultant, trainer, and author/editor. I am an experienced system admin, database admin, and infrastructure architect who makes a living solving other peoples’ problems. I really can’t afford to have something that is super high overhead to maintain running my sites. With Drupal releasing new versions with major security fixes once per month on average, and no automated update mechanism (and no built-in automated backup either), it becomes pretty cumbersome just to keep it updated.

This is in addition to my experiences trying to do e-commerce with Drupal. I tried to use one plugin, but soon found myself in dependency hell — a situation I’m not used to being in unless I’m on a command line somewhere. So, out with Drupal. I know it well and I’m sure I’ll find a use for it somewhere in my travels, but not now, and not for this.

Is Django the Future of LinuxLaboratory?

So I’m thinking of giving Django another shot. In fact, I thought I might try something new and interesting. Maybe I’ll build my Django app right in front of everyone, so that anyone who is interested can follow along, and so people can give me feedback and tips along the way. It also lets me share with people who have questions about a feature I’m implementing or something like that.

For fanboys of <insert technology here>, know this: I’m a technology whore. I consume technology like some people consume oxygen. I love technology, and I get on kicks, and every now and then, a “kick” turns into a more permanent part of my tool chest. Python is one such example. I’ve done lots with Python, but have never really made friends with it for web development. I got a webfaction account specifically because they support Python (and Django). I’ve done nothing with it. Now I think I might.

But not to worry! I own lots of domains that are sitting idle right now, and I’m considering doing a Ruby on Rails app for one of them, and I’m dying to do more with Lua. There’s only so much time!

Webfaction Django Users: Advice Hereby Solicited

So if you’re a webfaction customer using Django, please share your tips with me about the best way to deploy it. I’ve used nothing but PHP apps so far, and found that rather than use the one-click installs webfaction provides, it’s a lot easier to just choose the generic “CGI/PHP” app type and install the code myself. This allows me to, for example, install and update wordpress using SVN. Is Django a similar story, or does webfaction actually have an auto-upgrade mechanism for this? How are you keeping Django up to date?

Thanks!

I’m Offering Pro-Bono Consulting

I started my company about a year ago, but I’ve been doing consulting for a long time. In fact, my first job in the IT industry was working for a consulting firm. Before that, starting as far back as grade school, I was involved in a lot of volunteer civic and community service activities. I admire companies who get involved in their communities, or even outside of their communities, wherever help is needed.

As part of my business plan, I’ve put in place a policy of accepting one pro-bono consulting project per year. So far, I haven’t gotten any requests for free consulting work, so here’s my public shout out to let you know what types of services are available:

1. Speaking or Training. My specialties are things like advanced Linux administration and SQL, but I’m perfectly capable of delivering content for people who just need to know how the internet works, or want to know more about social media.Training, funny enough, has been the bulk of my business for the past year.

2. I can help with MySQL performance tuning on *nix systems, including finding hotspots related to the design of the database itself, or how your application code interacts with the database. If it happens that your MySQL server is performing poorly due to an underpowered system, I can also pinpoint which resource is dragging on the performance of your database.

3. If you just need random scripts written to perform *nix system administration tasks, I can consult with you about the requirements and write them for you. Note that while I can script in several languages, my preference for anything longer than 40 lines of code is Python.

4. I can build PC’s, install networks, set up firewalls and wireless routers, and all of the normal “office IT” functions, but note that my consulting is Linux consulting. I don’t work with Windows (well, I do, but not for free) ;-)

5. If there’s some other thing you’ve seen me blog about here, chances are I’ll be willing to perform a pro-bono consulting engagement to do it for you, or show you how to approach a problem, a large project, a migration, automation, monitoring, security or whatever.

Unless you happen to live within commuting distance to Princeton, NJ, work will be done remotely :)

Please email your request to jonesy at owladvisors dot com. Include your organization’s name, your contact info, and as much detail about the project and what your organization does as possible. The decision of which project to take on will be based solely on the information in your request!