Category Archives: Testing

What I’ve Been Up To

Historically, I post fairly regularly on this blog, but I haven’t been lately. It’s not for lack of anything to write about, but rather a lack of time to devote to blogging. I want to post at greater length about some of the stuff I’ve been doing, and I have several draft posts, but I wanted to list what I’ve been up to for two reasons:

  1. I use my blog as a sort of informal record of what I accomplished over the course of the year, hurdles I ran into, etc. I also sometimes use it to start a dialog about something, or to ‘think out loud’ about ideas. So it’s partially for my own reference.
  2. Someone might actually be interested in something I’m doing and want to pitch in, fork a repo, point me at an existing tool I’m reinventing, give me advice, or actually make use of something I’ve done or something I’ve learned and can share.

PyCon 2013

I’m participating again for the third year in the Program Committee and anywhere else I can help out (time permitting) with the organization of PyCon. It’s historically been a fantastic and really rewarding experience that I highly (and sometimes loudly) recommend to anyone who will listen. Some day I want to post at greater length about some actual instances where being really engaged in the community has been a win in real, practical ways. For now, you’ll have to take my word for it. It’s awesome.

I also hope to submit a talk for PyCon 2013. Anyone considering doing this should know a couple of things about it:

  1. Even though I participate in the Program Committee, which is a completely volunteer committee that takes on the somewhat grueling process of selecting the talks, tutorials, and poster sessions, it’s pretty much unrelated to my chances of having my talk accepted. In other words, submitting a talk is as daunting for me as it is for anyone. Maybe more so.
  2. Giving a talk was a really rewarding experience, and I recommend to anyone to give it a shot.

I just published a really long post about submitting a talk to PyCon. It’s full of completely unsolicited advice and subjective opinions about the do’s and don’ts of talk submission, based on my experiences as both a submitter of proposals and a member of the Program Committee, which is the committee that selects the talks.

Python Cookbook

Dave Beazley and I are really rolling with the next edition of the Python Cookbook, which will cover Python 3 *only*. We had some initial drama with it, but the good news is that I feel that Dave and I have shared a common vision for the book since just about day one, and that shared vision hasn’t changed, and O’Reilly hasn’t forced our hand to change it, which means the book should be a really good reflection of that vision when it’s actually released. I should note, however, that the next edition will represent a pretty dramatic departure from the form and function of previous versions. I’m excited for everyone to see it, but that’s going to have to wait for a bit. It’s still early to talk about an exact release date – I won’t know that for sure until the fall, but I would expect it to be at PyCon 2013.

PyRabbit

I’ve blogged a bit about pyrabbit before: it’s a Python client for talking to RabbitMQ’s RESTful HTTP Management API. So, it’s not for building applications that do message passing with AMQP — it’d be more for monitoring, polling queue depths, etc., or if you wanted to build your own version of the browser-based management interface to RabbitMQ.

Pyrabbit is actually being used. Last I looked, kombu was actually using it, and if memory serves, kombu is used in Celery, so pyrabbit is probably installed on more machines than I’m aware of at this point. I also created a little command shell program called bunnyq that will let you poke at RabbitMQ remotely without having to write any code. You can create & destroy most resources, fire off a message, retrieve messages, etc. It’s rudimentary, but it’s fine for quick, simple tests or to validate your understanding of the system given certain binding types, etc.

I have a branch wherein I port the unit tests for Pyrabbit to use a bit of a different approach, but I also need to flesh out more parts of the API and test it on more versions of RabbitMQ. If you use Pyrabbit, you should know that I also accept pull requests if they come with tests.

Stealth Mode

Well, ‘stealth’ is a strong word. I actually don’t believe much in stealth mode, so if you want to know just ask me in person. Anyway, between 2008 and 2012 I’ve been involved in startups (both bought out, by the way! East Coast FTW!) that were very product driven and very focused on execution. I was lucky enough to answer directly to the CEO of one of those companies (AddThis) and directly to the CTO of the other (myYearbook, now meetme.com), which gave me a lot of access and insight into the mechanics, process, and thinking behind how a product actually comes to be. It turns out I really love certain aspects of it that aren’t even necessarily technical. I also really find the execution phase really exciting, and the rollout phase really almost overwhelmingly exciting.

I’ve found myself now with an idea that is really small and simple, but just won’t go away. It’s kind of gnawing at me, and the more I think about it, the more I think that, given what I’ve learned about product development, business-side product metrics, transforming some stories into an execution plan, etc., on top of my experience with software development, architecting for scalability, cloud services, tools/technologies for building distributed systems, etc., I could actually do this. It’s small and simple enough for me to get a prototype working on my own, and awesome enough to be an actual, viable product. So I’m doing it. I’m doing it too slowly, but I’m doing it.

By the way, the one thing I completely suck at is front end design/development. I can do it, but if I could bring on a technical co-founder of my own choosing, that person would be a front end developer who has pretty solid design chops. If you know someone, or are someone, get in touch – I’m @bkjones on Twitter, and bkjones at gmail. I’m jonesy on freenode. I’m not hard to find :)

In and Out of Love w/ NoSQL

I’ve recently added Riak to my toolbelt, next to CouchDB, MongoDB, and Redis (primarily). I was originally thinking that Riak would be a good fit for a project I’m working on, but have grown more uncomfortable with that notion as time has passed. The fact of the matter is that my data has relationships, and it turns out that relational databases are actually a really good fit in terms of the built-in feature set. The only place they really stink is on the operations side, but it also turns out that I have, like, several years of experience in doing that! Where I finally lost patience with NoSQL for this project was this huge contradiction that I never hear anyone ever talk about. You know the one — the one where the NoSQL crowd screams about how flexible everything is and how it really fits in with the “agile” mindset, and then in another doc in the same wiki strongly drives home the message that if you aren’t 100% sure what the needs of your app are, you should really make sure you have a grasp on that up front.

Uhh, excuse me, but if I’m iterating quickly on an app, testing in production, iterating on what works, failing fast, and designing in the direction of, and in direct response to, my customers, HOW THE HELL DO I KNOW WHAT MY APP’S NEEDS ARE?

So, what I’m starting with is what I know for sure: my data has relationships. I’m experienced enough with NoSQL solutions to understand my options for modeling them & enforcing the relationships, but when the relational aspect of the data is pretty much always staring you in the face and isn’t limited to a small subset of operations that rely on the relationships, it seems like a no-brainer to just use the relational database and spend my time writing code to implement actual features. If I find some aspect of the data that can benefit from a NoSQL solution later, well, then I’ll use it later!

Unit Testing Patterns

Most who know me know I’m kind of “into” unit testing. It’s almost like a craft unto itself, and one that I rather enjoy. I recently started a new job at AWeber Communications, where I’m working on a next-generation awesome platform to the stars 2.0 ++, and it’s all agile, TDD, kanban, and all that. It’s pretty cool. What I found in the unit tests they had when I got there were two main things:

First, the project used bare asserts, and used dingus in “mock it all!” mode. Combined, this led to tests that were mostly effective, but not very communicative in the event of failure, and they were somewhat difficult to reason about when you read them.

Second, they had a pretty cool pattern for structuring and naming that gave *running* the tests and viewing the output a more behavioral feel to them that I thought was pretty cool, and looked vaguely familiar. Later I realized it was familiar because it was similar to the very early “Introducing Behavioral Driven Development” post I saw a long time ago but never did anything with. If memory serves, that early introduction did not introduce a BDD framework like the ones popping up all over github over the past few years. It mostly relied on naming to relay the meaning, and used standard tools “behind the curtain”, and it was pretty effective.

So long story short, those tests have mostly been ported to use the mock module, and inherit from unittest2.TestCase (so, no more bare asserts). The failure output is much more useful, and I think the pattern that’s evolving around the tests now is unfinished but starting to look pretty cool! In the process, I also created a repository for unittest helpers that currently only contains a context manager that you feed a list of things to patch, and it’ll automatically patch and then unpatch things after the code under test is run. It has helped me start to think about building tests in a more consistent fashion, which means reading them is more predictable too, and hopefully we spend less time debugging them and less time introducing new developers to how they work.

Slides, an App, a Meetup, and More On the Way

I’ve been busy. Seriously. Here’s a short dump of what I’ve been up to with links and stuff. Hopefully it’ll do until I can get back to my regular blogging routine.

PICC ’11 Slides Posted

I gave a Python talk at PICC ’11. If you were there, then you have a suboptimal version of the slides, both because I caught a few bugs, and also because they’re in a flattened, lifeless PDF file, which sort of mangles anything even slightly fancy. I’m not sure how much value you’ll get out of these because my presentation slides tend to present code that I then explain, and you won’t have the explanation, but people are asking, so here they are in all their glory. Enjoy!

I Made a Webapp Designed To Fail

No really, I did. WebStatusCodes is the product of necessity. I’m writing a Python module that provides an easy way for people to talk to a web API. I test my code, and for some of the tests I want to make sure my code reacts properly to certain HTTP errors (or in some cases, to *any* HTTP status code that’s not 200). In unit tests this isn’t hard, but when you’re starting to test the network layers and beyond, you need something on the network to provide the errors. That’s what WebStatusCodes does. It’s also a simple-but-handy reference for HTTP status codes, though it is incomplete (418 I’m a teapot is not supported). Still, worth checking out.

Interesting to note, this is my first AppEngine application, and I believe it took me 20 minutes to download the SDK, get something working, and get it deployed. It was like one of those ‘build a blog in -15 minutes’ moments. Empowering the speed at which you can create things on AppEngine, though I’d be slow to consider it for anything much more complex.

Systems and Devops People, Hack With Me!

I like systems-land, and a while back I was stuck writing some reporting code, which I really don’t like, so I started a side project to see just how much cool stuff I could do using the /proc filesystem and nothing but pure Python. I didn’t get too far because the reporting project ended and I jumped back into all kinds of other goodness, but there’s a github project called pyproc that’s just a single file with a few functions in it right now, and I’d like to see it grow, so fork it and send me pull requests. If you know Linux systems pretty well but are relatively new to Python, I’ll lend you a hand where I can, though time will be a little limited until the book is done (see further down).

The other projects I’m working on are sort of in pursuit of larger fish in the Devops waters, too, so be sure to check out the other projects I mention later in this post, and follow me on github.

Python Meetup Group in Princeton NJ

I started a Meetup group for Pythonistas that probably work in NYC or PA, but live in NJ. I work in PA, and before this group existed, the closest group was in Philly, an hour from home. I put my feelers out on Twitter, found some interest, put up a quick Meetup site, and we had 13 people at the first meetup (more than had RSVP’d). It’s a great group of folks, but more is always better, so check it out if you’re in the area. We hold meetings at the beautiful Princeton Public Library (who found us on twitter and now sponsors the group!), which is just a block or so from Triumph, the local microbrewery. I’m hoping to have a post-meeting impromptu happy hour there at some point.

Python Cookbook Progress

The Python Cookbook continues its march toward production. Lots of work has been done, lots of lessons have been learned, lots of teeth have been gnashed. The book is gonna rock, though. I had the great pleasure of porting all of the existing recipes that are likely to be kept over to Python 3. Great fun. It’s really amazing to see just how it happens that a 20-line recipe is completely obviated by the addition of a single, simple language feature. It’s happened in almost every chapter I’ve looked at so far.

If you have a recipe, or stumble upon a good example of some language feature, module, or other useful tidbit, whether it runs in Python 3 or not, let me know (see ‘Contact Me’). The book is 100% Python 3, but I’ve gotten fairly adept at porting things over by now :) Send me your links, your code, or whatever. If we use the recipe, the author will be credited in the book, of course.

PyRabbit is Coming

In the next few days I’ll be releasing a Python module on github that will let you easily work with RabbitMQ servers using that product’s HTTP management API. It’s not nearly complete, which is why I’m releasing it. It does some cool stuff already, but I need another helper or two to add new features and help do some research into how RabbitMQ broker configuration affects JSON responses from the API. Follow me on github if you want to be the first to know when I get it released. You probably also want to follow myYearbook on github since that’s where I work, and I might release it through the myYearbook github organization (where we also release lots of other cool open source stuff).

Python Asynchronous AMQP Consumer Module

I’m also about 1/3 of the way through a project that lets you write AMQP consumers using the same basic model as you’d write a Tornado application: write your handler, import the server, link the two (like, one line of code), and call consume(). In fact, it uses the Tornado IOLoop, as well as Pika, which is an asynchronous AMQP module in Python (maintained by none other than my boss and myYearbook CTO,  @crad), which also happens to support the Tornado IOLoop directly.

Lessons Learned Porting Dateutil to Python 3

The dateutil module is a very popular third-party (pure) Python module that makes it easier (and in some cases, possible) to perform more advanced manipulations on dates and date ranges than simply using some combination of Python’s ‘included batteries’ like the datetime, time and calendar modules.

Dateutil does fuzzy date matching, Easter calculations in the past and future, relative time delta calculations, time zone manipulation, and lots more, all in one nicely bundled package.

I decided to port dateutil to Python 3.

Why?

For those who haven’t been following along at home, David Beazley and I are working on the upcoming Python Cookbook 3rd Edition, which will contain only Python 3 recipes. Python 2 will probably only get any real treatment when we talk about porting code.

When I went back to the 2nd edition of the book to figure out what modules are used heavily that might not be compatible with Python 3, dateutil stuck out. It’s probably in half or more of the recipes in the ‘Time and Money’ chapter in the 2nd Edition. I decided to give it a look.

How Long Did it Take?

Less than one work day. Seriously. It was probably 4-5 hours in total, including looking at documentation and getting to know dateutil. I downloaded it, I ran 2to3 on it without letting 2to3 do the in-place edits, scanned the output for anything that looked ominous (there were a couple of things that looked a lot worse than they turned out to be), and once satisfied that it wasn’t going to do things that were dumb, I let ‘er rip: I ran 2to3 and told it to go ahead and change the files (2to3 makes backup copies of all edited files by default, by the way).

What Was the Hardest Part?

Well, there were a few unit tests that used the base64 module to decode some time zone file data into a StringIO object before passing the file-like object to the code under test (I believe the code under test was the relativedelta module). Inside there, the file-like StringIO object is subjected to a bunch of struct.unpack() calls, and there are a couple of plain strings that get routed elsewhere.

The issue with this is that there are NO methods inside the base64 module that return strings anymore, which makes creating the StringIO object more challenging. All base64 methods return Python bytes objects. So, I replaced the StringIO object with a BytesIO object, all of the struct.unpack() calls “just worked”, and the strings that were actually needed as strings in the code had a ‘.decode()’ appended to them to convert the bytes back to strings. All was well with the world.

What Made it Easier?

Two things, smaller one first:

First, Python built-in modules for date handling haven’t been flipped around much, and dateutil doesn’t have any dependencies outside the standard library (hm, maybe that’s 2 things right there). The namespaces for date manipulation modules are identical to Python 2, and I believe for the most part all of the methods act the same way. There might be some under-the-hood changes where things return memoryview objects or iterators instead of lists or something, but in this and other porting projects involving dates, that stuff has been pretty much a non-event most of the time

But the A #1 biggest thing that made this whole thing take less than a day instead of more than a week? Tests.

Dateutil landed on my hard drive with 478 tests (the main module has about 3600 lines of actual code, and the tests by themselves are roughly 4000 lines of code). As a result, I didn’t have to manually check all kinds of functionality or write my own tests. I was able to port the tests fairly easily with just a couple of glitches (like the aforementioned base64 issue). From there I felt confident that the tests were testing the code properly.

In the past couple of days since I completed the ‘project’, I ported some of the dateutil recipes from the 2nd edition of the book to Python 3, just for some extra assurance. I ported 5 recipes in under an hour. They all worked.

Had You Ported Stuff Before?

Well, to be honest most of my Python 3 experience (pre-book, that is) is with writing new code. To gain a broader exposure to Python 3, I’ve also done lots of little code golf-type labs, impromptu REPL-based testing at work for things I’m doing there, etc. I have ported a couple of other small projects, and I have had to solve a couple of issues, but it’s not like I’ve ever ported something the size of Django or ReportLab or something.

The Best Part?

I had never seen dateutil in my life.

I had read about it (I owned the Python Cookbook 2nd Edition since its initial release, after all), but I’d never been a user of the project.

The Lessons?

  1. This is totally doable. Stop listening to the fear-inducing rantings of naysayers. Don’t let them hold you back. The pink ponies are in front of you, not behind you.
  2. There are, in fact, parts of Python that remain almost unchanged in Python 3. I would imagine that even Django may find that there are swaths of code that “just works” in Python 3. I’ll be interested to see metrics about that (dear Django: keep metrics on your porting project!)
  3. Making a separation between text and data in the language is actually a good thing, and in the places where it bytes you (couldn’t resist, sorry), it will likely make sense if you have a fundamental understanding of why text and data aren’t the same thing. I predict that, in 2012, most will view complainers about this change the same way we view whitespace haters today.

“I Can’t Port Because…”

If you’re still skeptical, or you have questions, or you’re trying and having real problems, Dave and I would both love for *you* to come to our tutorial at PyCon. Or just come to PyCon so we can hack in the hallway on it. I’ve ported, or am in the process of porting, 3 modules to Python 3. Dave has single-handedly ported something like 3-5 modules to Python 3 in the past 6 weeks or so. He’s diabolical.

I’d love to help you out, and if it turns out I can’t, I’d love to learn more about the issue so we can shine a light on it for the rest of the community. Is it a simple matter of documentation? Is it a bug? Is it something more serious? Let’s figure it out and leverage the amazing pool of talent at PyCon to both learn about the issue and hopefully get to a solution.

The Makings of a Great Python Cookbook Recipe

I’ve seen some comments on Twitter, Buzz, Reddit, and elsewhere, and we’ve gotten some suggestions for recipes already via email (thanks!), and both Dave and I thought it’d be good to present a simple-to-follow ‘meta-recipe’; a recipe for making a great recipe that has a good shot at making it into the cookbook.

So let’s get down to what makes a good recipe. These are in no particular order:

Concise

When you read a recipe for apple pie, it doesn’t include a diatribe about how to grow and pick apples. This is in part because of space constraints (the book would be huge, or the coverage would be incomplete, take your pick). It’s also partly because you can probably assume that the reader has somehow procured apples and assumes they’ll need them for this recipe and all of that.

In a recipe for, say, an ETL script that hopes to illustrate Python’s useful string manipulation features, you probably don’t need to spend much time explaining what a database is, or take a lot of lines of code to deal with the database. It makes the code longer, and just distracts from the string manipulation goodness you’re trying to relay to the reader.

Short and sweet. “Everything you need, nothing you don’t”.

Illustrative

Recipes for the cookbook should be relatively narrowly focused on illustrating a particular technique, module (built-in or otherwise), or language feature, and bringing it to life. It need not be illustrative of a business process, remotely associated technology, etc.

So, if you want to write a recipe about os.walk, you don’t need to get into the semantics of the ext3 filesystem implementation, because that’s not what you’re trying to illustrate.

Practical

Above I noted that the recipe should be relatively narrowly focused on a technique or language feature. It should NOT be narrowly focused in terms of its applicability.

For example, if you wanted to illustrate the usefulness of the Python csv module, awesome! And if you want to mention that csv will attempt to make its output usable in Excel, awesome! But if you wanted to write a recipe called “Supporting Windows ’95 Excel Clients With Python” dealing only with Excel, specifically on Windows ’95, well… that’s really specific, and really a ‘niche’ recipe. It’d be better left for some future ‘Python Hacks’ book or something.

When you read a cookbook, you probably don’t seek out “How to make mulligatawny soup in a Le Creusetâ„¢ Dutch Oven Using an Induction Stove at 30,000 Feet”. Likewise, in order to be useful to a wider audience, your recipe should ideally not force so many assumptions onto readers who just want to make a good meal (so to speak).

Our devotion to the practical also means we don’t plan to include any recipes dealing with Fibonacci numbers, factorials, or the like. Leave those for some future “Python Homework Problems” book.

Well-Written

By ‘well-written’, I’m partially just lumping everything I just said all together under one title. However, in addition, I would ask that recipe authors resist the temptation to utilize unnecessary ‘cleverness’ that might make the code harder to read and understand, or be a distraction from what you’re trying to relay to the reader.

Just because you can get the job done with a nested list comprehension doesn’t mean you should. Open up in the code listing to allow easy comprehension by readers at all levels. If you must use nested list comprehensions, perhaps it warrants a separate recipe?

Nested list comprehensions are just an example, of course. I’m sure you can think of others. When you’re looking at your recipe, just ask yourself if there’s a simpler construct, technique, or idiom that can be used to achieve the same goal.

Pythonic

In general, follow the ‘import this’ rules like you would with any other Python code you write. “Sparse is better than dense”, “Readability counts”, etc. In addition, bonus points are given for following PEP 8.

But I’m not just talking about the code. Long-time Python coders (or maybe even not-so-long-time ones) come to realize that the Zen of Python applies not just to code, but to the way you think about your application. When Dave and I are reading recipe descriptions, we’re thinking in that mode. “Would we do this? Why would we do this? When would we do this? Is there a more Pythonic solution to this problem? Can this solution be made more Pythonic?”

When in doubt…

If you’re not sure about any of that, your default action should be to post your recipe on the ActiveState site. The reality is that posting a recipe there will help both the community and yourself. The feedback you get on your recipe will help you become a better coder, and it’ll help people evaluating your recipe to make sound decisions and perhaps consider things they hadn’t. It’s good all over. Getting into the book is just a nice cherry on the sundae.

Also, while unit tests are fantastic, they’re not expected to show up along with the recipe. ActiveState to my knowledge doesn’t have a mechanism for easily including test code alongside (but not embedded in) the recipe code. If you want to use doctest, great. If you want to point us at a recipe you’ve posted, you can include tests in that email, or not. It’s totally unnecessary to include them, although they are appreciated.

Questions?

If you have questions, email them to Dave and I at PythonCookbook at oreilly dot com. You can also post questions here in the comments of this post.

Nose and Coverage.py Reporting in Hudson

I like Hudson. Sure, it’s written in Java, but let’s be honest, it kinda rocks. If you’re a Java developer, it’s admittedly worlds better because it integrates with seemingly every Java development tool out there, but we can do some cool things in Python too, and I thought I’d share a really simple setup to get coverage.py’s HTML reports and nose’s xUnit-style reports into your Hudson interface.

I’m going to assume that you know what these tools are and have them installed. I’m working with a local install of Hudson for this demo, but it’s worth noting that I’ve come to find a local install of Hudson pretty useful, and it doesn’t really eat up too much CPU (so far). More on that in another post. Let’s get moving.

Process Overview

As mentioned, this process is really pretty easy. I’m only documenting it because I haven’t seen it documented before, and someone else might find it handy. So here it is in a nutshell:

  • Install the HTML Publisher plugin
  • Create or alter a configuration for a “free-style software project”
  • Add a Build Step using the ‘Execute Shell’ option, and enter a ‘nosetests’ command, using its built-in support for xUnit-style test reports and coverage.py
  • Check the ‘Publish HTML Report’, and enter the information required to make Hudson find the coverage.py HTML report.
  • Build, and enjoy.

Install The HTMLReport Plugin

From the dashboard, click ‘Manage Hudson’, and then on ‘Manage Plugins’. Click on the ‘Available’ tab to see the plugins available for installation. It’s a huge list, so I generally just hit ‘/’ in Firefox or cmd-F in Chrome and search for ‘HTML Publisher Plugin’. Check the box, go to the bottom, and click ‘Install’. Hudson will let you know when it’s done installing, at which time you need to restart Hudson.

Install tab

HTML Publisher Plugin: Check!

Configure a ‘free-style software project’

If you have an existing project already, click on it and then click the ‘Configure’ link in the left column. Otherwise, click on ‘New Job’, and choose ‘Build a free-style software project’ from the list of options. Give the job a name, and click ‘OK’.

Build a free-style software project.

You have to give the job a name to enable the 'ok' button :)

Add a Build Step

In the configuration screen for the job, which you should now be looking at, scroll down and click the button that says ‘Add build step’, and choose ‘Execute shell’ from the resulting menu.

Add Build Step

Execute shell. Mmmmm... shells.

This results in a ‘Command’ textarea appearing, which is where you type the shell command to run. In that box, type this:

/usr/local/bin/nosetests --with-xunit --with-coverage --cover-package demo --cover-html -w tests

Of course, replace ‘demo’ with the name of the package you want covered in your coverage tests to avoid the mess of having coverage.py try to seek out every module used in your entire application.

We’re telling Nose to generate an xUnit-style report, which by default will be put in the current directory in a file called ‘nosetests.xml’. We’re also asking for coverage analysis using coverage.py, and requesting an HTML report of the analysis. By default, this is placed in the current directory in ‘cover/index.html’.

execute shell area

Now we need to set up our reports by telling Hudson we want them, and where to find them.

Enable JUnit Reports

In the ‘Post-Build Actions’ area at the bottom of the page, check ‘Publish JUnit test result report’, and make it look like this:

The ‘**’ is part of the Ant Glob Syntax, and stands for the current working directory. Remember that we said earlier nose will publish, by default, to a file called ‘nosetests.xml’ in the current working directory.

The current working directory is going to be the Hudson ‘workspace’ for that job, linked to in the ‘workspace root’ link you see in the above image. It should mostly be a checkout of your source code. Most everything happens relative to the workspace, which is why in my nosetest command you’ll notice I pass ‘-w tests’ to tell nose to look in the ‘tests’ subdirectory of the current working directory.

You could stop right here if you don’t track coverage, just note that these reports don’t get particularly exciting until you’ve run a number of builds.

Enable Coverage Reports

Just under the JUnit reporting checkbox should be the Publish HTML Reports checkbox. The ordering of things can differ depending on the plugins you have installed, but it should at least still be in the Post-build Actions section of the page.

Check the box, and a form will appear. Make it look like this:

By default, coverage.py will create a directory called ‘cover’ and put its files in there (one for each covered package, and an index). It puts them in the directory you pass to nose with the ‘-w’ flag. If you don’t use a ‘-w’ flag… I dunno — I’d guess it puts it in the directory from where you run nose, in which case the above would become ‘**/cover’ or just ‘cover’ if this option doesn’t use Ant Glob Syntax.

Go Check It Out!

Now that you have everything put together, click on ‘Save’, and run some builds!

On the main page for your job, after you’ve run a build, you should see a ‘Coverage.py Report’ link and a ‘Latest Test Result’ link. After multiple builds, you should see a test result ‘Trend’ chart on the job’s main page as well.

job page

Almost everything on the page is clickable. The trend graph isn’t too enlightening until multiple builds have run, but I find the coverage.py reports a nice way to see at-a-glance what chunks of code need work. It’s way nicer than reading the line numbers output on the command line (though I sometimes use those too).

How ’bout you?

If you’ve found other nice tricks in working with Hudson, share! I’ve been using Hudson for a while now, but that doesn’t mean I’m doing anything super cool with it — it just means I know enough to suspect I could be doing way cooler stuff with it that I haven’t gotten around to playing with. :)

Nose Hates Me

I easy_install’d nose on my iMac some time in the last month, and tried to use it with options for the first time today, and I’ve found that a good number of the ones shown in ‘nosetests –help’ are actually not recognized when I run nosetests. Meanwhile, running nosetests with no options still works fine. This is 0.11.3 on OS X. Google gives me only references to plugins not being found. These are allegedly “built in”! Nobody replied on Twitter either, which is pretty odd in my experience. So here I am. Wtf is going on here?

For sure, -x and -v aren’t recognized, and -p *is* recognized. Using ‘-w’ with ‘.’ as an argument results in ‘no option -w’, but feeding it a non-existent directory results in a Python ValueError (/foo not found, or not a directory). Wtf?

In checking out /Library/Python/2.6/site-packages/nose-0.11.3-py2.6.egg/nose/config.py, I can see that these options are defined, so I’m a bit confused. Here’s a small sampling of output:

Brian-Joness-iMac:tests bjones$ nosetests -x 
Usage: nosetests [options]

nosetests: error: no such option: -x
Brian-Joness-iMac:tests bjones$ nosetests -v 
Usage: nosetests [options]

nosetests: error: no such option: -v
Brian-Joness-iMac:tests bjones$ nosetests 
..
----------------------------------------------------------------------
Ran 2 tests in 0.360s

OK

Clues hereby solicited.

Python Testing Beginner’s Guide: The Review

I try not to make a habit of reviewing technical books. I own more than my fair share of technical books, and I’ve been involved in publishing and even wrote a book myself. Most technical books, on the whole, are pretty bad.

So, there’s Disclaimer 1: you know the mindset I’m starting with. I’m a bit critical of tech books, publishers, and occasionally even authors (but usually publishers).

Most book reviews are also biased for one reason or another, and I hate to be lumped into a pool of “book reviewers” that I have nothing to do with. A great many of them are astroturfers trading favors with the publishing company or just cheerleading in general when in fact they’ve never even seen the book.

So, Disclaimer 2:

I requested a free “review copy” of Python Testing: Beginner’s Guide, specifically because I wanted to pick up some of the techniques I expected I’d find in the book on a project I was working on in the moment. The timing really could not have been better. They requested that I post a review, and I said that I would — “good or bad”.

So let’s talk about the book!

Things That Impressed Me

It’s an eBook

I got an eBook from Packt Publishing and dove into the book almost immediately, without having to wait behind some annoying old lady griping incessantly to an equally annoying emo-hipster wannabe behind the counter at a bookstore who inevitably can’t do whatever it is this old lady wants. We’ve all been there. eBooks rock. I think some publishers do better than others where eBooks are concerned, but that’s a tale for another day. I was happy I could at least get an eBook (PDF).

Code Listings

This book’s author and/or editor appears to have taken great pains to insure that the Python code listings do not break across pages in strange places. Python, as a language, isn’t exactly custom made for book layouts: since there are no brackets and only indentation is used to denote the scope of a given statement, if you have trouble telling how far a line on one page is indented relative to the line on the next page, it can be tough. Especially on newbies, but in some cases just as much for experienced coders. Anyway, I could not find a single instance of this issue in this book, and I was really impressed by that.

Example Scenarios

Ok, pretty much all example scenarios given by an author to provide some context to illustrate a concept are contrived. That doesn’t bother me. What does bother me is that most authors just shrug and say “this is how it is, and how it will always be, forever, so it doesn’t matter that my example scenario completely fails to engage the reader”. This author didn’t do that. He came up with example scenarios that were pretty much contrived, but he lent to them a “slice of life” that made them a billion times more interesting to actually read through, and I did!

I often just skim the explanation of the example scenario because most of them are so much alike, and are such null operations, that the pages spent explaining “Stupidest Program You’ll Ever Write: Take 1,112,039″ are a complete waste of space. But I read these, and really enjoyed them. I mean really, who sits down to explain Python Testing and says to themselves “I know! We’ll write a PID controller!”. The answer is Daniel Arbuckle, this book’s author.

Python Intro

There isn’t one in this book, and I’m happy about that. If you don’t know the language, you shouldn’t be reading a book devoted to testing code you don’t know how to write. That said, beginner books on languages could (and perhaps should, at least to an extent) teach by starting with testing, or at least integrate testing into the learning. Anyway, this book skips those really mind-numbing first three chapters about who Guido is and how ‘Python’ is not a reptilian reference. The author dives right into the meat of the matter. Very good.

The Author

I don’t know Daniel Arbuckle. Never met him. Never heard of him. Don’t think he wrote any other books that I know of. But the guy can really write. There’s only so much an editor or publisher can do for you in terms of your voice as an author. At some point, it’s just you, the author, telling a story to the reader, and efforts by the publisher to make it “more this” or “more that” just become really obvious and distracting.

Daniel Arbuckle writes like he really is excited to be talking to you about Python testing. He has interesting things to say, he writes in a good-natured, friendly, engaging tone, and he is thoughtful about his audience of beginners throughout. If you know Python, but are a little bewildered by testing, you’re going to understand everything Arbuckle says, and you’re going to have lots of light bulb moments. The shocking bit to me was this this is a guy who has a PhD in Computer Science, and a lot of PhD’s (in any topic area) write everything as if they’re writing their thesis. A thesis is not typically written to engage newcomers and non-academics.

Impact

At the start of this book, my experience with testing consisted of historically writing some really ugly code to do what amounts to functional testing, and I had, over the past year, taken to writing proper unit tests for some of my less-complicated projects. More complicated projects involving threading/multiprocessing with networked queues, databases and remote APIs? Well, I was trying to organize my code to make it more testable, but it impacted my development timeline to such an extent that testing took a back seat to deadlines, sadly (I’m the only person I know who has an actual, genuine *interest* in code quality).

At the end of this book, I still have some of the same problems, of course, but I am extremely excited and far more confident that I *will* eventually be able to work this into my breakneck development pace and have it be effective. I have a pretty firm grasp of both unittest and doctest, in addition to Nose, and one thing I *didn’t* have going in was an understanding that doctest is actually perfectly suited for testing in some areas where unittest might be a little overkill, harder to do, etc.

I also was able to put my own experiences with testing into perspective, and I was able to apply some things I learned pretty much right away.

Things That Did Not Impress Me

The Index

Here’s the thing, 99% of all tech book indexes are terrible, because 99% of all books have a computer-generated index that the author never sees until the published book reaches his doorstep. The books with really good indexes utilize input from the author, and I use the quality of an index as a sign of the quality of the publisher, the editor, their ability to manage the hectic process that is the creation of a book, and their commitment to quality.

This book’s index is not nearly the worst I’ve seen, but it’s pretty bad. If you’ve never noticed that the indexes of several books you own are terrible, you can probably disregard this.

The Editing

There’s the author, and then there’s the editor. When you’re looking for authors, you’re less concerned with whether they can write in accordance with the Chicago Style Manual and more interested in whether they can convey ideas to an audience using words. It’s the job of the publisher’s staff to deal with grammatical problems and issues with punctuation. So when you see bad grammar and punctuation (or even spelling) in a technical book, look to the publisher, not the author. The publisher’s job is to make the author look like a rock star. If you’re not thinking that, it’s not the author’s fault.

As it turns out, one of two things seems to have happened in this book:

  1. Daniel Arbuckle didn’t major in English, so his use of commas is off, and the editors failed to pick up on that (like, a lot), or
  2. The publisher hired a summer intern from the local high school to do Mr. Arbuckle the favor of inserting commas wherever she thought they looked pretty.

I don’t know which one of these things happened, but the misuse of punctuation in this book really bugged me.

Mock Coverage

Before I read this book, I perceived mocking objects as being the hardest part of testing my more complicated applications. After reading this book, I still feel that way.

Unlike Arbuckle’s coverage of both unittest and doctest (and nose) for performing basic unit tests, he only covers one mocking library/framework: Python Mocker. I was glad to go through the exercise, because it forced me to sit down and do things with a record/playback style framework, but I still feel like this style of framework just does not fit my brain. I really wish the book had one more chapter about object mocking using a library that did not use that model. There are a few out there. One I’ve found that looks good (if I could ever get time to dig into it) is Michael Foord’s Mock, which uses an ‘action/assertion’ model, much like the rest of the testing tools outside of mocking frameworks.

In All…

If you know Python, and you’re not testing yet, I strongly recommend this book to get you started. It covers unittest, doctest, nose, twill; the differences between unit testing, functional testing and integration testing; testing web sites; and lots of other stuff. It also has a ‘catch-all’ chapter that talks about coverage.py, installing nose as a post-commit hook to every VCS under the sun, etc.

I’m confident that you’ll find this book a useful introduction to the world of Python testing, and that it will equip you with the knowledge of both tools and concepts you’ll need to go off on your own and solve as yet unforeseen testing issues you might come across.