• Author: Doug Hellmann
• Pages: 1344
• Publisher: Addison-Wesley (Developer’s Library)
• ETA: June 5, 2011
• Amazon link: http://www.amazon.com/Python-Standard-Library-Example-Developers/dp/0321767349/ref=sr_1_1?ie=UTF8&qid=1307109464&sr=1-1-spell

What this book says it does:

From the book’s description:

This book is a collection of essays and example programs demonstrating how to use more than 100 modules from Python standard library. It goes beyond the documentation available on python.org to show real programs using the modules and demonstrating how you can use them in your daily programming tasks.

What this book actually does:

This book actually kinda rocks, in part because of its a unique take on documentation of the Python standard library. The Python standard library documentation is actually a pretty good high-level reference, and this book doesn’t seek to duplicate what’s there. Instead, it specifically seeks out places in the existing documentation that are underdocumented, undocumented, don’t have clear enough examples, or just don’t provide the value to the end user that they should for whatever reason. Even as good as the standard library documentation is, Doug easily cranked out 1000+ pages of invaluable information that has given me a much greater insight into the standard library modules that I use on a regular basis (and plenty that I don’t use on a regular basis).

How it works

The book is simply laid out by module. Using the multiprocessing module? It’s right there in the Table of Contents. It’s as easy to use as the standard library docs from a navigational perspective, and the index, it could be argued, is an improvement over docs.python.org’s  search behavior.

When you get to the module you’re looking for, you’ll primarily see code. There is enough English text to explain what the code actually does, but the main illustrative tool in this book is the code. This is not an easy thing to accomplish, but Doug provides a very nice and balanced presentation of the real meaty parts of your favorite standard library modules.

What’s Great About it

First, it provides both depth and breadth, it’s easy to find whatever you’re looking for, and if it’s not needed (usually because it’s well-covered in the standard library docs) it’s not there.

Second, the book is written by an authoritative, knowledgeable, experienced, and prolific Python developer. While he’s a creative thinker, his work is balanced by a healthy dose of pragmatism and grounded in best practices. Contrived as the examples might get at times, you won’t typically find code written by Doug that would garner sideways glances by experienced Python developers.

Third, it’s not a rehashing of the docs. In fact it skips coverage of things that are well-documented in the docs. Yes, the book does contain simple introductory material for each module to give the uninitiated some context, but that’s different from a book that takes existing docs and just moves the letters around. Doug does a great job of getting you into the good stuff without much fluff.

Fourth, there’s almost zero fluff. I’d love to see a statistical breakdown of the number of lines of code vs. text in this book. And the good part really isn’t that he’s put so much code in the book, it’s that he presents with code alongside the text in a way that insures readers don’t get lost.

Fifth, this wasn’t a rush job. Doug has been writing this content in the form of his Python Module of the Week blog series for a few years now. Most of the work was editing, finessing, updating, and testing (and retesting) the code, not developing the content from scratch. So what’s there in the book is not just a braindump from Doug’s brain: it’s had the benefit of peer review and feedback from the blog, email, etc., and that adds a ton of value to the final product in my eyes.

What’s Not Great About it

I insist on including bad things about everything I review, because nothing is perfect, and the more people talk about things they don’t like, the more makers start to listen and make things better.

To be honest, the only thing I found lacking in this book is the index. This should not shock anyone who is a tech bibliophile. Most indexes, at least on tech books, are pretty bad (ironic since tech books often serve as references, which makes the index pretty crucial). Also, consider that this review is based on a review copy of the book, so it’s possible that the final version will have a totally awesome index.

Ah, one other thing (which also might be due to this being a review copy): there are no tab markers. Since Python defines scope using whitespace, not having indentation markers in any medium containing page breaks can lead to confusion if a code sample crosses page boundaries. The alternative to having the markers is to insure that the code samples don’t cross page boundaries. It’s not possible for me to know if they’ll do one or both of these before the final printing.

The Final Word

My petty complaints about the index and indentation markers are not only trivial, but they both may be fixed in the final printing. I saw nothing so bad that I wouldn’t highly recommend this book, and I’ve seen tons and tons of stuff that would make me highly recommend this book. I’m using this book myself on a fairly regular basis, and it’s an effective, easy-to-use tool that makes a great companion reference to the standard library.

Buy it.

Dateutil does fuzzy date matching, Easter calculations in the past and future, relative time delta calculations, time zone manipulation, and lots more, all in one nicely bundled package.

I decided to port dateutil to Python 3.

Why?

For those who haven’t been following along at home, David Beazley and I are working on the upcoming Python Cookbook 3rd Edition, which will contain only Python 3 recipes. Python 2 will probably only get any real treatment when we talk about porting code.

When I went back to the 2nd edition of the book to figure out what modules are used heavily that might not be compatible with Python 3, dateutil stuck out. It’s probably in half or more of the recipes in the ‘Time and Money’ chapter in the 2nd Edition. I decided to give it a look.

How Long Did it Take?

Less than one work day. Seriously. It was probably 4-5 hours in total, including looking at documentation and getting to know dateutil. I downloaded it, I ran 2to3 on it without letting 2to3 do the in-place edits, scanned the output for anything that looked ominous (there were a couple of things that looked a lot worse than they turned out to be), and once satisfied that it wasn’t going to do things that were dumb, I let ‘er rip: I ran 2to3 and told it to go ahead and change the files (2to3 makes backup copies of all edited files by default, by the way).

What Was the Hardest Part?

Well, there were a few unit tests that used the base64 module to decode some time zone file data into a StringIO object before passing the file-like object to the code under test (I believe the code under test was the relativedelta module). Inside there, the file-like StringIO object is subjected to a bunch of struct.unpack() calls, and there are a couple of plain strings that get routed elsewhere.

The issue with this is that there are NO methods inside the base64 module that return strings anymore, which makes creating the StringIO object more challenging. All base64 methods return Python bytes objects. So, I replaced the StringIO object with a BytesIO object, all of the struct.unpack() calls “just worked”, and the strings that were actually needed as strings in the code had a ‘.decode()’ appended to them to convert the bytes back to strings. All was well with the world.

What Made it Easier?

Two things, smaller one first:

First, Python built-in modules for date handling haven’t been flipped around much, and dateutil doesn’t have any dependencies outside the standard library (hm, maybe that’s 2 things right there). The namespaces for date manipulation modules are identical to Python 2, and I believe for the most part all of the methods act the same way. There might be some under-the-hood changes where things return memoryview objects or iterators instead of lists or something, but in this and other porting projects involving dates, that stuff has been pretty much a non-event most of the time

But the A #1 biggest thing that made this whole thing take less than a day instead of more than a week? Tests.

Dateutil landed on my hard drive with 478 tests (the main module has about 3600 lines of actual code, and the tests by themselves are roughly 4000 lines of code). As a result, I didn’t have to manually check all kinds of functionality or write my own tests. I was able to port the tests fairly easily with just a couple of glitches (like the aforementioned base64 issue). From there I felt confident that the tests were testing the code properly.

In the past couple of days since I completed the ‘project’, I ported some of the dateutil recipes from the 2nd edition of the book to Python 3, just for some extra assurance. I ported 5 recipes in under an hour. They all worked.

Had You Ported Stuff Before?

Well, to be honest most of my Python 3 experience (pre-book, that is) is with writing new code. To gain a broader exposure to Python 3, I’ve also done lots of little code golf-type labs, impromptu REPL-based testing at work for things I’m doing there, etc. I have ported a couple of other small projects, and I have had to solve a couple of issues, but it’s not like I’ve ever ported something the size of Django or ReportLab or something.

The Best Part?

I had never seen dateutil in my life.

I had read about it (I owned the Python Cookbook 2nd Edition since its initial release, after all), but I’d never been a user of the project.

The Lessons?

1. This is totally doable. Stop listening to the fear-inducing rantings of naysayers. Don’t let them hold you back. The pink ponies are in front of you, not behind you.
2. There are, in fact, parts of Python that remain almost unchanged in Python 3. I would imagine that even Django may find that there are swaths of code that “just works” in Python 3. I’ll be interested to see metrics about that (dear Django: keep metrics on your porting project!)
3. Making a separation between text and data in the language is actually a good thing, and in the places where it bytes you (couldn’t resist, sorry), it will likely make sense if you have a fundamental understanding of why text and data aren’t the same thing. I predict that, in 2012, most will view complainers about this change the same way we view whitespace haters today.

“I Can’t Port Because…”

If you’re still skeptical, or you have questions, or you’re trying and having real problems, Dave and I would both love for *you* to come to our tutorial at PyCon. Or just come to PyCon so we can hack in the hallway on it. I’ve ported, or am in the process of porting, 3 modules to Python 3. Dave has single-handedly ported something like 3-5 modules to Python 3 in the past 6 weeks or so. He’s diabolical.

I’d love to help you out, and if it turns out I can’t, I’d love to learn more about the issue so we can shine a light on it for the rest of the community. Is it a simple matter of documentation? Is it a bug? Is it something more serious? Let’s figure it out and leverage the amazing pool of talent at PyCon to both learn about the issue and hopefully get to a solution.

Well, I am, and I’m here to get the rumor mill started by sharing some predictions for this year’s PyCon.

Packaging

While last year there was some hubbub around packaging, most notably the poster images declaring that pip and distribute were ‘the new black’, this year that’s old news. The pip and distribute mantra will bear repeating for those who are still behind the times of course, but this year I think there will be more hand-wringing over packaging, probably focusing more on the data and distribution mechanism than the tools. I, for one, am looking forward to any light Tarek Ziade and other’s on the distutils2 team can shed on new metadata elements, new build procedures (I hear setup.py is going away someday), and what’s going on with distutils, distribute, pip, distutils2, PyPI, etc.

My prediction? Packaging, while not very exciting, will garner a lot of attention again this year.

Python 3

Dave Beazley and I will be doing a tutorial on Python 3, but besides that, I think there’s going to be a lot of hallway (and lounge) discussion about all things Python 3. This would mirror the clamor around the topic recently online, which I don’t really expect to slow down. Python 3 is, of course, discussed every year, but I think this year’s PyCon will be Python 3′s Woodstock: the time and place where minds merge onto an idea together and harmonize and all that stuff.

My prediction? We’ll see a good number of commitments from projects to port to Python 3, GSOC projects related to Python 3, and more funny looks at projects not getting their porting underway in 2011 as a direct result of PyCon 2011.

Scale

Multiprocessing and threading will be hot topics again for two reasons: First, lots of people just look at the GIL and reflexively get heartburn without doing any actual analysis to see if they even need to care about it in the context of a particular application. Second, the uptake for Python in ‘web scale’ environments continues to grow, so that’s bound to be reflected in the attendee demographics if you will, and therefore the perceived interest in the topic.

PyPy is going to loom large at this year’s conference. Some may decide after PyCon 2011 that PyPy is the future for them. Unladen Swallow will be declared dead, in spite of some eager undergrads making empty promises about picking it up. Most who care will opt for the project that appears to be gaining steam, not losing it.

Generator-based coroutines will continue to be a really cool but relatively unused oddity, and Eventlet and the like will continue to have their adherents and go with the flow.

Event-based/async frameworks will garner some attention. Tornado, gUnicorn, and others will do well. Twisted followers will say that all others are either redundant or fatally flawed.

My prediction? Tornado and ZeroMQ will make a big splash, this being the first year it’s being covered (I think). People will still advocate multiprocessing over threading after the conference, and Twisted will start to see declines in usership after PyCon 2011.

Potpourri

• Python core development contributions will increase thanks to efforts by the PSF and core development communities, though not without some growing pains.
• Michael Foord’s ‘mock‘ library will take off as a de facto standard module in the toolbelts of Python developers everywhere.
• Fabric’s parallel branch will be merged as the default either before or as a result of activity at PyCon 2011.
• Flask will announce a merger with another web framework in 2011, in part due to conversations and activity at PyCon.
• PyCharm will become recognized as the preeminent Python IDE as a result of everyone getting to PyCon only to find that everyone else they were going to advocate it to is already using it.
• Some outside the scientific community will attend a Python-Science talk anyway and come away with valuable techniques they can apply to distributed computing problems. Some outside the web scalability domain will attend related talks anyway and take something back to the enterprise computing space. Some outside the cloud space will attend a cloud talk and find a cool project to put in the cloud. As a result:
• At least 30 new Python projects will be created as a direct result of PyCon 2011.

So let’s get down to what makes a good recipe. These are in no particular order:

Concise

When you read a recipe for apple pie, it doesn’t include a diatribe about how to grow and pick apples. This is in part because of space constraints (the book would be huge, or the coverage would be incomplete, take your pick). It’s also partly because you can probably assume that the reader has somehow procured apples and assumes they’ll need them for this recipe and all of that.

In a recipe for, say, an ETL script that hopes to illustrate Python’s useful string manipulation features, you probably don’t need to spend much time explaining what a database is, or take a lot of lines of code to deal with the database. It makes the code longer, and just distracts from the string manipulation goodness you’re trying to relay to the reader.

Short and sweet. “Everything you need, nothing you don’t”.

Illustrative

Recipes for the cookbook should be relatively narrowly focused on illustrating a particular technique, module (built-in or otherwise), or language feature, and bringing it to life. It need not be illustrative of a business process, remotely associated technology, etc.

So, if you want to write a recipe about os.walk, you don’t need to get into the semantics of the ext3 filesystem implementation, because that’s not what you’re trying to illustrate.

Practical

Above I noted that the recipe should be relatively narrowly focused on a technique or language feature. It should NOT be narrowly focused in terms of its applicability.

For example, if you wanted to illustrate the usefulness of the Python csv module, awesome! And if you want to mention that csv will attempt to make its output usable in Excel, awesome! But if you wanted to write a recipe called “Supporting Windows ’95 Excel Clients With Python” dealing only with Excel, specifically on Windows ’95, well… that’s really specific, and really a ‘niche’ recipe. It’d be better left for some future ‘Python Hacks’ book or something.

When you read a cookbook, you probably don’t seek out “How to make mulligatawny soup in a Le Creuset™ Dutch Oven Using an Induction Stove at 30,000 Feet”. Likewise, in order to be useful to a wider audience, your recipe should ideally not force so many assumptions onto readers who just want to make a good meal (so to speak).

Our devotion to the practical also means we don’t plan to include any recipes dealing with Fibonacci numbers, factorials, or the like. Leave those for some future “Python Homework Problems” book.

Well-Written

By ‘well-written’, I’m partially just lumping everything I just said all together under one title. However, in addition, I would ask that recipe authors resist the temptation to utilize unnecessary ‘cleverness’ that might make the code harder to read and understand, or be a distraction from what you’re trying to relay to the reader.

Just because you can get the job done with a nested list comprehension doesn’t mean you should. Open up in the code listing to allow easy comprehension by readers at all levels. If you must use nested list comprehensions, perhaps it warrants a separate recipe?

Nested list comprehensions are just an example, of course. I’m sure you can think of others. When you’re looking at your recipe, just ask yourself if there’s a simpler construct, technique, or idiom that can be used to achieve the same goal.

Pythonic

In general, follow the ‘import this’ rules like you would with any other Python code you write. “Sparse is better than dense”, “Readability counts”, etc. In addition, bonus points are given for following PEP 8.

But I’m not just talking about the code. Long-time Python coders (or maybe even not-so-long-time ones) come to realize that the Zen of Python applies not just to code, but to the way you think about your application. When Dave and I are reading recipe descriptions, we’re thinking in that mode. “Would we do this? Why would we do this? When would we do this? Is there a more Pythonic solution to this problem? Can this solution be made more Pythonic?”

When in doubt…

If you’re not sure about any of that, your default action should be to post your recipe on the ActiveState site. The reality is that posting a recipe there will help both the community and yourself. The feedback you get on your recipe will help you become a better coder, and it’ll help people evaluating your recipe to make sound decisions and perhaps consider things they hadn’t. It’s good all over. Getting into the book is just a nice cherry on the sundae.

Also, while unit tests are fantastic, they’re not expected to show up along with the recipe. ActiveState to my knowledge doesn’t have a mechanism for easily including test code alongside (but not embedded in) the recipe code. If you want to use doctest, great. If you want to point us at a recipe you’ve posted, you can include tests in that email, or not. It’s totally unnecessary to include them, although they are appreciated.

Questions?

If you have questions, email them to Dave and I at PythonCookbook at oreilly dot com. You can also post questions here in the comments of this post.

It’s Python 3, Cover to Cover

Go big or go home. The third edition will be a Python 3 Cookbook. This by itself makes this a rather large undertaking, since it means modules used in earlier editions that don’t work in Python 3 can’t be used, and so those old recipes will need to be scrapped or rewritten.

You heard it right: if a module used in the last edition of Python Cookbook doesn’t work in Python 3, it won’t be in this edition. This includes some modules for which several recipes already exist, like dateutil and wxPython, and other modules I would’ve liked to use for illustrative purposes, like Tornado and psycopg2. I guess there’s still some time for smaller modules to port to Python 3 and be left alone in this edition, but I don’t know how realistic that is for something like wxPython.

It’s going to be ok.

I can’t find any modules (yet) in the second edition for which Python 3-compatible substitutes don’t exist, and in some cases I found myself wondering why a separate module was used at all when what needs doing isn’t a whole lot of work in pure Python anyway. I guess if the module is there and stable, why not use it, eh? Fair enough.

Three Authors?

Actually, yes. David, myself, and YOU, where YOU is anyone who posts good Python 3 recipes to the ActiveState ‘Python Cookbook’ site. This is actually not new. If you look at the last edition, you’ll see separate credits for each recipe. The Python Cookbook has always been a community effort, featuring recipes by some very familiar names in the Python community.

Anyway, just in case that wasn’t direct enough:

Go to the ActiveState Python Cookbook site and post a Python 3 recipe, and if it’s solid, your name and recipe may well be in the book.

Basically, ActiveState gives us free reign over their Python Cookbook content, so it’s a convenient way to let the community contribute to the work. If it’s in there, and it’s good, we can use it. They’re cool like that.

Three Questions

Answer these in the comments, or send email to Dave and I at PythonCookbook at oreilly.

1. What are your three favorite recipes from either the 1st or 2nd edition?
2. What are your three least favorite recipes from either the 1st or 2nd edition?
3. What are three things (techniques, modules, basic tasks) you’d like to see covered that weren’t covered in earlier editions?

Three Cool Things About the Third Edition

1. Tests. When the book comes out, we’ll make the unit tests for the recipes available in some form that is yet to be determined (but they’ll be available).
2. Porting help. We’re not going to leave module authors out in the cold. We’re going to provide some help/advice. We’ve both written code in Python 3, and dealt with the issues that arise. I’m still porting one of my projects and have another in line after that, so I’ll be dealing with it even more.
3. Dave and I are both overwhelmed with excitement about this book, about Python 3, and about working with you on it. Come help us out by posting Python 3 recipes (tests are also nice, but not required) on ActiveState, and shoot us an email at PythonCookbook at oreilly dotcom.

There will be other cool things, too. We’ll let you know, so stay tuned to this blog, Dave’s blog, and you should definitely follow @bkjones and @dabeaz on Twitter, since we’ll be asking for opinions/resources/thoughts on things as we go.