Plug-ins: isn’t there a better way?

If there’s one thing that bothers me about using a ready-made solution like wordpress for my blog, it’s plug-ins. I hate software plug-ins. The first question every support engineer for any software product that supports plugins asks in response to a trouble report is “are you using any plugins?” And when you say “yep, I’m using plugins!” the reply from support is to disable them immediately and see if the trouble goes away. That’s a problem.

What’s worse, if the plugins are maintained by a third party (often the case), there’s no telling whether or not they’ll exist when the next version of the base software is released, or whether they’ll be supported in future versions of the software.

Two examples that touch my daily life are Firefox, and WordPress.

Lately (since around March) I’ve been having lots of trouble with Firefox. I thought upgrading to Firefox 3 would’ve helped, but it really didn’t. Running it on OS X, Firefox hangs frequently enough that I’m actually considering using Safari (I do NOT like Safari). Know what happened right around that time? Ah – I found the firefox plugins for managing EC2 and S3. So today I’ll uninstall those and see if it helps.

With WordPress, there are two things I’m missing: I need to let readers subscribe to comments via email, and I need better Google AdSense for Search integration with WordPress. Both things are kinda maybe supported in one version or another “but should work under…” – whatever. I don’t really want to spend my time downloading, reading the documentation to do the install, doing the install and configuration, etc., and then finding out that it doesn’t work, or worse, having it look on the surface like it works, but then finding later that it fails in evil-but-silent ways.

These two products are by no means exceptions. Moodle, PHP-Nuke, XOOPS, MediaWiki, Twiki, Postnuke… and for that matter, OpenLDAP, BIND, SSH, MySQL, Sendmail, PAM… all have plugins available written by other folks, and all have bitten me at one point or another. Usually when it comes time to upgrade the base software.

I’m not saying anything new here. People have had this problem with lots of different software products for a long time. My question is “why is this still a problem?” I’m not asking this because I have some magical obvious solution or answer, I’m asking because I feel like there’s probably more to it than I’m grasping. I’m not a masterful developer, or even a masterful software project manager, so I’m calling on all of you who are (or are closer than I am) to help me understand the problem. Some day, I might find myself in a position to take the wrong or right path where plug-ins are concerned, and I’d like to be more informed than I am so I can avoid putting users in the position I find myself in when I use other peoples’ software. Has Joel blogged this yet? If so, I can’t find it. Links please?

PyWorks 2008, November 12-14, Call for Papers Open!

Yes, the same folks who bring you Python Magazine and php|architect magazine (and several other things, like online training, a full line of books, and more conferences), are hosting our first ever Python conference! You can see more about it, and the Call for Papers, at the conference site.

The hotel which once hosted php|works in Atlanta is actually large enough to host both php|works *and* PyWorks in the same venue, so this year the two conferences will be held at the same time, and the plan/hope is to be able to have talks that are generic enough to be of use to either audience, like talks about scaling MySQL, or SVN management, or Hadoop, or Amazon Web Services, or something like that. In any case, the attendees of either conference will be allowed to cross over to attend talks at the other if they so choose, which I think is pretty cool. Maybe we can add more languages in future years and just have it be called “LANG ’08″ or something. See the PyWorks Call for Papers if you have ideas!

I’m really excited to get down to Atlanta and meet the guys I’ve interacted with down there from the Python community, including Doug Hellmann, whom I’ve worked pretty closely with over the past several months. There’s also a thriving Python community in that area, I hear. Looking forward to it!

Why should I pay for this AWS design decision?

I was writing a utility in Python (using boto) to test/play with Amazon’s SQS service. As boto isn’t particularly well documented where SQS specifically is concerned, I also plan to post some examples (either here or on Linuxlaboratory.org, or both). When I had some trouble getting a message that was sent to a queue, I went to the Amazon documentation, and found this little gem in the Amazon Web Services FAQ

I am sure that my queue has messages, but a call to ReceiveMessage returned none. What could be the problem?

Due to the distributed nature of the queue, a weighted random set of machines is sampled on a ReceiveMessage call. That means only the messages on the sampled machines are returned. If the number of messages in the queue is small (less than 1000), it is likely you will get fewer messages than you requested. If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response. Your application should be prepared to poll the queue until a message is received. Note that with the 2008-01-01 version of Amazon SQS, you’re charged for each request you make, so set your polling frequency with that in mind.

So… if you were planning to decouple application components using SQS using an ‘eventual consistency’ model, keep in mind that they’re using the same model, and that they’re charging you for the privilege of eventually getting the messages you’ve already paid to put there, but aren’t necessarily available at any given point in time. I personally think this is a little goofy, and wrong.

If I put a message in a queue, I should be charged for actually getting the message. I should *not* be charged for checking to see if Amazon’s internal workings have made my messages available to me yet.

O’Reilly: Give your authors Safari access!

UPDATE – 2008-06-23 – A member of O’Reilly’s editing team commented that this privilege has *NOT* been discontinued, and all O’Reilly authors should receive a free Safari account. Thanks a bunch, Mary, for the clarification (see comments for more).

I learned from one of the authors of the recently released second (read: first, squared) edition of High Performance MySQL that O’Reilly apparently did away with the idea of giving O’Reilly book authors free Safari accounts. Lame.

I do not know why in the world they would discontinue this offering for authors. Perhaps they’re not aware, but a great many of the O’Reilly authors are also bloggers. Tech bloggers. Some of them write on the O’Reilly blogs themselves, but almost all of them blog outside of that arena as well. And guess what they blog about? Well, lots of stuff, but there’s plenty of blogging about “something I learned”, or “this book rocks”, etc. Heck, we even blog about products we use — I’ve even blogged about Safari… *today* even!

In a world where people are paid to blog about products, it surprises me that O’Reilly wouldn’t offer people who are already actively blogging in and around their content, and who have actually formally joined the O’Reilly family, the opportunity to become users, and thereby advocates, of their other offerings.

I am an O’Reilly author, and have a free Safari account (we’ll see how long that lasts after this post goes live). I can think of *plenty* of instances where I’ve recommended that people who don’t have an account try to get one, or try to get their employer to get them (or their whole site) an account. I consult (as do TONS of O’Reilly authors), and I’ve also recommended to my clients that they get Safari accounts for their technical staff. Had O’Reilly not offered me the free account, that would never have happened. I am confident that the amount of money grossed by O’Reilly due to my big mouth since 2005 is approaching 6 figures, if it hasn’t exceeded that already.

Not to mention the fact that having the account is a very real, very sincerely felt way to make the authors feel appreciated, because lord knows we don’t write books for the money.

So, Tim, do you think you could find it in your budget to give the guys with probably the most popular MySQL Performance blog (and probably consulting outfit as well) free Safari accounts? Please? If they agree to put some badge on their blog or something? (I’d gladly do that as well).

Here’s to hoping they see the light, fellas.

For anyone flying to a conference – flyer beware

I’m going to OSCON in July, and I know that just about everyone I know who is a participant in this crazy life we call IT (or web 2.0, or whatever it’s called now), is flying to a conference or something in 2008. I’m starting to notice more and more posts like this one, so if you can avoid it, don’t put anything in a checked bag that you can’t afford to lose, and avoid US Airways, and pass it on, because when you see the list of things they don’t cover in their lost baggage policy, you’ll suddenly feel like you’re lucky to still have anything you ever checked with your bags.

Might as Well Use Python

One of my jobs after being promoted to Director of IT at AddThis.com is to find a senior PHP developer. I’ve learned a few things from the search we’ve been doing over the past several months:

  1. What we really want is a senior developer, versed in PHP
  2. There are apparently none of those in existence who are not happily employed already
  3. According to a comparison of resumes received to my own, I am an uber-senior PHP guru.

First, know that I do not bill myself as an uber-senior PHP guru. I know what one looks like, because I worked with many of them while I was Editor in Chief of php|architect magazine some years ago. Those guys are sharp.

Anyway, the story is that we’ve received lots of resumes, most of which were unceremoniously tossed. We interviewed several people, though, and we found a lot of cut-n-paste coders, people whose idea of refactoring is moving from one CMS to another, and people who have only ever programmed in PHP and (I quote) “well, that and HTML”. Another candidate couldn’t tell me what the purpose of a left join was in spite of having database written all over his resume. Data validation, basic SQL constructs, basic security, basic problem solving — almost all of it was missing. We found *one* candidate that fit the bill, and he was manipulated into another position by a rotten recruiter (is that redundant?)

One candidate. We live and operate in the most densely populated state in the nation, sandwiched between two major urban areas (Philly and NYC), and in all of the resumes we received, *one* candidate knew enough PHP, in addition to knowing enough about development and stuff, that we would’ve hired him.

We might just as well use Python, eh? Sure, the market for Python developers doesn’t look very big on the face of things, but there are a few things working in our favor here:

  1. There are a couple of good-sized Python shops in this area (Ask.com, for example, apparently has a large Python dev shop in the area, if Dice is any indication), so there are *some* Python developers local (not that local is a hard requirement).
  2. All of the Python developers I know are developers *first*, Python coders *second*. If Python didn’t exist, they would probably still be developers. They have some generic notion of what a sane program looks like, and how things are typically done, the pros and cons of different techniques and patterns, etc.
  3. While I don’t consider myself senior at it, I happen to know and really like Python. I use it for all of my sysadmin coding needs, and am slowly getting into Django. This brings up the additional point that if work ever slowed for a Python web developer at AddThis, they could probably help me do some other really cool stuff on the back end of things to help me build and scale the next-gen version of AddThis. I would probably not code anything on the back end in PHP.

I thought I was alone in having issues finding a PHP developer, and I had just been plugging along with my search, but I’ve discovered that friends in other parts of the country, as well as in Canada, are having the exact same issues with finding PHP developers as I am. One buddy doesn’t even need someone as senior as I do, but can’t find a qualified candidate. Blech.

If you happen to have missed our job ad and are a senior PHP developer, send your CV to “jonesy” at the domain I work for. If you are a senior Python web developer, do the same.

MySQL Data Sharding Toolkit in Python

This rocks. It’s not complete, but Pyshards is the closest thing I’ve seen to a real attempt at making a more or less generic sharding toolkit, written in Python. This is not just great because it’s written in Python or because it helps people who need sharding capabilities in MySQL. It’s great because having a toolkit to use for this benefits the community by creating a point of reference for how to get things done, and can help unite those who are treading into this territory and help them all get a leg up on this beast that is “sharding”.

I, for one, have found ways (so far) to avoid having to do this. It’s a good bit of complexity for data that would otherwise be very simple, and an infrastructure architecture that would otherwise also be simple (by design). But one of the things that makes sharding seem complex is that there aren’t any standardized tools to aid the admin in setting up, and (worse) maintaining/rebalancing shards.

Run your own cloud

What if you’re a sysadmin that supports a large development community, or a large systems research facility, or something else that requires you to provision systems that, perhaps, *aren’t* cookie-cutter machines? A lot of different solutions actually exist for doing this, with varying degrees of complexity and usefulness. However, in order to give people the highest level of flexibility without making your own life onerous, why not run your own EC2-like cloud?

If you’ve ever used the ROCKS beowulf cluster OS, you know how easy it can be to get a fully-functioning cluster installed in a matter of just a couple of hours, or perhaps even minutes (assuming the hardware is there and working and connected and all of that). If you don’t know, trust me, it’s easy. Really easy. If you’ve installed a linux distro from a CD, you can install a cluster with ROCKS, I promise.

Taking that a step further, Eucalyptus allows you to take your ROCKS cluster and make it into an EC2-like compute cloud. It installs as a ROCKS “roll”.