rrdpy – Thanks, Corey!

I have a somewhat unique situation to deal with in terms of monitoring. I need to put a graph a bunch of historical data mined from web server logs. I can get so far with loghetti, which is coming along and is great for certain things, but there’s a bridge missing between it and something like MRTG. I’m pretty sure that with a custom output plugin and Corey’s rrdpy, I can make it the rest of the way. In fact, I had been poring over the documentation for RRDTool and the various language bindings figuring out which way to go first just as this module was released.

I still may decide to graph the historical data using some generic ‘feed this your data in one big heap and this will chart it’ type of thing, but even that may be possible here, since rrdpy includes rrd_make.py, which may or may not (I haven’t looked yet) support the requisite arguments you need to pass to get historical data to work (I think you need to support a start time, for example).

Ubuntu 8.04 and Python Editors

So I updated one of my laptops to Ubuntu 8.04 pretty much as soon as it was available. I’ve been using my MacBook Pro laptop for everything for probably over a year now, because I grew tired of the hobby that *is* running Linux on a laptop and getting everything to work. I’ll note that I *do* run Linux on every server I maintain that I can think of :)

The first test for this laptop was wireless. I bought this laptop (Lenovo T61) specifically because it got rave reviews for its Linux compatibility. I was careful to order the laptop with the proper video and wireless chipsets that had the best support. However, 2 things annoyed me so much that I went back to the MacBook for everything:

  1. Wireless hung, and hung often, and in a way that it was unrecoverable.
  2. Lenovo put the Escape key in the worst place they could possibly put it, especially for a Vim user. Changing the key mapping caused issues with other apps, and configuring the key mapping inside .vimrc doesn’t help on the 30 other servers I use it on (ssh’d in from this laptop) :-/

Really, it was the wireless that did it. I work 100% remotely on everything I do. So, 8.04 seems to have fixed the wireless issues. The next thing I wanted to do was check out all of the Python IDE/editors I couldn’t use on the Mac (or, not easily). So I used Synaptic Package Manager to install all of the ones I could find. I’m sorry to say that I personally had Problems with most of them:

  • DrPython launched fine, but using the file browser to open a file resulted in…. a no-op. I’m sorry, but an editor needs to be able to open a file.
  • PyPE failed to launch altogether! It looks like it’s going to open, it spins for 5-10 seconds, and then just disappears. No window is ever shown, but a tab does appear in the bar on the bottom of the screen.
  • Pida allows you to choose an external editor, so I chose Vim, and that kinda worked, but I really just want the key bindings, not the whole editor, and there’s no option to use some default built-in editor that has code folding and autocompletion and stuff. It appeared to me to be so close to gvim that I decided to skip it. I tried to stick around and give it a chance by reading the docs, but alas, the only thing under “Help” is “About”. Seems there are still a number of open source developers more concerned with getting credit than getting users.
  • Stani’s Python Editor looked pretty nice, but I couldn’t find any easy way to change the syntax coloring, and while there is a manual, you have to donate to get your hands on it. This is nonsense. If you want to sell some kind of advanced documentation, fine, but you can’t expect me to donate to a project that I don’t even know if I want to use yet! “Please pay me so that you can see if this product fits your needs”…. it just doesn’t work that way. What you’ve done is given me a product that is complex enough that you pretty much need a manual just to get started, and then deprived me of that. Why not just give me the manual and a 30 day trial, after which I have to donate? I’d have no problem doing that if I planned on keeping it around. In fact, that’s how plenty of Mac applications work. I’ll pay for software that does what I need, but this game that’s being played is just offensive.
  • Eclipse with PyDev, I can use this, but I don’t like it a whole lot. The good news is that there’s an SVN plugin (subclipse), and a plugin for vi keybindings if I want to pay for it (it’s only $20 – not bad if you use it a lot). The interface is a little clunky to me, and there’s no easy “change your syntax color scheme to this” type functionality. If you want a dark background and light colored text, you actually have to go to one place to change the background color, the color of the line numbering area, etc., and then go to another place to change the colors associated with the different elements of your particular language. That’s annoying for two reasons: first, it’ll take forever to get things the way I want, and second, if I installed this on another machine, I couldn’t just move over some kind of theme file and have my settings ready to go (as far as I know).

In the end, it looks like my three favorite editors are still Komodo Editor, JEdit, and Vim. What’s your favorite Python editor for Linux?

Python Magazine April Issue is Out

Hi all,

This month’s Python Magazine has been released, containing a few really great articles, including on about using the Google API and Google Spreadsheets to create a database “in the cloud”. For you scientist types, there’s also an excellent article about BioPython. For XCode users, there’s an article about scripting XCode with Python, and there’s also some in-depth coverage of PyTables, which I thought was really interesting. Have a look, and enjoy!

Spring Means Blooming Flowers… and Ideas

I seem to have found a pattern in my own internal workings. In the fall, I work furiously and get a lot done. Around the time of the winter holidays, I almost always do major personal web site changes and upgrades according to a mental list I’ve compiled over the previous year.

In the spring, I shake off the winter (I’m not a fan of winter), I brew my first batch of beer for the season (which symbolizes the end of winter, because I brew outdoors), and my brain starts to be flooded with new ideas. They range from the simplistic (maybe we should consider replacing windows in the house this year), to the slightly odd (why isn’t there a bluetooth setup that pairs two devices and alerts you if they get out of range, so if my daughter strays too far…), to the really useful (I should really take on that woodworking project to build that bookcase we desperately need), to the GEEKY!

This year I seem to be having a lot of geeky ideas. The difference is that, this year, I finally feel empowered enough to go after some of them. One idea that has come up is building an online brewer’s workshop. I would just build a GUI to do this for myself, but then I’d have to deal with which widget set to use, which platforms to support, and whatever else. Also, the final step in the evolution of a lot of GUIs is webification anyway. So I *think* this might be a job for Python, and I *think* I might try to do this using Django, which is fully supported by my web host (finally – see yesterday’s post)!

Brewing is one of those things that you can make as complex as you care to get. I started brewing with a buddy using a Coleman picnic cooler, a few buckets, and some odds and ends from the kitchen. Now I have a full three keg system, with pumps, plate chillers (small plate heat exchangers), fancy false bottoms, cool valves and tubing, and it involves relatively little manual labor. And that complexity can infect recipe development as well. Hops add bitterness by leeching alpha acids into the wort (the liquid that is not yet beer). Hop utilization calculations can be non-trivial and depend on many other factors in your system. Other characteristics depend heavily on the percent of available sugars you’re able to extract from the grains, your ability to keep a mash at a given temperature for a fixed period of time. This is easier to predict if you know, for example, the thermal mass of the vessels involved, and how much heat will be lost when you combine water and grain and stir. There are also proteins at work in the mash which can gum things up enough to make draining the liquid off a chore, so knowing what water/grain ratio to use is also important. And how quickly can you bring wort from boiling down to a temperature more friendly to yeast at the end of the cycle?

That’s a small fraction of the considerations you *could* make when brewing. I didn’t even touch on pH and water characteristics, or yeast attenuation! Needless to say, brewing with any consistency would be a great challenge and take a good bit more preparation without some tool to help you figure out how much water you’ll need, how many ounces of hops for how long, and how much grain you need to mash (and for how long), etc. There are lots of tools to help brewers out with this kind of stuff (ProMash is a popular one). The problem I have is that these tools are mostly commercial, proprietary, platform-specific ventures. I’d like to put one on the web that is at least “good enough”, and free for anyone to use. I’m open source that way (I’m happy to release the source as well).

Another tool I’d love to see is one that would let me manage my consulting business online. If BestPractical’s RT had a good PayPal plugin that would let you charge per ticket or charge for a bundle of so many tickets or something, that’d be a good start, but I’ve mucked with the code for RT (it’s written mostly in Perl), and it wasn’t a pleasant experience. This wouldn’t be a complete solution either, because most of my work is *not* simple support tickets, it’s large projects. For those I’d like people to be able to pay invoices online. There’s lots more I’d like to add on top of that, but that’s the general gist of it, and in the past I’ve been unable to find a really good solution, where “really good” is a completely nebulous term barely defined in my own head. :)

In addition to those ideas, I registered a couple of domains over the past year, and I hope to do some cool things with them as well if I ever get some time away from work and consulting. Oh yeah – I’ll also continue working on loghetti! Keep any eye out for updates. Maybe some people reading this have similar interests and would like to collaborate. Ciao for now!

New web host: check!

If you’re reading this post, it means that the DNS entries have propagated, and you’re viewing my blog at it’s new location: WebFaction!

Thanks to everyone who left comments in response to my cry for web host recommendations. Almost everyone mentioned webfaction in the comments, and I got a few emails with that recommendation as well. After verifying a couple of things with the folks at WebFaction, I signed up, and just about an hour ago I imported my wordpress blog into a new instance of wordpress that was set up for me automatically by WebFaction.

A side benefit of WebFaction over my old host that I hadn’t considered: I get to pick my username and password to use for my SSH login. My old provider assigned the username and password, and they were both long cryptic strings. After about a year with that provider, I had only in the last month or so finally memorized my user name and password.

Ok, I’m off to go enable more plugins and do s’more cleaning up and moving over more domains and stuff. Wish me luck!

Can’t find the Ruby Kool-aid

I picked up the Apress book about Ruby for Systems Administrators, because I plan on learning enough about Ruby to make it a viable tool that I can use if I have to use it at some point. I still plan on doing that, but in reviewing the first couple of chapters, I don’t think I’m going to like Ruby very much, and I think I’ll have to be dragged kicking and screaming to use it. Maybe as I gain some practical experience with it this feeling will subside.

I should say that I did all of my sysadmin programming in shell and perl for many years. I still use them somewhat regularly, if only to maintain existing code or to write code for clients who require Perl, or to work on systems that don’t have Python installed (yes, these exist. Basically, unless you’re using Linux, Python is not “just there”). If I have a choice or some say in the matter, I’ll tend toward Python if I’m writing new code. I’m pretty sure that all of the non-web code that I’m writing for AddThis, for example, is Python (web code is in PHP, and I do database development using straight-up SQL, because we’re using MySQL – if I were using PostgreSQL, there’s a chance I might use Python for that as well).

When I started using Python, I also looked at Ruby. It was one of those periods of time where I was pretty much out looking for a language to add to my repertoire. I had experience with Java, C/C++, PHP, Perl… enough languages that at this point I knew exactly what I wanted in a language: I wanted a language that I could use no matter what I was doing. Of course, I do mostly systems coding, but I wanted a language that would be enjoyable enough to use, and agile enough, and capable enough, and available on enough platforms, and <long list here> that I could use it for web and perhaps even GUI scripting as well (I tend to avoid scripting GUIs in any language because I just think going straight to the web skips an oft unnecessary step in the evolutionary cycle).

Turns out, Python won because it happened to be in heavy use on a couple of projects I was loosely involved in at my job at the time. However, I’ll state that I volunteered to be involved in them so I could get a better grasp of Python, and I’m glad I did. My feeling at that time was that Ruby looked too much like Perl to me, and I was really making an effort to move away from Perl, because after 10 years of coding in a language, you should not still feel like a complete neophyte every time you have to use it.

After more reading about Ruby, I’ve discovered that, due to the fact that Ruby supports things that don’t exist in Perl, yet tries to adhere to a Perl-ish notion that symbols are good visual indicators of various things (though the same symbols mean completely different things between the two), it actually looks even more cryptic to me than Perl does.

This is completely a “doesn’t fit my brain” thing. I’m sure Ruby is a fine language, and if I made regular enough use of it, I’d probably get used to parsing all of the symbols (though the same could be said for Perl and even shell as well). Python has some weirdness to it as well, which also took some getting used to. Unfortunately, I don’t really have a project to use Ruby on right now, and nothing forcing me to use it, and no existing projects that would be easy for me to involve myself in. I guess I’ll just have to find some sysadmin task that needs doing and see how Ruby works out. My primary language, though, is likely to remain Python, as I find it about 60 million times easier to read than Perl, and 100 million times easier to read than Ruby. I learn a lot by reading code, so I guess the ability to parse it readily is of primary importance to me in picking up a new language.

Open Source, Microsoft, and the Future

Last week, I read a great blog post (and even better comments), and then this week I read something else that made me think more about Microsoft, open source, and Microsoft’s direction going into the future.

By way of disclaimer, I’m indifferent to Microsoft. I don’t use their products if I don’t have to, but I understand why they still exist and understand their place in the “technological landscape”. I’m heavily involved in various aspects of various open source communities. I’ve contributed code to a good number of open source projects, I’ve started my own open source projects, and I’ve worked to publish two magazines about two different open source programming languages. I also consult, and stipulate in my contracts that tools developed which contain no site-specific logic are owned by me, and released as open source software on one of my web sites (where I also document some of what I do/have done for anyone who finds it useful). I also wrote a book about a certain open source operating system, and frequently write articles about open source technologies for various web outlets and magazines.

With that out of the way (and so much for “Anonymous Geek” eh?)…

In the comments section of Hal McLeod’s blog post, “how does a software company make money, if all software is free?”, Hal himself asks the question:

“What has got to happen at MSFT if it wishes to remain a relatively happy, successful, interesting company for the next thirty or so years?”

I think that’s a fantastically interesting question. Some in the anti-Microsoft crowd are quick to point out steps they could take that are either disingenuous or naive, or both. If you get past all of that and really try to put yourself in the shoes of a Microsoft executive, it becomes much, much more interesting, and really, quite hard. This is a company that is publicly held. This means that Microsoft has a fiduciary responsibility to their shareholders to grow the company. Maybe putting ourselves in the shoes of a Microsoft exec is to abstract. Let’s try this:

Imagine that your total net worth in life is completely invested in Microsoft’s stock. You now have a vested interest in what Microsoft does going forward. How would you like them to proceed, keeping in mind that the result of their actions has a direct and dramatic effect on your life, and the welfare of your family?

It’s just not a simple question. Anyone who claims to have some silver-bullet, “it’s as simple as x” solution is kidding themselves into thinking that they have anywhere near enough information to make such a call. It’s not likely that their net worth is tied to Microsoft’s stock price ;-)

Certainly, some notion of open source is floating around the minds of the Microsoft brass. I say “some notion” because when you read what people at Microsoft say about open source, the only thing that’s clear is that “open source” in their minds is, at best, some nebulous term that seems to change meaning depending on who the speaker is. Certainly, it doesn’t seem to match what people in the open source community would call “open source”. Which brings me to this post I read from Todd Bishop’s Microsoft Blog, where Ray Ozzie (new chief software architect of Microsoft, taking the reigns from Bill Gates) is quoted (by Todd) as saying:

My position toward open source generally is that it’s a part of the environment. It’s very useful for developers to be able to get the source code to certain things, to modify them. Microsoft fundamentally, as a whole, has changed dramatically as a result of open-source as people have been using it more and more. The nature of interoperability between our systems and other systems has increased. I can tell you from an inside perspective … when you build a new product, immediately you start thinking, how shall this product expose its APIs.

…Open source is a reality. We have a software business that is based on proprietary software. We tactically or strategically, depending on how you look at it, will take certain aspects of what we do and we will open-source them where we believe there is a real benefit to the community and to the nature of the growth of that technology in open-sourcing it. … The bottom line is we believe very much in the quality of Microsoft products and we are an (intellectual-property) based business. But we live in a world together with open-source, and we have to make it possible for you to build solutions, or customers to build solutions, that incorporate aspects of that.

The first thing that stands out to me is that the guy either doesn’t “get” open source, or he’s purposely dumbing down the concept because maybe the audience consists of shareholders or some other group that doesn’t “get” open source. In short, developers are not the only people who benefit from having access to the source code, and developers inspecting source code may have absolutely zero interest in modifying it. But this is not the important part of what Ozzie is saying.

What he’s saying is:

  • We have no choice but to accept the reality that is open source software, and the open source community (to a lesser extent)
  • We’d love to find a way to embrace more of these philosophies if we could find a way to do it without being sued out of existence by our shareholders in a flood of class action suits.
  • That said, where we can add value to a product or technology by opening it up (where that term is defined by MSFT), we’ll do that.
  • Short of that, it looks like we’ll at least have to make it possible to integrate/interface with open solutions to keep from being sued out of existence by other parties in a flood of anti-competition suits.

That’s how it sounded to me, anyway. So how does this speak to the question of what Microsoft needs to do to remain healthy and prosper over the next 30 years? It says a good bit by saying little, and unfortunately, the main answer to the question this leaves us with is this:

MIcrosoft needs to engage more, and more directly, with various communities, including the open source communities around various projects, and understand the people and processes, so that it can at least understand how the processes can help them succeed, even if they are to remain a 100% IP-based, proprietary company — because right now, it seems clear that they don’t understand, and they haven’t asked the right questions of the right people to help them over the hump.

Say what you want about Microsoft, they tend to hire smart people. However, if their product line is any indicator, their internal culture is geared toward production, and creative energy is channeled toward increasing productivity and adding value to the bottom line in as efficient a manner as possible. Can’t land that big client? Well, if we just add a random, arbitrary feature to Word, they’ll upgrade all of their desktops and Office installs. A couple of day’s work, and we land millions of dollars. Great.

I have to believe morale at Microsoft is pretty low. I have to believe that there are a lot of employees who, when asked what they do for a living, used to say, proudly, “I work for Microsoft”, who now say “I’m a programmer”, or worse, “I work for the machine”. Both are sentiments from someone who feels like there’s no creativity left in what they do. They’re just workhorses.

If I had to pick a place to focus on within the internals of Microsoft to help them achieve happiness in 30 years, I’d say focus on the people. Make your people happy in what they do, set their imaginations free to change processes, let them be in charge of how they channel their creativity, and I think Microsoft will be completely floored by the results.

Of course, that’s idealistic, and somewhat disingenuous as well. You can’t just let people do whatever they want whenever they want, but what about a Google-like plan to give workers some designated time to work on personal projects? Maybe it wouldn’t be 20% to start, or maybe it wouldn’t work as a % of time at all, but find some way to allow workers to drive, and to work on things independent of the huge project teams and managerial hierarchies they’re chained to on a daily basis. I’m not talking about hiring more people to work in the research group, I’m talking about parallelizing the research effort and harvesting creative cycles from the entirety of the corporation instead of filtering ideas up through people you hope have some clue what a good idea is. Let the ideas bear themselves out and grow a life of their own. Set up internal communities where people can share their ideas and work together on them regardless of what cell block they’re in.

The business is about software, but the people make the software, so the business is really about the people. Channel some of that creative talent at the top of the managerial tree to figure out how to manage less and give people more freedom. I can’t imagine it wouldn’t pay off.

Need a new web host

Ok, this blog is currently hosted by 1and1.com, and I think that needs to change, which is sad, because up until recently, I’ve been pretty happy with the performance. However, I recently had an issue, and some things came to light about my package that I wasn’t aware of.

First of all, the maximum time a process can run is about 6 seconds. Second of all, the maximum number of processes you can have running at once is 12. When I asked if this was a limitation of my *package*, the answer I got was that it was a limitation of all shared hosting accounts.

This blog started throwing 500 errors some time yesterday. I called support late last night and some lady said that she was running a tool that should fix my issue, and to call back in 30 minutes if the problem persists. Well, I’m sick as a dog, and it was late, so I went to bed. This morning, the problem was still there. I called support 3 times today, got cut off twice, and the third one was the charm — he told me that I was over the process number limit. Of course, he was looking at the error log.

Oh yeah, third of all, I don’t have access to the error log. This is also true of all shared hosting packages.

So, in a nutshell, this is what I need:

  • Access to the access and error logs
  • SSH access to the server
  • A bigger process limit, and perhaps a longer per-process runtime limit.
  • PHP and MySQL
  • email *and* phone support (1and1 only has phone support)
  • ability to host at least 10 domains from the account

Since I’ll likely be moving anyway, I’ll add that I’d also very much like to move to a host that has *REAL SUPPORT* for running Python-powered web sites. Python has truly inspired me to *want* to program, and to write more interesting programs. It’s an empowering language, and I’d like to use it on the web in addition to the work I do with it for my various clients, which is almost 100% systems work. I have yet to write anything in Python that has any kind of graphical interface on it.

All of that said, do not misconstrue that to mean that I’ll go without PHP. I won’t. WordPress uses PHP, and I like wordpress (so sue me). I also do some PHP/MySQL work, and I’d really rather not use more than one web host if I can help it. Well… actually… I already use more than one web host — but I don’t want to increase that number if I can help it.

If you’re using a web host you like that meets these requirements, FIRST verify the error log and process number limitations, and then let me know. Thanks!

Before everyone goes nuts, I WILL NOT use a VPS. I have had clients where I’ve had to work on VPS systems, and I have tons of friends who’ve tried it, and plenty who are still using them. The ones who left had lots of stories, I have lots of my own stories, and the ones still using them will likely not be using them for too much longer, because they’re collecting their own stories as they go along. The bottom line is that they’re more money for less. One VPS I used promised all kinds of great system resources, and I used them only to find that at least half of them were used by the provider to monitor and manage the VPS. The system’s configuration was a nightmare, mish-mosh of convoluted, non-standard nonsense, and although I had a root account, severe limitations were put on what I could do with the VPS. Disgusting. I won’t do it to myself.

Ok, that’s it. It’s “lose”. Not “loose”.

Every. Single. Day. This. Happens.

I read a *LOT* of online technical documentation. Come to think of it, I read a *LOT* of documentation offline as well. I also occasionally read things like blogs and comments and stuff. In all of my reading, I have found that the most prevalent mistake made by the writer in terms of grammar and spelling is using the word “loose” in place of “lose”. So here’s the rule:

“Lose” is a verb, as in “I will lose my job if I do that”, or “Please lose my number”. Other forms include “losing” as in “I’m losing my mind”, and the ever-popular “loser”, as in “That Anonymous Coward is such a loser”.

On the other hand, the word “loose” is an adjective, as in “he’s got a few screws loose” or “that development team has somewhat loose morals – I’ve seen them at conferences”, or “loosely coupled”.

How about this: if you want to describe loss, lose the extra “o” — copyright me, today. :-D