Musings of an Anonymous Geek

Having a Google account is sometimes useful in ways you hadn’t planned for. For example, at a few different employers I’ve been at, I’ve had to prepare for reviews by providing a list of accomplishments to my supervisor. One decent tool for generating this list is email, though it can take some time. Another useful tool is the Web History feature of your Google account.

Though this isn’t necessarily indicative of everything I’ve accomplished in the first half of 2008 per se, it’s definitely indicative of the types of things I’ve generally been into so far this year, and it’s interesting to look back. What does your Web History say?

• Gearman - this is used by some rather large web sites, notably Digg. It reminds me a little of having Torque and Maui, but geared toward more general-purpose applications. In fact, it was never clear to me that PBS/Maui couldn’t actually do this, but I didn’t get far enough into Gearman to really say that authoritatively.
• How SimpleDB Differs from a Relational Database - Links off to some very useful takes on the “cloud” databases, which are truly fascinating creatures, but have a vastly different data management philosophy from the relational model we’re all used to.
• Reblog - I found this in the footer of someone’s blog post. It’s kinda neat, but to be honest, I think you can do similar stuff using the Flock browser.
• Google Finance APIs and Tools - did I ever mention that I had a Series 7 & 63 license two months after my 20th birthday? I love anything that I can think for very long periods of time about, where there’s lots and lots and LOTS of data to play with, where you can make correlations and answer questions nobody even thought to ask. Of course, soon after finding this page I found the actual Google Finance page, which answers an awful lot of potential questions. The stock screener is actually what I was looking to write myself, but with the data freely available, I’m sure it won’t be long before I find something else fun to do with it. I’m not a fan of Google’s “Feeds” model, but I’ve dealt with it before, and will do it again if it means getting at this data.
• Bitpusher - it was recommended to me as an alternative to traditional dedicated server hosting. Worth a look.
• S3 Firefox Organizer - This is a firefox plugin that provides an interface that looks a lot like an FTP GUI or something, but allows you to move files to and from “buckets” in Amazon’s S3 service.
• Boto - A python library for writing programs that interact with the various Amazon Web Services. It’s not particularly well-documented, and it has a few quirks, but it is useful.
• OmniGraffle - A Visio replacement for Apple OS X. I like it a lot better than Visio, actually. It has tons of contributed templates. You shouldn’t have any trouble making the switch. A little pricey, but I plunked down the cash, and have not been disappointed.
• The Python Queue Module according to Doug - Doug Hellmann’s Python Module of the Week (PyMOTW) should be published in dead tree form some day. I happen to have some code that could make better use of queuing if it were a) written in Python, and b) used the Queue module. I was a little put off by the fact that every single tutorial I found on this module assumed you wanted to use threading, which I actually don’t, because I’m not smart enough…. though the last person I told that to said something to the effect of “the fact that you believe that means you’re smart enough”. Heh.
• MySQL GROUP modifiers - turns out this isn’t what I needed for the problem I was trying to solve, but the “WITH ROLLUP” feature was new to me at the time I found it, and it’s kinda cool.
• Wordpress “Subscribe to Comments” plugin - Baron suggested that it would be good to have this, and I had honestly not even thought about it. But looking around, this is the only plugin of its kind that I found, and it’s only tested up to WP 2.3x, and I’m on 2.5x. This is precisely why I hate plugins (as an end user, anyway. Loghetti supports plugins)
• Lifeblogging - I had occasion to go back and flip through some of the volumes of journals I’ve kept since age 12, wondering if it might be time to digitize those in some form. I might digitize them, but they will *not* be public I don’t think. Way too embarrassing.
• ldapmodrdn - for a buddy who hasn’t yet found all of the openldap command line tools. You can’t use ‘ldapmodify’ (to my knowledge) to *rename* an entry.
• Django graphs - I haven’t yet tried this, because I’m still trying to learn Django in what little spare time I have, but it looks like there’s at least some effort towards this out there in the community. I have yet to see a newspaper that doesn’t have graphs *somewhere* (finance, sports, weather…), so I’m surprised Django doesn’t have something like this built-in.
• URL Decode UDF for MySQL - I’ve used this. It works really well.
• Erlang - hey, I’m game for anything. If I weren’t, I’d still be writing all of my code in Perl.
• The difference between %iowait in sar and %util in iostat - I use both tools, and wanted the clarification because I was writing some graphing code in Python (using Timeplot, which rocks, by the way), and stumbled upon the question. Google to the rescue!
• OSCON ‘08 - I’m going. Are you going? I’m also going to the Oregon Brewers Festival on the last day of OSCON, as I did in ‘06. Wonderful!
• Explosion at one of my hosting providers - didn’t affect me, but… wow!
• hypertable - *sigh* someday…when there’s time…
• Small-scale hydro power - Yeah, I’m kind of a DIYer at heart. I do some woodworking, all my own plumbing, painting, flooring, I brew my own beer, I cook, I collect rain in big barrels, power sprinklers using pool runoff to give my lawn a jumpstart in spring… that kind of stuff. One day I noticed water coming out of a downspout fast enough to leap over one of my rain barrels and thought there must be some way to harness that power. Sadly, there really isn’t, so I did some research. It’s non-trivial.
• You bet your garden - I also do my own gardening and related experiments.
• RightScale Demo - WATCH YOUR VOLUME - a screencast showing off RightScale’s features. Impressive considering the work it would take me, a lone admin, to set something like this up. The learning curve involved in effectively/efficiently managing/scaling/monitoring/troubleshooting EC2 is non-trivial.
• Homebrew Kegerator - Maybe if this startup is bought out I can actually afford this thing to put my homebrewed beer in. The 30-year-old spare fridge in the basement is getting a little… gamey.
• The pound proxy daemon - I use this. It works well enough, but I’ve crashed it under load, too. I’ve also had at least one hosting provider misconfigure it on my behalf, and I had to go and tell them how to fix it :-/
• Droid Sans Mono - a fantastic coding font. Installing this font is in my post-install routine for all of my desktops.
• Generator tricks for systems programmers - David Beazley has made available a lot of Python source code and presentation slides from what I imagine was a great talk (if you’re a systems guy, which I am).
• The Wide Finder Saga - I found this just as I was writing Loghetti. There are still some things in Mr. Lundh’s code that I haven’t implemented, but it was a fantastic lesson.
• Using gnu sort for IP addresses - I’ve used sort in a lot of different ways over the years… but not for IP addresses. This is a nice hack for pulling this off with sort, but it doesn’t scale very well when you have millions of them, due to the sort utility’s ‘divide and conquer’ method of sorting.
• Writing an Hadoop/MapReduce Program in Python - this got me over the hump.
• Notes on using EC2/S3 - This got me over some other small humps
• BeautifulSoup - found while searching for the canonical way to screen scrape with Python. I’d done it a million times in Perl, and you can do it with httplib and regex and stuff in Python if you want, but this way is at least a million times nicer.

Well, that’s a decent enough summary I guess. As you can see, I’ve been doing a good bit of Python scripting. Most of my code these days is written in Python instead of Perl, in part because I was given the choice, and in part because Python fits my brain and makes me want to write more code, to push myself more. I’ve also been dealing with things involving “cloud” computing and “scalability” — like Hadoop, and EC2/S3. I haven’t done as much testing of the Google utility computing services, but I’ve used their various APIs for some things.

So what’s in your history?

Steve Jobs is one of the best presenters you could ever hope to see. He’s great at tapping into that part of your brain that makes you just want whatever it is he’s holding. But this time, it was a little different.

You see, I already have an iPhone. I bought one in February. I didn’t buy the very first model that came out because it was lacking some stuff that was really important to me - most notably, it only had Apple apps, which was severely limiting, and IMAP support was limited to Yahoo! accounts, which was absurd. With those two obstacles out of the way, I found it useful enough to spend my employer’s money on, but not my own. In the end, it was a business decision, and I still think an iPhone is a better deal than a Blackberry hands down. Especially the new 3G, which has addressed some “enterprise” concerns. Thing is, I don’t care about any of that.

I want some really really really simple things that I haven’t heard anything about, and I want one thing that is perhaps slightly harder but essential.

The slightly-harder-but-essential thing is voice commands. I can hardly believe that were on the third generation of a phone without having voice commands. You can get a $30 Nokia made in 2002 that has voice commands for crying out loud. Without voice commands, it’s unclear to me how this phone is useful in a hands-free environment at all. Have I missed a feature somewhere? Is there something I can plug my iPhone into while I’m driving that will parse the voice commands and do the right thing with the iPhone? I know that if you have an Acura TL the car itself parses the voice commands, but I don’t know if there’s some generic thing that *doesn’t* cost $40k that’ll do the same basic job? Anyone?

The other stuff consists mainly of small application features:

• The ability to bookmark or otherwise somehow save “Directions” in the Maps application. This way, if I’m driving, following the directions in Maps, and need to search for a gas station or coffee shop, I don’t then have to go back and punch in the information again to get my directions back.
• Why the heck doesn’t mail let you read in landscape mode?!?!?!
• I’d *REALLY* like to be able to send and receive photos in text messages. I don’t use it often, but when you need it, you need it.
• The ‘.com’ shortcut should pretty much *always* be visible on the keyboard.
• Make email alerts a per-account setting instead of only alerting for the default or for all accounts or whatever it is the iphone does now. Let me treat email accounts like phone contacts and assign different alert settings for each account just like I set different ring tones for different contacts.
• Let me bookmark phone numbers so I can just hit a button on my home screen to dial them (in the absence of voice commands).
• Make the bluetooth support do some neat trick that’ll make it be actually worth turning on.
• I do a lot of system administration, and I’d love a usable, locally-installed ssh client that I don’t have to perform surgery to install. I don’t want to hack my phone, really. I also refuse to use a web interface to access an ssh client. If you’re doing that, stop right now, and go change every password you have.

On the development front, it would actually be really nice if they supported maybe *one* parsed scripting language for iPhone development. Even if they did like AppEngine and provided a somewhat stripped version of Python it would be something I could use. But that’s a rant for another day.

I’ve been working with what I used to call “utility computing” tools for about 6-9 months. However, for about the past 2 months, I’ve been seeing the term “cloud computing” all over the place, and there is so much buzz surrounding it that it’s reaching that magical point best described using Alan Greenspan’s words: “Irrational Exuberance”.

When Alan Greenspan used those words to describe the attitudes of investors toward the markets, what he was basically saying was that there were people who didn’t really know what they were doing, putting more money than they ought, into things they knew relatively little about. Further, he was saying that the decisions people were making with regards to where to put their money were a) bad, or at least b) not based on sound reasoning, or the ‘facts on the ground’.

This, I think, is where we are at with “cloud computing”. The blog post that put me over the edge is this one, for the record. I read Sean’s writings often enough, but this one strikes me as being a little off, a little sensationalistic, not based in reality, and a little misleading.

Maybe he just didn’t put enough qualifiers in there. His post might make more sense if he limited its scope and provided more facts, but I guess it’s just an opinion piece so he decided not to go that route, and that’s his prerogative I guess.

By limiting the scope, I mean he should’ve realized that there are millions of web sites currently scaling quite nicely without the use of cloud computing. In addition, some of the new ones that are having issues are also not using cloud computing, and when they hit bumps in the road, they make it through, and the great thing is that they also share their stories, and those stories indicate that a cloud (or, the current cloud offerings) wouldn’t have helped much (there’s lots of other evidence of that too). What would’ve helped is if they had paid more attention to:

• monitoring
• initial infrastructure design
• their own app code and app design

My brain has a set of rules that software project websites get tested against. Each time a project site fails to comply with a rule, I get ever-so-slightly more annoyed, and ever-so-slightly less likely to use the software in question (if there are alternatives, this is even maybe not so “slightly”). 

I thought I’d list these rules because I suspect others are like me: we’re extremely busy, we work too many hours, and are involved with too many projects to spend hours trying to figure out what some piece of code someone mentioned once in IRC actually does. 

But first, know that this site actually complies with just about every single rule there is, so it’s a great template to work from if your site needs brushing up. 

• First and foremost, tell me, right away, what this thing does, the problem it solves, and (at a high level) the approach taken to solve the problem. 
• Tell me the language it’s written in. If it’s open source, and it’s written in a language I hack in, *and* it solves a problem I need solved, maybe I can help out, or be encouraged that if something flakes, I can fix it, or at least speak the developer’s language if I have to describe the issue to the folks upstream. 
• Tell me what OS is required, and preferably what OS/version is tested with. 
• Give me a full list of dependencies with links to go get them, or give me a link to “Dependencies”, or to an install document that lists them. 
• Tell me the current version, and the date it was released. Beta versions and dates are nice too. If there is a timed release schedule, tell me that. 
• Keep the information up-to-date. I shouldn’t have to wonder if your software is going to work under OS X 10.5 or RHEL 5, or if your plugin will work under the latest version of Drupal/Django/Moodle/MySQL/Joomla/Firefox…
• BONUS: a very simple architectural drawing that shows me exactly what components make up the whole. The one for CouchDB is as good as any I’ve ever seen (assuming it’s accurate). 
• BONUS: if screenshots are applicable, use them. They communicate a million times more information using a million times less real estate and bandwidth. They can communicate things you didn’t even know you were communicating. Of course, that could be good or bad, but it keeps you honest, and customers like that :-) 

UPDATE: if anyone knows of a non-broken syntax highlighting plugin for wordpress that supports bash or some other shell syntax, let me know :-/

Apache logs, database backups, etc., on busy web sites, can get large. If you rotate logs or perform backups regularly, they can get large and numerous, and as we all know, large * numerous = expensive, or rapidly filling disk partitions, or both.

Amazon’s S3 service, along with a simple downloadable suite of tools, and a shell script or two can ease your life considerably. Here’s one way to do it:

1. Get an Amazon Web Services account by going to the AWS website.
2. Download the ‘aws’ command line tool from here and install it.
3. Write a couple of shell scripts, and schedule them using cron.

Once you have your Amazon account, you’ll be able to get an access key and secret key. You can copy these to a file and aws will use them to authenticate operations against S3. The aws utility’s web site (in #2 above) has good documentation on how to get set up in a flash.

With items 1 and 2 out of the way, you’re just left with writing a shell script (or two) and scheduling them via cron. Here are some simple example scripts I used to get started (you can add more complex/site-specific stuff once you know it’s working).

The first one is just a simple log compression script that gzips the log files and moves them out of the directory where the active log files are. It has nothing to do with Amazon web services. You can use it on its own if you want:

#!/bin/bash

LOGDIR='/mnt/fs/logs/httplogs'
ARCHIVE='/mnt/fs/logs/httplogs/archive'
cd $LOGDIR
if [ $? -eq 0 ]; then
for i in `find . -maxdepth 1 -name "*_log.*" -mtime +1`; do
gzip $i
done

mv $LOGDIR/*.gz $ARCHIVE/.
else
echo "Failed to cd to log directory"
fi

Before launching this in any kind of production environment, you might want to add some more features, like checking to make sure the archive partition has enough space before trying to copy things to it and stuff like that, but this is a decent start.

The second one is a wrapper around the aws ’s3put’ command, and it moves stuff from the archive location to S3. It checks a return code, and then if things went ok, it deletes the local gzip files.

#!/bin/bash

cd /mnt/fs/logs/httplogs/archive
for i in `ls *.gz`; do
s3put addthis-logs/ $i
if [ $? -eq 0 ]; then
echo "Moved $i to s3"
rm -f $i
continue
else
echo "Failed to move $i to s3... Continuing"
fi
done

I wish there was a way in aws to check for the existence of an object in a bucket without it trying to cat the file to stdout, but I don’t think there is. This would be a more reliable check than just checking the return code. I’ll work on that at some point.

Scheduling all of this in cron is an exercise for the user. I purposely created two scripts to do this work, so I could run the compression script every day, but the archival script once every week or something. You could also write a third script that checks your disk space in your log partition and runs either or both of these other scripts if it gets too high.

I used ‘aws’ because it was the first tool I found, by the way. I have only recently found ‘boto‘, a Python-based utility that looks like it’s probably the equivalent of the Perl-based ‘aws’. I’m happy to have found that and look forward to giving it a shot!

This is *not* a ripoff of xkcd (though I read that regularly, and so should you) - this is seriously the best graphic I can come up with, and it does the job. Yesterday I looked at doing all kinds of stuff to my iPhone. I wanted to see if I could get Python and a full-fledged Django installation on my iPhone and create the first web 2.0 application created completely from the bathroom. Just kidding, but I wanted to do some pretty evil stuff. Turns out that, as of now, there isn’t a clean simple way to get a lot of stuff to work without hacking the iPhone in some way, or resorting to things that are completely ridiculous. I’m sorry, but there has to be an easier way to get SSH and a terminal on there. It’s just not that critical right now, and this thing was damned expensive.

I decided to wait it out and see what comes our way over the summer.

Here’s the email I received on Saturday from The Planet, where I have some dedicated servers hosted:

Dear Valued Customers:
This evening at 4:55 in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.
This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.
There is no impact in any of our other five data centers.
I am sorry that this accident has occurred and apologize for the impact.

That’s pretty rough. Lucky for me, nothing I do is solely dependent on any of those machines. However, I think it’s probably pretty common for startups to rely heavily on what amount to single points of failure due to a lack of funds or manpower/in-house skills to set things up in a way that looks something like “best practices” from an architecture standpoint.

Building a startup on dedicated hosting is one thing. Running a production web site at a single site, perhaps with a single machine hosting each service (or even a few or *all* of the services) is something dangerously different. However, building a failover solution that spans outages at the hosting-provider level can also be quite difficult and perhaps expensive. You have to really come up with hard numbers that will help you gauge your downtime tolerance against your abilities and budget. While a restrictive budget might mean less automation and more humans involved in failover, it can be done.

What kinds of cross-provider failover solutions have saved your bacon in the past? I’m always looking for new techniques in this problem domain, so share your ideas and links!

Startups are pretty fascinating. I work for a startup, and one of my good friends works for another startup. I’ve also worked for 2 other startups, one during the first “bubble”, and another one a few years later. Oh my, how the world of web startups has changed in that time!

1999: You must have funding

The first startup I was ever involved in was a web startup. It was an online retailer. They were starting from nothing. My friend (a former coworker from an earlier job) had saved for years to get this idea off the ground. He was able to get a few servers, some PCs for the developers he hired, and he got the cheapest office space in all of NYC (but it still managed to be a really cool space, in a way that only NYC can pull off), and he hosted every single service required to run the web site in-house. If I recall correctly, he had a web and database server on one machine, and I believe the primary DNS server was on an old desktop machine he had laying around the house. This gave him the ability to build the completely, 100%-functional prototype, and use it to shop for funding.

It worked. They got funding, they bought more and bigger servers. They got UPSes for them (yay!), they got more cooling, a nicer office, and they launched the site, pretty much exactly as the prototype existed, and things were pretty stable. Unfortunately, the VCs who took seats on the board after the first round of financing didn’t understand the notion of “The Long Tail”, so they eventually went under, but that’s not the point.

The point is, that was 8 or 9 years ago. It costed him quite a good bit of his hard-earned savings just to get to a place where he could build a prototype. A prototype! He only really knew Microsoft products, and buying licenses for Microsoft SQL Server, and the developer’s tools (I forgot what they were using as an IDE, but they were a ColdFusion shop) was quite a chunk of money. My friend really only had enough money to put together a prototype, and they were playing “beat the clock” — trying to get a prototype done, and shop for (and get) funding, before the money ran out, because they couldn’t afford the hardware, power, cooling, big-boy internet connection, and the rest of what goes into a production infrastructure. The Prototype->VC->Production methodology was pretty typical at the time.

2003: Generate Some Revenue

In 2003, a couple of years after the bubble burst, I was involved in another startup. This one was 100% self funded, but has been rather successful since. By this time, dedicated hosting was just affordable enough that it was doable for a startup that had some revenue being generated, and that’s what my friend did. He also outsourced DNS completely (through his registrar, if memory serves), but he still hosted his own email, backup, and some other services in-house. He had plenty of hiccups and outages in the first year, but overall it ran pretty well considering all of the things he *didn’t* have to be concerned with, like power, cooling, internet uplinks, cable trays, etc. The world was becoming a friendlier place for startups.

2008: Do it all, on the cheap

Nowadays, the world is a completely different place for startups, and a lot of this is due to the rich set of free (or very cheap) resources available to startups that make it possible for them to do a production launch without the VC funding that used to be required just to get the hardware purchased.

In 2008 you can outsource DNS for relatively little money, and it’ll have the benefit of being globally distributed and redundant beyond what you’re likely to build yourself. You can get Google Apps to host your email and share calendars and documents. You can store backups on Amazon’s S3. You can use something like Eclipse, Komodo Edit, or a number of language-specific environments like Wing IDE or Zend Studio to do “real development” (whatever your definition of that is) for free or relatively cheap. You can also get a free database that is reasonably well-stocked with features and capabilities, a free web server that runs 65%+ of the internet sites in existence, and if you have the know-how (or can get it), you can actually host anything you want, including your entire production infrastructure (within reason, and subject to some caveats) on Amazon’s EC2, for a cost which is tied to what you use, which is cheaper in a lot of cases than either buying or leasing a dedicated server. Multisourcing has arrived!

In looking at this progression from “you must have funding”, to “you’re going to need to generate a little revenue”, to “do it all, on the cheap”, the really obvious question this all raises is:

“Now what?”

Well, this whole 2008 situation is making things better, but… how do I put this… “It’s not soup yet”.

First of all, there is no single platform where you can realistically do everything. Google’s AppEngine is nice, but it has its limitations, for example, you don’t have any control over the web servers that run it, so you can’t, say, add an Apache mod_rewrite rule, or use a 301 redirect, or process your log files, etc. Troubleshooting this application based solely on input from people who are having issues with your app would be difficult.

Amazon’s service gives you far more control, and if you need it, that’s great, but it completely changes how you architect a solution. I think that some of these things are good changes, and are things we should all be thinking about anyway, but Amazon forces you to make decisions about how to plan for failure from the moment you decide to go this route — even if it’s for prototyping, because until persistent storage on EC2 is a reality available to the entire user base, whenever an EC2 instance disappears, so does every bit of the data you added to it. You’ll have to start from scratch when you bring up another instance. You’re also going to have to add more scripts and utilities to your toolbelt to manage the instances. What happen when one disappears? How do you fail over to another instance? How can you migrate an IP address to the running instance from the failed one? How you do all of these things, in addition to just building and installing systems, is different, and that means learning curve, and that means overhead in the form of time (and expense, since you have to pay for the Amazon services to use them to learn on).

There are also now “grid” solutions that virtualize and abstract all of your infrastructure, but give you familiar interfaces through which to manage them. One that I’ve used with some success is AppLogic, but other services like GoGrid and MediaTemple have offerings that emphasize different aspects of this niche “Infrastructure-as-a-service” market. Choose very carefully, and really think about what you’ll want to do with your infrastructure, how you want to manage it, monitor it, in addition to how you’ll deliver your application, and also think about how you’ll be flexible within the confines of a grid solution before you commit, because the gotchas can be kind of big and hairy.

None of these are whole solutions. However, any of them could, potentially, some day, become what we would now call a “perfect solution”. But it still wouldn’t be perfect in the eyes of the people who are building and deploying applications that are having to scale into realms known seemingly only inside some brain vault that says “Google” on it. What those of us outside of that vault would like is not only Google-like scalability, but:

• global distribution, without having to pledge our souls in exchange for Akamai services. It’s great that I can build an infrastructure on EC2 or GoGrid, but I’d like to deploy it to 10 different geographic locations, but still control it centrally.
• the ability to tightly integrate things like caching distribution network services with the rest of our infrastructure (because CDNs are great at serving, but not so much at metrics)
• SAN-like (not NFS-like) access to all storage from any/all systems, without sacrificing the IO performance needed to scale a database properly.
• As an admin, I want access to all logs from all services I outsource, no matter who hosts it. I don’t believe I can access, for example, our Google Apps logs, but maybe I’ve forgotten to click a tab somewhere.
• A *RELATIONAL* database that scales like BigTable or SimpleDB

There’s more to it than this, even, but I’ve stopped short to make a point that needs making. Namely, that these are hard problems. These are problems that PhD candidates in computer science departments do research on. I understand that. The database issue is one that is of particular interest to me, and which I think is one of the hardest issues (not only because of its relationship to the storage issue, by the way). Data in the cloud, for the masses, as we’ve seen, involves making quite a few rather grandiose assumptions about how your schema looks. Since that’s not realistic, the alternative is to flatten the look of the data, and take a lot of the variables out of the equation, so they don’t have to make *any* assumptions about how you’ll use/organize the data. “Let’s make it not matter”. Genius, even if it causes me pain. But I digress…

The idea here is just to give some people a clue what direction (I think) people are headed in.

These are also very low-level wants. At a much, much, much higher level, I’d like to see one main, major thing happen with all of these services:

• Get systems administrators involved in defining how these things are done

I’m not saying that because I want everything to stay the same and think a system administrator will be my voice in that or something. I do *NOT* want things to stay the same, believe me. I’m saying it because it seems pretty obvious to me that the people putting these things together are engineers, and not systems administrators. Engineers are the people you call when you want to figure out how to make something that is currently 16GB fit into 32MB of RAM. They are not the people you call when you want to provide a service/interface/grid/offering/whatever that allows systems folks to build what amounts to a company’s IT infrastructure on a grid/instance/App/whatever.

Here’s a couple of examples:

When I first launched an AppLogic grid, a couple of things jumped out at me. The partitions on the components I launched were 90% full upon first boot, they had no swap partition, and there was no consistency between OS builds, so you can’t assume that a fix on one machine can be blown out via dsh or clusterssh to the rest. The components were designed to be as small as possible, so as to use as little of the user’s alotted resources as possible. In addition, mount points created in the GUI management interface and then mapped to a component… don’t cause the component to have any clue what you just did, which raises the question “umm… why did I bother using the GUI to map this thing to this component if I just have to edit /etc/fstab and mount it in the running instance myself anyway? Back to consistency, this is unlike if you had, say, allocated more RAM or storage, or defined a new network interface on the device in the GUI.

There is no part of EC2 or S3 that looks like a sysadmin was involved in that. It’s a programmer’s platform, from what I can tell. For programmers, by programmers. Luckily, I have enough background in programming that I kind of “get it”, but while I might be able to convince myself that there are parallels between how I approach programming and building infrastructures, it still represents a non-trivial context switch for me to move from working deeply at one to working deeply at the other, so mixing the two as a necessity for moving forward is less than nice.

There is no “database in the cloud” service that looks remotely like there was a database systems person involved at all, that I can tell. I’ll confess to not having used BigTable or SimpleDB, but the reason is because I can’t figure out how to make it useful to me at the moment. These tools are not relational, and my data, though it’s been somewhat denormalized for scaling purposes (compromises taken carefully and begrudgingly - I’d rather change database products, but it’s not in the cards), is nonetheless relational. I’ve considered all kinds of object schemes for data in the past, and I still think that there’s some data for which that can work well, but it’s not currently a solution for me. Once you look at the overhead in managing something like EC2, S3, AppLogic, etc., the very last thing you need is the overhead of a changing data storage/manipulation paradigm.

Should I be hiring systems folks, or developers? Both? Ugh. Just when I thought you could finally get away with building a startup with nothing more than an idea, a sysadmin and a coder, here they go roping me back into hiring a team of developers… to manage the systems… and the data. No good (and I mean *NO GOOD*) can come of developers managing data. I know, I’ve seen ‘em do it.

All of that said, I use all of this stuff. Multisourcing is here to stay - at least until someone figures a whole bunch of stuff out to make unisourcing a viable alternative for systems folks, or they collectively redefine what a “systems person” is, which is an extremely real possibility, but is probably quite a ways off. My $.02. Flame at will

If you’re reading this post, it means that the DNS entries have propagated, and you’re viewing my blog at it’s new location: WebFaction!

Thanks to everyone who left comments in response to my cry for web host recommendations. Almost everyone mentioned webfaction in the comments, and I got a few emails with that recommendation as well. After verifying a couple of things with the folks at WebFaction, I signed up, and just about an hour ago I imported my wordpress blog into a new instance of wordpress that was set up for me automatically by WebFaction.

A side benefit of WebFaction over my old host that I hadn’t considered: I get to pick my username and password to use for my SSH login. My old provider assigned the username and password, and they were both long cryptic strings. After about a year with that provider, I had only in the last month or so finally memorized my user name and password.

Ok, I’m off to go enable more plugins and do s’more cleaning up and moving over more domains and stuff. Wish me luck!

I picked up the Apress book about Ruby for Systems Administrators, because I plan on learning enough about Ruby to make it a viable tool that I can use if I have to use it at some point. I still plan on doing that, but in reviewing the first couple of chapters, I don’t think I’m going to like Ruby very much, and I think I’ll have to be dragged kicking and screaming to use it. Maybe as I gain some practical experience with it this feeling will subside.

I should say that I did all of my sysadmin programming in shell and perl for many years. I still use them somewhat regularly, if only to maintain existing code or to write code for clients who require Perl, or to work on systems that don’t have Python installed (yes, these exist. Basically, unless you’re using Linux, Python is not “just there”). If I have a choice or some say in the matter, I’ll tend toward Python if I’m writing new code. I’m pretty sure that all of the non-web code that I’m writing for AddThis, for example, is Python (web code is in PHP, and I do database development using straight-up SQL, because we’re using MySQL - if I were using PostgreSQL, there’s a chance I might use Python for that as well).

When I started using Python, I also looked at Ruby. It was one of those periods of time where I was pretty much out looking for a language to add to my repertoire. I had experience with Java, C/C++, PHP, Perl… enough languages that at this point I knew exactly what I wanted in a language: I wanted a language that I could use no matter what I was doing. Of course, I do mostly systems coding, but I wanted a language that would be enjoyable enough to use, and agile enough, and capable enough, and available on enough platforms, and <long list here> that I could use it for web and perhaps even GUI scripting as well (I tend to avoid scripting GUIs in any language because I just think going straight to the web skips an oft unnecessary step in the evolutionary cycle).

Turns out, Python won because it happened to be in heavy use on a couple of projects I was loosely involved in at my job at the time. However, I’ll state that I volunteered to be involved in them so I could get a better grasp of Python, and I’m glad I did. My feeling at that time was that Ruby looked too much like Perl to me, and I was really making an effort to move away from Perl, because after 10 years of coding in a language, you should not still feel like a complete neophyte every time you have to use it.

After more reading about Ruby, I’ve discovered that, due to the fact that Ruby supports things that don’t exist in Perl, yet tries to adhere to a Perl-ish notion that symbols are good visual indicators of various things (though the same symbols mean completely different things between the two), it actually looks even more cryptic to me than Perl does.

This is completely a “doesn’t fit my brain” thing. I’m sure Ruby is a fine language, and if I made regular enough use of it, I’d probably get used to parsing all of the symbols (though the same could be said for Perl and even shell as well). Python has some weirdness to it as well, which also took some getting used to. Unfortunately, I don’t really have a project to use Ruby on right now, and nothing forcing me to use it, and no existing projects that would be easy for me to involve myself in. I guess I’ll just have to find some sysadmin task that needs doing and see how Ruby works out. My primary language, though, is likely to remain Python, as I find it about 60 million times easier to read than Perl, and 100 million times easier to read than Ruby. I learn a lot by reading code, so I guess the ability to parse it readily is of primary importance to me in picking up a new language.