Comments (6)

By m0j0, October 22, 2009 9:02 pm

New Baby!

I know this is my geek blog, but geeks have kids too, so first I want to announce the birth of our second daughter, Sadie, who was born on September 15th. She’s now over a month old. This is the first time I’ve stayed up late enough to blog about her. Everyone is healthy, if slightly sleep-deprived

New Job!

The day before Sadie’s birth, I got a call with an offer for a job. A *full-time* job, as a Senior Operations Developer for MyYearbook.com. After learning about the cool and very geeky things going on at MyYearbook during the interview process, I couldn’t turn it down. I started on October 5, and it’s been a blast digging into all of the cool stuff going on there. While I’m certainly doing my fair share of PHP code review, maintenance, and general coding, I’m also getting plenty of hours in working out the Python side of my brain. I’m finding that while it’s easier switching gears than I had anticipated, I do make some really funny minor syntax errors, like using dot notation to access object attributes in PHP ;-P

What I find super exciting is something that might turn some peoples’ stomachs: at the end of my first week, I sat back and looked at my monitors to find roughly 15 tabs in Firefox open to pages explaining various tools I’d never gotten to use, protocols I’ve never heard of, etc. I had my laptop and desktop both configured with 2 virtual machines for testing and playing with new stuff. I had something north of 25 terminal windows open, and 8 files open in Komodo Edit.

Now THAT, THAT is FUN!

The projects I’m working on run the gamut from code cleanups that nobody else has had time to do (a good tool for getting my brain wrapped around various parts of the code base), to working on scalability solutions and new offerings involving my background in coding *and* system administration. It’s like someone cherry-picked a Bay Area startup and dropped it randomly 30 minutes from my house.

My own business is officially “not taking new clients”. I have some regular clients that I still do work for, so my “regulars” are still being served, but they’ve all been put on notice that I’m unavailable until the new year.

New Car!

I’m less excited about the new car, really. I used to drive a Jeep Liberty, and I loved it. However, in early September, before Sadie’s arrival, it became clear to me that putting two car seats in that beast wasn’t going to happen. The Jeep is great for drivers, and it has some cargo space. It’s not a great vehicle for passengers, though.

At the same time, I was running a business (this was before the job offer came along), and I was finding myself slightly uncomfortable delivering rather serious business proposals in a well-used 2003 Jeep. So, I needed something that could fit my young family (my oldest is 2 yrs), and that was presentable to clients. So, I got a Lexus ES350.

I like most things about the car, except for the audio system. It seems schizophrenic to me to have like 6 sound ‘zones’ to isolate the audio to certain sets of speakers, but then controls like bass and treble only go from 0 to 5. Huh? And the sound always sounds like it’s lying on the floor for some reason. It’s not at all immersive. The sound system on my Jeep completely kicked ass. I miss it. A lot.

Other News

I’ve submitted an article to Python Magazine about my (relatively) recent work with Django and my (temporarily stalled) overhaul of LinuxLaboratory.org, and my experiences with various learning resources related to Django. If you’re looking to get into Django, it’s probably a good read.

I’ve been getting into some areas of Python that were previously dark, dusty corners, so hopefully I’ll be writing more about Python here, because writing about something helps me to solidify things in my own brain. Short of that, it serves as a future reference point in case it didn’t get solidified enough

My sister launched The Dance Jones, a blog where she talks about fitness, balance, dance, and stuff I should probably pay much more attention to (I’m close to declaring war on my gut). Also, if you ever wanted to know how to shoulder shimmy (and who hasn’t wanted to do that?), you should check it out

Comments (6)

By m0j0, September 3, 2009 2:55 pm

I recently upgraded my primary workhorse (a MacBook Pro) to Snow Leopard. Before I did, I decided to go through and take stock of all of the documents and software I’d accumulated. While I was doing this, I simultaneously got into a conversation with a buddy of mine about the software he uses on his Macs. Turns out he maintains a whole page devoted to (mostly non-geek, but still somehow geeky) Mac software he uses.

I decided to go ahead and list the software I use for stuff whether it was geeky or not. Then I realized that pretty much all of the software I use is kinda geeky. I guess if you’re someone who’s going to create a list of software you use, it’s pretty hopeless.

So… here’s what I’m using. Suggestions welcome in the comments!

Social Media

• Tweetie
• Twitterlocal
• Tweetdeck
• TweetGrid
• Brightkite (iPhone)
• LinkedIn

My Twitter account updates my Facebook status. My Brightkite checkins update the location information on my Twitter account. It also sends a tweet… which updates my Facebook status. I pay less attention to the ongoing status in my LinkedIn account, but it gets updated automatically as well, I just don’t remember how or by what anymore.

I’ve tried a bunch of Twitter clients. Tweetie is “good enough”. It’s the one I use most often. If I need something hardcore I use Tweetdeck or TweetGrid, which has the benefit of being web-based.

TwitterLocal lets you put in your location and a radius, and then shows you tweets from people who are discernibly near you. I think Brightkite does a better overall job with this, since its whole reason for being is to be location-aware, but it seems like I get fewer updates than with TwitterLocal.

Communication

• Colloquy
• Tweetie
• Mail
• Skype
• Google Talk

Right. Twitter is also a communication tool. I have, in fact, checked in with people via Twitter. It’s not how I typically use it, but I think it counts

I have to use both Skype and Google Talk because I’m on the road a lot (I’m a consultant) and there are enough hotels who do stupid things with their network that I’m forced to use whichever one works on that particular network. Though I mostly use GMail for mail, it’s gone down a few times on me, so it’s good to have Mail around. I’ve recently found GMail notifier to be almost useless as well, so when I use Mail, I find that getting alerted to incoming messages frees my brain. I use Mail.appetizer to show me previews of incoming mail so I don’t have to switch gears from what I’m doing to see the latest spam. Note, however, that it’s not quite ready for Snow Leopard.

I haven’t tried Mail in Snow Leopard yet. If they ever fix the search functionality (I find it useless) I’ll stop using the GMail interface. I’ve tried thunderbird, but its search is even worse (or was, the last time I tried it).

Fun Stuff

• GarageBand
• Flickr Uploadr
• iPhoto
• StellaOSX
• Sim City 4

I play guitar and piano, and have also played drums, saxophone, and lots of other noise-making apparatuses. I like that GarageBand will let me put down bass and drum tracks without having to own a bass or drum set.

I also enjoy photography, though I don’t often get out on long quiet hikes in nature or gastronomical adventures that would make for the kinds of stunning things I see on Flickr all the time. However, I do have a family, and we do travel, so while not even 10% of my pics on Flickr are stock quality photos, at least 90% of them are interesting to me personally

iPhoto I see as a necessary evil these days. I used to love it, but now that it tries to help me out by autocategorizing on things that, as it turns out, are pretty arbitrary in the context of my life, I don’t like it as much. It’s good for quick touch-ups though. I’ve saved a number of pics with it.

StellaOSX is an Atari 2600 emulator for the Mac that comes with like, I dunno, thousands of ROMs? If you miss your old Atari games, and you have a Mac, it’s all you’ll ever need.

Sim City 4 is a city-building game. If you haven’t heard of Sim City before, it’s not like the Sims. At all. I don’t get that game, in fact. Sim City is a game where you have to try to build a city, build its wealth and prestige, and try to keep the residents happy as well.

Productivity

• Things for Mac (and iPhone)
• Google Calendar
• iCal

Things for Mac is the first application I’ve personally seen that seamlessly syncs with Things for my iPhone. It works great. It’s not a full-blown project management solution, but it’s more than a todo list. It’s not about work-related stuff, either. Things is really about keeping my personal things in order. I have to call the township for an inspection on my recent AC replacement, schedule for a followup doctor visit for my dog, hire an insulation contractor by the fall, send out my quarterly taxes, make a dentist appointment… that kind of stuff. It’s also a great place to put ideas for blog posts and stuff, and since it’s right there on my iPhone, I don’t forget as many ideas anymore. I can’t say enough good things about Things, so I’ll just say go try it.

Google Calendar and iCal are kept in sync, so I don’t have to use the horrifically slow Google Calendar on my iPhone. I can sync to iCal on the desktop, sync that to my iPhone, and use iCal on the phone as well. Why the whole calendar synchronization thing has to *still* be hard after like 4 years of trying is beyond me.

Office

• Keynote (and Remote for iPhone)
• Pages
• Google Docs
• OpenOffice
• OmniGraffle Pro
• Preview
• Dictionary
• ScreenFlow

Keynote makes doing things that are hard in PowerPoint and impossible in OpenOffice or Google Docs easy as all getout. As a trainer, I spend a lot of time putting content together and trying to find new ways to make it more engaging, less boring, etc. (not that I’ve been accused of being boring, mind you)

I deliver all of my training from a MacBook Pro using either the remote that came with my laptop or the Remote iPhone application. Usually I can’t use Remote for iPhone because of restrictions regarding the wireless network, but I sometimes use it at home to rehearse new content.

I do use Google Docs for lots of other stuff. It’s not what I’d call full-featured, but when you discover that it’s integrated with Google Talk, it actually makes real-time collaboration pretty nice. Sadly, Microsoft Word is still the only word processing application I’ve seen with offline collaboration features that I’d call “pretty good”. Nothing I’ve seen recently can do what Word did 5 years ago in terms of collaboration. Again — sad.

Preview is a PDF viewer, but it also will do screen grabs. I know there’s a keyboard shortcut to do screen captures. I think it’s shift-command-4. I’m just as happy opening Preview, which is right there on the Dock anyway. It’s better than the old utility Apple provided for this, which would only save in TIFF format.

I feel like people look at me strange when I say that I use a dictionary every single day I’m on the computer (so… every day). I used it for this post, as a matter of fact (“apparatuses” still doesn’t sound right to me). I wish there was an app that could tell you how often you’ve used an app in the last day, week, month, etc. I’ll bet the Dictionary app outnumbers Mail (I usually only use Mail when GMail is down).

System Maintenance

• Time Capsule/Time Machine
• AppCleaner
• Disk Inventory X
• Apple Remote Desktop

I bought a Time Capsule. It’s an Apple product. It’s an enclosed 1TB hard drive inside of a wireless access point. It also has a USB port where you can connect a hub and then connect up other external USB hard drives, and a USB printer that can then be shared with the whole network without running a long-in-the-tooth Mac G4 with the mirrored doors and the fan that sounds like the landing of the mothership…. uh…. I mean… It’s really easy to use! I use it to back up all of the Macs in the house. The iPhone backs up to my Mac, so that’s covered too.

AppCleaner isn’t horribly useful, but I do use it, and it helps slightly. Maybe. It’s supposed to help you get rid of apps you no longer use, but it still leaves behind seemingly everything that would normally be left behind if you just opened Terminal and typed “sudo rm -rf ./AppName”. I give it the benefit of the doubt. Maybe it catches some stuff sometimes, and then I know all of the usual suspects that hang on to old app cruft, so I can clean some of it out manually without too much fuss.

Disk Inventory X is pretty cool. It presents a tree map view of the contents of your hard drive which makes it dead easy to spot where the disk hogs are. And here I was writing scripts for this It’s a great spotting tool, but because it’s constantly scraping the disk, it’s quite slow. You also can’t select multiple things in the interface and move them all to the trash at one time, which would be nice. Still, it definitely helped me find stuff I didn’t know was there, and that was taking up lots and lots of space.

Apple Remote Desktop isn’t something I use often, but it’s handy to have around. It lets you do all kinds of advanced stuff by connecting to the desktop of a remote Mac, but I just do simple things with it. If you didn’t know about it, it’s worth at least being aware of.

System Administration/Geekery

• Terminal
• Vim
• SSH Tunnel Manager
• VMware Fusion
• Cisco VPN Client

This is the “where do I start” section for me. I do lots of geekery, and these tools facilitate a lot of the geekery. I stuck with the basics here. I use Terminal because tons of what I do is on the command line. There are things I do on the command line for which GUI applications exist, but to be honest, some of those cost money, and none of them are as efficient or reliable as the command line. I know that makes me sound like an old graybeard, but it’s mostly true. A GUI that really makes something you already know how to do on the command line easier is rare.

Vim, of course, runs inside of Terminal. If I’m writing a bunch of code across lots of files or something, I’ll try to use Komodo Edit (and I might upgrade to Komodo IDE), but if I’m on a remote machine, or I just need to do a quick edit here or there, one file at a time, I’ll just use Vim. Vim can do window splitting and code folding and stuff like that, so Komodo isn’t a requirement for me, it’s just slightly more convenient, and it has Vi key bindings

SSH Tunnel Manager is a GUI for managing SSH tunnels. Go figure. I’ve been using it for years now, but to be honest, if I don’t use it for a while, the interface becomes unintuitive to me and I go back to the command line or my SSH config file to set up tunnels.

VMware Fusion is great. I can test the latest Linux distros without devoting a whole machine to them, or I can run Windows and test web stuff in IE. There seems to be no end to the stuff I find myself using VMware Fusion for. Surprising.

I’m told there’s a VPN client built into Snow Leopard, but I haven’t tested it out yet. Some have reported issues, so hopefully they don’t bite me.

Programming/Development

• Komodo Edit
• Vim
• Python
• Django
• Mercurial
• Subversion
• Fabric
• Terminal
• VMware Fusion
• Firebug

Komodo Edit is my favorite editor for writing code, period. If it didn’t have Vi keybindings, I’d likely just use Vim. And I do, sometimes. My first-choice language these days is Python, but I still write plenty of PHP, shell, SQL, Perl, etc. The Mac comes with XCode as an optional install, and I should really give it another shot, but in the past I’ve felt that it was kind of overwhelming, not to mention kinda clunky and slow.

Django is a Python web framework that comes with a development stand-in web server so you can do all of your development on the laptop, test it all locally, then push out to some environment that more closely matches production.

Speaking of pushing out changes, I mostly use Mercurial for my own projects nowadays, and I rather like it, but lots of things still use Subversion, which is wildly popular. My open source project actually uses Subversion with Google Code, but Google recently announced Mercurial support for hosted projects, so I’ll need to look at changing that over.

Fabric is a deployment tool. It’s written in Python and uses the paramiko library, which I found interesting, because I’d written a couple of automation scripts using paramiko that would have been easier to do with Fabric. I’ve only done simple things with Fabric so far, but it’s worth a look if you do a lot of rsync-ish stuff, followed by some “ssh in a for loop” stuff, supported by some cron jobs…. Fabric can really ease your life.

VMware Fusion is used in a programming context in two ways: to test web stuff on IE (I have an XP VM), and to work with libraries that are more convenient to work with under Linux than on the Mac. Sometimes Linux distros have things built-in that I’d have to build from source (along with all the dependencies) on the Mac.

Firebug is just basically a necessity if you do any kind of web development. It lets you inspect the design elements on the page visually, as well as in code, which makes debugging your CSS so easy it’s almost a non-event.

So… what tools are you using?

Comments (9)

By m0j0, August 17, 2009 12:25 pm

So, I’ve taken a break from active development on my project to take a step back and really get a good development workflow together. I’ve been fighting with various components of my development workflow, and in the end decided to compromise: I won’t have something that looks exactly like production, but I’ll have something that works and is easy to use. I’ll make up for it by having a staging environment on the same host as the production deployment, which will catch any differences between dev and production that result from my non-identical setup on my laptop.

In getting things going, one thing I ran into immediately was that things on my dev box are different from production: database credentials, paths to media, etc., and since I’m not using Apache and a reverse proxy some of the paths in settings.py will also be different. So what to do?

Turns out there’s an entire page on the Django wiki detailing the ways in which people keep their dev and production settings from trampling each other. There are also various blog posts where people have come up with interesting ways to make things work properly, usually by taking advantage of the fact that your app’s settings are just python code. As such, you can perform any valid Python wizardry you want to make the right things happen.

I don’t think there’s anything necessarily wrong with doing all of that stuff, but what I’m wondering is this: why not just tell your version control system of choice to *ignore* your settings.py file? This way, you can have a settings.py file on your dev box that works perfectly for your dev environment, and a separate one in production that works perfectly for that environment. Never the twain shall meet.

If you’re using one of the methods described on the Django site or any of the blog posts, what are you getting out of it that couldn’t be accomplished by ignoring settings.py? It just seems like it’s simpler and cleaner to do it that way, but I assume there’s some benefit to jumping through these hoops that I’m missing.

Input is hereby solicited. Please tweet/link/post this wherever, ‘cos I’d like some opinions on the matter.

Comments (1)

By m0j0, May 19, 2009 12:05 pm

First, I want to point out the only thing I hate about this application: the name. It’s bad for me, it’s bad for the developers… it’s just bad. If I ever had a problem, how am I supposed to do a search for that? If the developers want to find blog posts like this one, how are they supposed to do that?

Aside from the name, Things for Mac really, truly shines. I’ve tried lots and lots and lots of task managers, project managers, life managers, blah, blah, blah. I’ve probably tried 15 of them, and really tried hard to like them, but I just couldn’t. Some are too rigid and structured, while others are completely lacking any structure whatsoever. Some have a learning curve that forces you to make time to sit down and learn them. Others take pains to pay special attention to every aspect of your life (a model doomed to failure). I don’t want to mention all of the ones I’ve tried, because they’re not bad products and don’t deserve bad press. Mostly, they all worked well — they just were not right for me.

Things, on the other hand, I like. It’s just the right balance of features, structure, and ease-of-use. After watching the screencast introduction, I was off and running, and haven’t looked back. I wrote a short poem (a la Dr. Seuss) about my experience:

I do not like them on my Dock,
I do not like them in Firefox.
I do not like them in my car,
I do not like them near or far!
I do not like most “task”ish apps,
but I find I like this “Things” for Mac!

Ok, it’s totally corny. On to what I like about it.

On the Desktop

I actually didn’t expect to like the idea of a desktop application. Of the 15 other tools I’ve tried, only one or two others had a desktop app, and those two apps were opposite ends of the spectrum: one required a PhD to use, and the other was more like a free-form note pad. Things is neither. If you’re in a rush and just want to make sure you don’t forget a task, you can click on ‘Inbox’, type in as many or as few task details as you want, and be done with it. Later, you can drag the task into a project, or add scheduling information, make it repeat, etc.

Even better, it has a keyboard shortcut that’s settable, so if the app is running somewhere, you can just use that to do a “Quick Entry” without leaving the app you’re currently using. Very handy.

Though Things is not project management software, you can create “Areas of Responsibility”, and then have projects inside of those areas. This is perfect for me. I have areas of responsibility like “Home”, “Work”, and “Blog”, and inside the “Home” area I have all of my home improvement projects. My “Work” area contains a list of projects, which themselves contain high-level tasks (I leave the detailed stuff to actual project management software), and also projects which are related to business development, which I don’t use project management software for. The “Blog” area is a perfect place to jot down ideas for blog posts that I can go back to later and check off when they’re completed.

You don’t have to worry too much about being reminded about upcoming due dates, assuming you set due dates (which you don’t have to do). When due dates are approaching, Things will move the tasks into the “Today” queue, so if you have some down time during the day and aren’t sure what to tackle next, click “Today”, and start on something! I find that there’s almost always something in that queue that I have time to do before I need to (or am able to) get back to what I was doing. That, alone, is worth the $50 I paid for Things.

On the Go!

I’m not sure I would be so into Things if it weren’t for the accompanying iPhone app, which pretty much rocks. I guess if that didn’t exist, they would’ve considered having a companion web interface or something, but really, given the choice, I’d rather have it be mobile than solely on the web (though I’d like to see Things work on non-Apple mobile devices too).

One of the really awesome things about the iPhone app is that it syncs over the air to the desktop app. Of course, both have to be running, and they need to be on the same network, but my desktop app is *always* open, and I have yet to be inconvenienced by anything related to the sync functionality. Some folks have requested mac-to-mac sync capability, which isn’t currently there, but I don’t need it, so it’s not an issue for me.

While the iPhone app is missing some features of the desktop application, 90% of what’s in the desktop app is doable from the iPhone. The only notable exception for me personally was an apparent inability to create new areas of responsibility. Indeed, this is the one place where the interface is a little inconsistent. While clicking “Inbox” or “Today” will bring you to a list of tasks and projects, they chose to list projects and areas of responsibility in the sidebar for easy perusal and drill-down capabilities. It works well in the desktop app, but it’d be nice to make “areas” a queue on the iPhone, just like “Projects” is.

A Compeling Duo

In the end, Things has two things going for it: First, it has the nicest, easiest to use interface. Second, it has the ability to sync over the air with my iPhone, which also has a nice interface. In the old days, you might leave your bulky planner at your desk and carry around a note pad, and this mimicks that kind of experience, but takes away the need to sync them up later by hand.

It’s simple, but not too simple. Structured, but not too structured. I’ve been using it for a little over 2 weeks now, and have no plans to change. I know this sounds like an ad, so I should also say that I’m not being paid for this and don’t work for the company that makes it. I do recommend you check it out, though, if you’re in the market for something to help manage your time.

Comments (0)

By m0j0, March 25, 2009 8:52 am

I know, I know. I haven’t been posting nearly enough. But I did come across two URLs that are too handy not to pass on:

1. Command-line-fu: this is a repository of handy one-liners submitted by pretty much anyone. You can log in with OpenID or register on the site itself. I expect this, or something like it, will become a great resource. You can browse the sweet one-liner goodness, or “grep the archive”. Nice.
2. Down for Everyone or Just me? is a site that’s handy to know about if you’re, say, holed up in a hotel room, forgot to set up port forwarding on your FIOS router, and so don’t have a remote shell to test from, and you can’t reach a site. Pop the url into this site, and it’ll test access for you. Of course, it’s limited — it’ll change url’s with “:22″ to “%3A22″, so you’re not going to get it to be a generic service tester, but still… handy!

Enjoy!

Comments (1)

By m0j0, February 20, 2009 10:53 pm

Marc Andreessen was on Charlie Rose last night, and I missed it. A buddy told me about it, and I wanted to watch, but things just got in the way. So here it is.

So, this is the very first time I’ve ever embedded video into a blog post. I couldn’t help myself.

Why?

I’ve never even heard Marc Andreessen talk until tonight, to be honest. I’ve been a huge fan of his actual technical work, and I’ve read some of his writings, and you almost can’t help but follow his career if you work with internet-related things directly, but I’ve somehow missed him at all of the conferences, never seen an interview… until now. And you know, it turns out that in this interview, he validates several posts that have been lying around on this blog for some time.

He talks about the evolution of web commerce and cloud computing and how it lowers the bar for startups.

He talks about news, newspapers and how they absolutely must kill the print edition, now.

He talks about social media, where it came from, where it’s going, “viral”, etc.

This is not to compare myself to Andreessen in any way — that’s ludicrous. But it’s nice to get some validation from on high for some of the thoughts and ideas I’ve had. Now if only I could write a tool that will lay the foundation for the next generation of human interaction, I’ll be all set

It should go without saying that I learned some things, but the biggest thing I learned came from just a tiny little quip buried in the middle of the video somewhere. He says, while talking about the iPhone, that it was “beamed in from 5 years in the future”. I think problems should be thought about that way in general. I’ve adopted a new way of thinking about products and services from pondering on this for all of 5 minutes. Find a service that solves a problem now, or find a problem that exists now — either one. Now think about how that problem will be solved in 5 years. Now set a deadline for solving that problem in 1 year.

Sounds impossible? Not a chance. Impossible is just another excuse to get creative, change your perspective, rethink the problem, and produce a solution. Listening to really smart people talk can be inadvertently inspiring. Thanks Marc!

Comments (0)

By m0j0, February 19, 2009 2:03 pm

Sometimes I run through the search terms people use within my site, or to get to my site, and I see some interesting stuff. Over the years, I’ve written perhaps hundreds of technical blog posts and articles at various sites and in magazines (and a book), but I have never once touched on the “history” command. In spite of that, someone searched for “what does the history command do”, and somehow landed on this site. Here’s a quick overview I give people who take my Linux From the Ground Up class (I’m currently booking on-site training for the July-September period, by the way).

The history command is built into the bash shell (and zsh, and other shells as well). This means you won’t find it sitting around in /usr/bin. You also will get funny results if you run “man history”, which will bring you to a man page about all of your bash builtins instead of one specific man page for “history”.

Typical (Simple) Usage

You have a shell session open. You’ve been doing some work, and you’ve run several commands. One of the commands was a long nasty compilation command which contained environment variable settings, lots of flags, and references to specific files. It was long, and you don’t want to type it again, but you have to because the compilation failed. You’ve fixed the issue, and want to run the same compilation command.

Instead of typing all of that stuff in all over again, you have a couple of choices (more really, but we’re starting slowly):

1. If you’re on a machine that uses readline capabilities, you can use the up arrow to move back through your previous commands. This is inefficient.
2. You can type “history” at the command line

Typing “history” at a shell prompt by itself will print a numbered list of commands you have run. Let’s say your compile command is number 50 in the list. Now typing “!50″ at the command prompt will run that command again without you having to type in the whole disgusting thing.

More Useful Usage of “history”

If you know that you’ve only run, say, “netstat” with one set of arguments, and you keep running it over and over again, you can use a shortcut by just typing “!netstat”. That will run that last command in the history that starts with “netstat”. This is prone to error if you run a command often, but with different arguments.

If you need to run a long ugly command again immediately, but you need to change just a single argument, you can also use carets to do string substitution. So, if you just ran “netstat -plant | grep :80″ and you now want to check for “:22″ instead of “:80″, instead of typing in the whole command again, you can just type “^80^22″. Note that if you ran, in this order:

1. netstat -plant | grep 80
2. netstat -plant | grep 111
3. ^80^22

You’ll get an error. It only works on the last command you ran.

You can also search through your command history in bash using Ctrl-R. This will alter your prompt, and when you start typing the command you’re looking for, it will do an incremental search through your command history and put what it finds on the line. When the right one pops up, hit enter, and the command will run. Note that if you need to make a quick edit to the command before it runs, moving the cursor will paste the command on your prompt, which will return to normal, and you can edit it before hitting enter.

History Command Quirks

There’s one quirk of history that bites lots and lots of people, myself included. If you’re aware of it, you can avoid it biting you. You see, bash keeps its command history for the current shell in memory, and then writes out that memory to your ~/.bash_history file… when the shell exits. There are times when this can be problematic, for example if you open another shell and want to run a command you just ran in the first shell. It won’t be in your history, because the first shell is still open, so its history hasn’t been written out to disk yet.

There are two ways to get the history that’s in memory onto disk. The first is to exit the first shell. The second is to run “history -w” in the first shell. Either will write the history to the ~/.bash_history file, but be forewarned that doing this overwrites anything that was previously in the ~/.bash_history file! Maddening, isn’t it?

If you configure bash ahead of time by adding “shopt -s histappend”, you can tell bash to append, rather than overwrite the history file.

Another quirk of how history is setup to work by default is that it saves every single command you type. If you open a terminal and type nothing but:

ls
cd
cd -
ls
ls -l

Those all take up space in your history. The environment variable HISTSIZE can be set to something like 10,000 to account for this if you want. It’s set to 1000 by default on Red Hat systems. The other thing you can do is tell the history mechanism to conserve space in history by ignoring certain commands, patterns, duplicates, etc, using HISTCONTROL and HISTIGNORE

HISTCONTROL can be set to “ignorespace”, which will not put commands starting with a space in history. I’ve never been able to train myself to type a space before ‘ls’, so I’ve never used that setting. Slightly more useful is the “ignoredups” setting, which will stop the command you just ran from getting into history if it’s a duplicate of the command run just before it. Still not what I’m looking for. Better is “erasedups”, which will delete all previous instances of that command in history before writing the current command to history.HISTCONTROL can take multiple values, separated by colons.

If you just want to make sure “ls” commands never get written to history, you can just use HISTIGNORE for that. It can be set to a colon-delimited list of patterns or commands that, if matched, disqualify the command from being entered into history. So, you can do something like this:

HISTIGNORE=ls:cd:df:du

Note that these are essentially command matches, and must match exactly, starting from the beginning of the line. So, ” du” (with a space in front) will be put into the history list, and so will “du -h” (no preceding space) will also make it. You can use, for example, HISTIGNORE=du* if you want to catch that command and anything that follows it.

Really Hardcore History

There’s a presentation online that talks about way more nitty-gritty details of history than I have time to cover. It’s great work, though, so I recommend you check it out here.

Comments (10)

By m0j0, January 19, 2009 4:34 pm

I ran into a student from a class I taught last summer. He’s a really sharp guy, and when I first met him, I was impressed with just how much Perl he could stuff into his brain’s cache. He would write what he called ‘one-liners’ in Perl that, in reality, took up 5-10 lines in his terminal. Still, he’d type furiously, without skipping a beat. But he told me when we met that he no longer does this, because I covered awk in my class.

His one-liners were mostly for data munging. The data he needed to munge was mostly data that was pretty predictable. It had a fixed number of fields, a consistent delimiter — in short, it was perfect for munging in awk without using any kind of esoteric awk-ness.

One thing I cover in the learning module I’ve developed on Awk is aggregation of data using pretty simple awk one-liners. For example, here’s a pretty typical /etc/passwd file (we need some data to munge):

root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/etc/news:
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
gopher:x:13:30:gopher:/var/gopher:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
apache:x:48:48:Apache:/var/www:/sbin/nologin
avahi:x:70:70:Avahi daemon:/:/sbin/nologin
mailnull:x:47:47::/var/spool/mqueue:/sbin/nologin
smmsp:x:51:51::/var/spool/mqueue:/sbin/nologin
distcache:x:94:94:Distcache:/:/sbin/nologin
nscd:x:28:28:NSCD Daemon:/:/sbin/nologin
vcsa:x:69:69:virtual console memory owner:/dev:/sbin/nologin
rpc:x:32:32:Portmapper RPC user:/:/sbin/nologin
rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin
nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
mysql:x:27:27:MySQL Server:/var/lib/mysql:/bin/bash
dovecot:x:97:97:dovecot:/usr/libexec/dovecot:/sbin/nologin
squid:x:23:23::/var/spool/squid:/sbin/nologin
ldap:x:55:55:LDAP User:/var/lib/ldap:/bin/false
pcap:x:77:77::/var/arpwatch:/sbin/nologin
ntp:x:38:38::/etc/ntp:/sbin/nologin
rpm:x:37:37::/var/lib/rpm:/sbin/nologin
haldaemon:x:68:68:HAL daemon:/:/sbin/nologin
named:x:25:25:Named:/var/named:/sbin/nologin
xfs:x:43:43:X Font Server:/etc/X11/fs:/sbin/nologin
jonesy:x:500:500::/home/jonesy:/bin/bash

It’s not exotic, cool data that we’re going to infer a lot of interesting things from, but it’ll do for pedagogical purposes.

Now, let’s write a super-simple awk aggregation routine that’ll count the number of users whose UID is > 100. It’ll look something like this:

awk -F: '$3 > 100 {x+=1} END {print x}' /etc/passwd

The important thing to remember is that awk will initialize your variables to 0 for you, which cuts down on some clutter.

Let’s abuse awk a bit further. What if we want to know how many users use each shell in /etc/passwd, whatever those shells may be? Here’s a one-liner that’ll take care of this for you:

awk -F: '{x[$7]+=1} END {for(z in x) {print z, x[z]} }' /etc/passwd

While awk doesn’t technically support multi-dimensional arrays, it also doesn’t have to be numerically indexed. So here, we tell awk to increment x[$7]. $7 is the field that holds the shell for each user, so if $7 on the current line is /bin/bash, then we’ve told awk to increment the value in the array indexed at x[/bin/bash]. So, if there’s only one line containing /bin/bash up to the current record, then x[/bin/bash]=1

There’s a lot of great things you can move onto from here. You can do things that others use Excel for right in awk. If you have checkbook information in a flat file, you can add up purchases only in a given category, or, using the technique above, in every category. If you store your stock purchase price on a line with the current price, you can use simple math to get the spread on each line and tell you whether your portfolio is up or down. Let’s have a look at something like that. Here’s some completely made up, hypothetical data representing a fictitious user’s stock portfolio:

ABC,100,12.14,19.12
FOO,100,24.01,17.45
BAR,50,88.90,94.33
BAZ,50,75.65,66.20
NI,23,33.12,43.32

Save that in a file called “stocks.txt”. The columns are stock symbol, number of shares, purchase price, and current price, in that order. This awk one-liner indexes the ‘x’ array using the stock symbol, and the value at that index is set to the amount gained or lost:

awk -F, '{x[$1]=($2*$4)-($2*$3)} END {for(z in x) {print z, x[z]}}' stocks.txt

Hm. Actually, that’s kind of inefficient. I realized while previewing this post that I can shorten it up a bit like this:

awk -F, '{x[$1]=($2*($4 - $3))} END {for(z in x) {print z, x[z]}}' stocks.txt

Glad I caught that before the nitpickers flamed me to a crisp. Always preview your posts! ;-P

Ah, but of course, that’s not enough. This spits out the gain and loss for each stock, but what about the net gain or loss across all of them? You only need to tweak a little bit:

awk -F, '{x[$1]=($2*($4-$3)); y+=x[$1]} END {for(z in x) {print z, x[z]}; print "Net: "y}' stocks.txt

We just added the assignment of the ‘y’ variable before the “END”, and then added a print statement after the “END”.

I hope this helps some folks out there. Also, if your team needs to know stuff like this, I do on-site training!

Comments (2)

By m0j0, December 23, 2008 9:48 am

[UPDATE: 2009/08/08]: I’ve now gotten stuck on two separate projects, trying to find a bridge between Python code that generates data, and javascript code that is apparently required in order to present it. I haven’t found such a bridge. For one project, I was able to do what I needed with ReportLab (there was no webification requirement). For this project, it’s pretty useless if it’s not on the web, imho. So until I either break down and decide to deal with the monstrosity that *is* javascript, or someone else takes this code and runs with it, here it sits. There is only a slight chance I’ll decide Javascript is something I want to deal with. I’ve written some code in javascript. I’ve never enjoyed it, and I’m not very good at it.

While putting together the US Technical Conferences calendar over the past week or two, I noticed that the location of probably 80% of them (I’m guessing – it’s probably higher) is somewhere in California. I’ve always noticed that there is this trend to hold technical conferences in California, because there’s an enormous concentration of technology workers there. But c’mon! How about an OSCON East or something?

Anyway, I have more going on this holiday season than usual, but I always try to spend some of the downtime doing something interesting, and the Google Maps API is probably one of the few Google APIs I’ve never used. What better way to expose the inequity in conference locales than to plot all of my Tech Conference calendar events on a Google Map? Oh, and it’d be useful for people to be able to visually see where conferences are, and maybe color-code the markers according to Q1 2009, Q2 2009, etc.

I’ll be doing this in Python, by the way, though I’m pretty sure I’ve decided to go ahead and use javascript for the Maps portion. There *is* a Python utility that attempts to relieve you of the hellhound that is javascript, but it’s not documented at all that I’ve seen, and wrappers that attempt to generate javascript tend to be flaky (including one I wrote myself – I’m not picking here). I’m also using geopy for the geocoding, because it provides more flexibility than hard-coding calls against the Google Maps API.

By the way, this isn’t like some new fantastic idea I had. Someone else came up with a solution that works quite some time ago, but it involved several steps including Yahoo pipes and stuff. I really just wanted a script that, called with a parameter or two, would dump the appropriate .html file into a directory, or better, would just be called directly from the browser and take input (baby steps). Later, there are aspirations of plugging it into Django, an area where Google Maps has already seen some integration work, and another victim of my recent exploration/experimentation.

So, I already have some prototype code that might be useful for others who are maybe just starting out with the Calendar API. This bit of code will create an authenticated CalendarService object and dump the title and location of events on a calendar of your choosing. The assumption here is that the location is in some form that is parseable by whatever geocoding service you decide to use. For me (for now), I’m just using city and state locations – not addresses. Here goes:

#!/usr/bin/env python

try:
  from xml.etree import ElementTree # for Python 2.5 users
except ImportError:
  from elementtree import ElementTree
import gdata.calendar.service
import gdata.service
import atom.service
import gdata.calendar
import getpass
import atom
import getopt
import sys
import string
import time
import geopy

calendar_service = gdata.calendar.service.CalendarService()
calendar_service.email = 'you@gmail.com'
calendar_service.password = getpass.getpass()
calendar_service.ProgrammaticLogin()
cal_id = 'id_from_calendar_settings_page_or_email_for_your_default_cal'
feed = calendar_service.GetCalendarEventFeed('/calendar/feeds/'+cal_id+'/private/full')
geo = geopy.geocoders.Google('your_google_api_key')

print feed.title.text  # the name of the calendar we're sucking feeds out of.
for event in feed.entry:
  print event.title.text, event.where[0].value_string # The event name, and location.
  (lat,lon) = geo.geocode(event.where[0].value_string)
  print lat,lon

It’s a start. Next I need to figure out the steps between this stage and making the actual cool stuff happen. Hopefully I’ll be back with an update right around the new year.

Comments (21)

By m0j0, December 17, 2008 4:42 pm

Point all of your chain-mail-forwarding family and friends at this post. It’s a collection of things people have said to me, or that I’ve overheard, that reveal little tidbits about what people are thinking when they use IE.

I have to use IE – it’s my internet!

IE is not your internet. IE is what’s known as a web browser. There are lots of different web browsers. IE just happens to be the one that comes with Windows. It doesn’t make it a good browser or anything. It’s just there in the event that you have no other browser. If the only browser on your system is IE, the first thing you should do is use it to download Firefox by clicking here.

If IE is so horrible, how come everyone uses it?

They don’t, actually. There was a time not too long ago where over 90% of internet users used IE. However, with the constant flood of security issues (IE usage really should be considered dangerous at this point), IE’s horrible support of web standards (which makes it hard for web developers to create cool sites for you to use), and its inability to keep up with really cool features in modern browsers, its share of the internet usage market has been declining steadily over the last couple of years. In fact, this source puts IE usage at around 45% currently, so not even a majority of people use IE anymore, if statistics are to be believed. Accurate statistics for browser use are difficult to nail down, and are probably more useful to discern a trend, not hard numbers. Still, the usage trend for IE is moving downward, steadily, and not particularly slowly. If you’re still using IE, you’re almost a dinosaur. Just about the entire tech-savvy world has migrated over to Firefox, with small contingents choosing Safari (Mac only) and Chrome (Windows only). Very small camps also use Opera and Konqueror.

This is also not to be trusted, but it’s my opinion based on observation of the IT field over the past 10 years: of the 40% of people still using IE, probably half of them are forced to use it in their offices because they don’t have the proper permissions on their office computers to install anything else. The other half probably just don’t realize they have any choice in the matter. You do. There are other browsers. I’ve named a few in this post. Go get one, or three, of them.

Will all of the sites I use still work?

It has always been exceedingly rare that a web site actually *requires* IE in order to work properly. Your online banking, email, video, pictures, shopping, etc., will all still work. The only time you might need IE around is to use the Microsoft Update website. In all likelihood, you’ll be much happier with your internet experience using something like Firefox than you ever were with IE. Think about it this way: I’m a complete geek. I use the internet for things ordinary users didn’t even know you could do. I bank, shop, communicate, manage projects, calendars and email, registered and run my business completely online. It’s difficult to think of a task that can be done on the internet that I don’t use the internet for, and I haven’t used IE in probably 8 years, and have not had any issues. If you find a web site that absolutely, positively CANNOT be used UNLESS you’re viewing it with IE, please post it in the comments, and I’ll create a “hall of shame” page to list them all, along with alternative sites you can access WITHOUT IE, which probably provide a better service anyway

I’m not technical enough to install another browser.

Who told you that?! That’s silly. You installed Elf Bowling didn’t you? C’mon, I know you did. Or what about that crazy toolbar that’s now fuddling up your IE window? Or those icons blinking down near the clock that you forgot the purpose of. At some point, you have installed something on your computer, and it was, in all likelihood, harder to do than installing Firefox would be. It’s simple. You go here, click on the huge Firefox logo, and it presents you with super-duper easy instructions (with pictures!) and a download. It takes less than 3 minutes to install, and you DO NOT have to know what you’re doing in any way or be geeky in any way to install it. If you can tell whether you’re computer is turned on or not, you’re overqualified to be a professional Firefox installer.

I Like IE. I have no problems with IE.

Whether you realize it or not, you have problems with IE, believe me. I had a cousin who said he had no problems with IE too. Then he came to my house one day, knocked on my door, and when I opened it, he handed me a hard drive from his computer. He said that all of his pictures of his first-born child were on there, and his computer had contracted a virus, and he couldn’t even boot from the hard drive. So it was up to me to recover the only pics he had of his only son being born. True story. Turns out, I tracked down the virus on the hard drive, and it was contracted by IE. Also, it wasn’t the only virus he had. If you think you’re safe because you have antivirus software, you’re sadly mistaken. He had it installed too, but it hadn’t been updated in 6 months, so any viruses released since the last update weren’t recognized by the antivirus software, and were allowed to roam freely onto his hard drive.

There has never, in the history of browsers, been a worse track record with regards to security than IE. Never. I promise – but you’re free to Google around for yourself. Half of the reason antivirus software even exists is purely to protect IE users (though email viruses are a problem independent of what browser you use, admittedly).

The other reason you might say you like IE is because you’ve never used anything else. As an alternative, I strongly suggest giving Firefox a shot.

Why do you care what browser I use?

I’m a technology guy. I’m one of those people that would work with technology even if he wasn’t being paid. Some people care about cooking, or quilting, or stained glass, or candlemaking, or knitting, or sewing, or horticulture, or wine. Heck, my mom cares about every single one of those things! Me, I care about technology, and I care about the internet. I want the internet to be a better place. Browsers play a non-trivial role in making the internet a better place. Also, one reason I care about technology is that it helps people do things they might otherwise be unable to do. Browsers enable users to do great things, and it allows us developers to make great things available to you. But when countless hours are spent trying to make things work with IE, it just slows everything down, and you don’t get cool stuff on the internet nearly as fast as you could.

So, it’s less about me caring what browser you use. In fact, I don’t really care if you use Firefox or not, it just happens to be the best browser out there currently. If you want to try something completely different, I encourage that too. It’s more about me caring about technology, the internet, and your browsing experience.