Archive for August, 2009

Like many Unix users, I long ago created a ~/bin/ directory in my home directory and added it to my PATH so that I could supplement the wonderfully rich set of basic Unix commands with some conveniences and shell scripts of my own devising.

The problem, of course, was the chance of collision. Because my shell script names tended to be short and pithy collections of lowercase characters, just like the default system commands, there was no telling when Linux would add a new command that would happen to have the same name as one of mine. This was actually not very likely on, say, a System V Revision 3 workstation in the 1980s, but the trouble became quite a bit more acute when I moved into the world of Debian. Red Hat never really worried me, because they packaged (comparatively) so little software. But Debian today supports a huge number of commands; my modest Ubuntu laptop shows several thousand available:

$ apt-file search -x '^/usr/bin/[^/]*$' | wc -l
21733

The solution was obviously to adjust my command names in such a way that they were still easy to type, but would never be chosen as system command names. For me, “easy to type” means not having to use the shift key, and very few characters turned out to be available, unshifted, on a modern keyboard. The lower-case letters are the very characters used in system commands; brackets, backslashes, the colon, the back-tick, and the single-tick all had a special meaning to the shell; and the slash and dot characters both mean something special in a filename. (The slash divides directory names from filenames, and thus cannot appear in a filename itself, while the dot means “hide this file from normal browsing” if it leads the name, and separates a file from its extension in many other cases.)

There was but one character left: the simple, modest comma.

A quick experiment revealed in a flash that the comma was exactly the character that I had been looking for! Every tool and shell that lay in arm's reach treated the comma as a perfectly normal and unobjectionable character in a filename. By simply prefixing each of my custom commands with a comma, they became completely distinct from system commands and thus free from any chance of a collision.

And, best of all, thanks to the magic of tab-completion, it became very easy to browse my entire collection of commands. When trying to remember which of my commands are available in my ~/bin/ directory on a given system, or when simply trying to remember what some of my commands are called, I simply type a comma followed by tab and my list of commands appears:

$ ,«tab»
,complete-scp        ,go-thpgp            ,range
,complete-ssh        ,gr                  ,svn-store-password
,coreoff             ,hss                 ,umount
,coreon              ,mount-thpgp
,find                ,mount-twt

I heartily recommend this techique to anyone with their own ~/bin/ directory who wants their command names kept clean, tidy, and completely orthogonal to any commands that the future might bring to your system. The approach has worked for me for something like a decade, so you should find it immensely robust. And, finally, it's just plain fun.

I am home from a relaxing vacation to the Midwest, and while I was gone last week my excellent publishing team released the July issue Python Magazine to the world. I am particularly pleased that two of the feature articles in this issue come from important segments of the Python world that we have not heard much from in previous issues.

First, IronPython, the .NET version of Python for Windows, is the topic of Jonathan Hartley's article about acceptance testing. He illustrates that, regardless of the language in which you write your .NET application, you can deploy simple strategies to make your application testable through a Python test harness, and thereby bring Python's strong flexibility as a testing language to bear on a product that you might be writing in another .NET language.

Second, Malthe Borch, a veteran of the Zope community, shares how he has written Chameleon, one of the fastest template language implementations available for Python. By processing each template and turning it into Python bytecode before it is ever used to render a single page, Malthe eliminates a huge amount of redundant processing as that same code is used over and over again. His library is a key ingredient in the new high-efficiency web frameworks appearing in the Zope world. His work might even (fingers crossed) become one of the components that the Plone community uses as they streamline their framework and move towards a lighter and more agile “Plone 4”.

Other technical topics covered are: the Hadoop map-reduce framework; the concept of hash functions, and how they apply to Python; and the new string formatting operator which Guido hopes will replace all of the percent-signs that currently litter our code. Wrap it all up with an editorial by Steve Holden about EuroPython 2009 and an editorial by me, and you have a complete issue! If you do not find Python Magazine sitting on your local newsstand, then I hope you will avail yourself of a subscription and, as always, let me know what topics you would like to see covered in future issues. Enjoy!

Well, drat.

Thanks to more than an hour of work today, I have a pretty list of a few dozen commands that make it easy for a WebFaction account holder to install the powerful lxml Python package for parsing HTML and XML under their hosting account. You can read Ian Bicking's wonderful blog post “lxml: an underappreciated web scraping library” for more information on why you want to be using lxml instead of any of its alternatives.

So, why do I say “drat”?

First, because I just tried out my instructions on another of my WebFaction accounts, and there the extra steps weren't even necessary; this other server of theirs already had lxml's dependencies installed! I suppose, had I been a bit more patient, that this support ticket that I glanced over this morning would have inspired me to ask WebFaction to install the libraries lxml needs on the server where I myself was working. But it felt like some sort of offense against symmetry to rely on something that WebFaction doesn't install everywhere, and I was perhaps just in too big of a hurry. Which, of course, cost more time in the end.

The other reason I say “drat” is because, now that I look at Ian's post again after all these months, I see that he has instructions for making the package install its own dratted copies of the system libraries it needs! Too bad that lxml's own installation instructions omit this crucial piece of information.

How typical, and how predictable. It turns out that I just needed to listen to Ian Bicking more carefully. How often we fail to do that, as individuals and as a Python community. Listen to Ian Bicking, everyone. Listen.

I am very happy with my first weeks of hosting sites at WebFaction. They have taken so much responsibility off of my shoulders. They run nginx in front of my web applications on my behalf. I fill out a simple form, placing Pylons under "/" and Wordpress under "/blog", and they write all of the redirect rules to accomplish it. I list the domains and sub-domains on which my site should appear, and they configure the virtual hosting for me.

WebFaction has even finally wrested domain name control from my fingers. Gone are the installations of bind9 that used to be running everywhere, and the smattering of shell scripts with which I wrapped “nsupdate” to publish new host names. Now, I accomplish host name changes in a few moments through the control panel on WebFaction. This gives me more time to do creative things for my customers, and gives me one less daemon that I have to keep upgraded if I want to avoid security problems. (If you want to try them out for yourself, using this WebFaction sign-up link will credit me as the one who recommended them.)

This move to making WebFaction my default choice for hosting new client projects is only the most recent chapter in the very long story of my devolving responsibilities on to other people. The previous steps went something like this: