Archive for the ‘Computing’ Category

Imagine my surprise, while writing my first PEP 302 compliant import hook this afternoon, to carefully watch “sys.modules” for the results of my import but see it suddenly grow by nearly two hundred modules! What on earth had I done wrong? Some quick experiments revealed that my only sin was having the temerity to raise an exception. Let's try raising a simple NameError:

>>> import sys
>>> len(sys.modules)
35
>>> foo
...
NameError: name 'foo' is not defined
>>> len(sys.modules)
225

That's 190 extra modules — merely importing them takes around 60 ms on my laptop! Where are they all coming from? And how could an exception cause so many imports, including such illustrious modules as “email”, “mimetools”, and “xml”?

After reading Ubuntu's “sitecustomize.py” file and all of its consequences, the situation became clear. Their apport crash-reporting subsystem instruments Python with an exception hook that, when invoked, discovers that my system says “enabled=0” in my “/etc/default/apport” file and so it undertakes no special crash logging. But, on the way to loading the routine that performs this simple check, it performs two quite flagrant and unnecessary imports, pulling in both “apt” (that brings with it 83 packages) and “apport” (an additional 107 packages).

The solution? I have removed the “python-apport” package, along with the “ubuntuone-client” suite that depends on it. After the uninstall, exceptions are — wonderfully enough — not causing a single import of a new module! Now, finally, I can continue writing my import hook in peace.

Now that I use Google Chrome almost exclusively, I miss the fact that a running Firefox instance could be controlled from the command line so that Emacs could call for a new tab when I clicked on a URL. It would run a command something like this:

firefox -remote 'openURL(http://example.com/, new-tab)'

But after a few months of manually cutting and pasting URLs into Chrome — which wasn't actually that bad, since the address bar in Chrome is such a convenient and large target — I decided that I needed a real solution. After not finding anything like a -remote option, I discovered that Chrome can at least be run with a debugging port open:

google-chrome --remote-shell-port=9222

The protocol that Chrome speaks is primitive enough that it was quick work to implement a small client in Python. Rather than merely cutting and pasting its code here on my blog, or even be satisfied with making it available on bitbucket, I decided to place the code inside of a new Python package and make it generally available on PyPI as chrome_remote_shell.

Thanks to this simple package, a four-line program (not counting the shebang and comment) is now all that I need to ask Google Chrome to open a new tab:

#!/usr/bin/env python
# Name this file "google-chrome-open-url"
import sys
import chrome_remote_shell
shell = chrome_remote_shell.open()
shell.open_url(sys.argv[-1])

To teach Emacs to start using Google Chrome when I clicked on a link, I only needed to supply it with two new settings:

(setq browse-url-browser-function
      'browse-url-generic)
(setq browse-url-generic-program
      "google-chrome-open-url")  

And now everything works. I hope that these notes prove useful to someone else. Enjoy!

The September issue of Python Magazine appeared on the web late last week and only now, as a new week has started, am I finally sitting down to announce it! The articles range from technically heavy development topics to high-level thoughts about the whole Python community, with plenty in between.

I have to say that our prettiest article this month is “Using Python to Create Beautiful Documents” by Yusdi Santoso, who shares the basic secrets to document generation that he learned when building the EuroPython 2009 brochure using a Python program. Traditional typesetting and computer typography were both interests of mine when I was growing up, so it was fun to read Yusdi's introduction to using ReportLab to generate PDF documents. I look forward to his follow-up article that we will soon be publishing, on the specific techniques that he used in creating the EuroPython booklet.

The other technical articles are an introduction to using SOAP in Python; a guide to displaying objects in a Mac OS X GUI created with PyObjC; an article introducing Python's own built-in Tkinter GUI toolkit; and a small excursion of my own that attempts to explain the popular “trick” (well, it really confused me the first time I saw it!) of defining a decorator using a pair of nested functions. I should confess that my own article contains what is probably this issue's biggest mistake, as pointed out quite promptly by alert reader Emanuel Woiski: in the code sample that is its whole crux of my example, I somehow managed to omit one of the most crucial lines, shown here in bold:

def log(function):
    def log_wrapper(*args):
        print "called %s%s" % (
            function.__name__, tuple(args)
            )
        return function(*args)
    return log_wrapper

I suppose I will now need remedial cut-and-paste training of some sort.

Finally, the issue is rounded out by three articles that move back from Python coding and step out to wider vantage points. Justin Lilly provides an excellent guide to customizing your Vim setup so that it becomes a powerful Python integrated development environment. Steve Holden muses about why diveristy is so difficult and reveals some of the recent goings-on surrounding the diversity statement that the Python Software Foundation has been working on. And my own editorial seeks to point any Python Magazine readers who do not yet have a strong connection with the wider community in the direction of greater engagement with the world of Python.

All in all, I think the issue is a nice mix of fact, experience, and opinion. Please consider subscribing if you would like to hear more about what people are doing with Python, and how. I enjoy reading it; so might you.

Importing a normal, rectangular map of Middle-earth as a Google Earth overlay is too narrow toward the north.

I wanted to measure distances in Tolkien's Middle-earth. While a flat map distorts such measurements, it occured to me that Google Earth can correctly measure both lines and paths across the curved surface of the globe. I soon found excellent documentation for using image overlays with Google Earth, so I downloaded a map of Middle-earth and tried placing it on the globe.

Imagine my disappointment when I saw the result shown in the above image! At first I made the mistake of not holding down the Shift key when resizing the image in Google Earth; the Shift key is absolutely critical for the image to maintain its aspect ratio as you stretch it to the right dimensions. But even after learning this habit, it was painfully clear that the Middle-earth map's projection was different from that expected by Google Earth: the map is far too narrow at the top.

Obviously, it was time to pull out Python, my favorite programming language, and see whether the Python Imaging Library could help me make short work of converting a map from one projection to another.

My Python table at the Atlanta Linux Fest. You can also watch a short video of me demonstrating a depth-first search to some students who dropped by the table. (Thanks, Richard Davies, for the video!)

Running the Python table at the Atlanta Linux Fest this past weekend was a really incredible experience.

First, there was the great feeling that the pillars of the Python community were standing behind me as I stepped forward to represent my favorite programming language. It was Andrew Kuchling who noticed that exhibitor tables at the Fest were free for non-profits like the Python Software Foundation, and Steve Holden who forwarded me a heads-up since I live in Atlanta (the Fest had not yet made it on to my radar). The inimitable Aahz personally shipped me the promotional kit, including a huge “Python” banner and stacks of brochures, that he himself had just used at OSCON 2009. And, completing the loop, it was Andrew who followed up to ask if there were any last things that I needed, and sent me a pile of over one hundred Python stickers that wound up being very popular at the Fest. (I returned home with exactly one, which is sitting next to me on my desk as I type this!)

Here are some lessons that I learned from the experience:

I would like some advice from Zope and Plone folks about how to create forms that are not only easy for other developers to specialize, but which allow several specializations to be composed together. While I have used zope.formlib and z3c.form before for simple tasks, I have not yet been able to tell whether they support these more advanced kinds of operations.

Some background: I am doing some work on GetPaid for Plone with the generous funding of Derek Richardson who, if his dreams had not carried him away from grad school at the end of the Spring semester, would have tackled this same work as part of the 2009 Google Summer of Code.

The current mechanisms that GetPaid provides for customizing its checkout process are very primitive, and my task is to improve them. That is why I have been thinking about customizing forms.

The August issue of Python Magazine is out, and the cover issue is one of the most exciting that I have had the privilege to publish. Following up on his popular talk at PyCon 2009, professor of computer science Dr. Bill Punch has written an article for us with his colleague Dr. Richard Enbody about replacing C++ with Python in Michigan State University's introductory programming class.

The difficulty they faced was that the second and subsequent classes in Michigan State's curriculum have continued using the C++ language; only the initial course could change to Python. While non-majors taking the introductory course get to learn Python and leave, the computer science majors have to then start the second course without any specific knowledge of C++. How much, skeptics wondered, would it hurt the Python students to be a full semester behind in their C++ knowledge compared to students who took the “old” C++ version of the introductory course?

The circumstances at Michigan State wound up being perfect for making the change to Python a full-fledged, statistically valid experiment to determine how much student grades would be hurt because of the switch. Read the article for the detailed results, but I can report — with great satisfaction — that the Python students proved themselves the equals of their C++ peers, and also that very exciting and unexpected results ensued among the students whom the study was not looking at: the non-majors, for which this was their only programming course, and the seasoned grad students, who rolled up their sleeves and learned Python in order to be able to TA the course. Again, see the article for all of the details!

What else do we have this month? Articles about how easy concurrency becomes with Stackless Python; about documenting your software projects using the powerful Sphinx documentation system; about creating simple graphs with the industry-strength matplotlib plotting library; a guide to the details of hashing algorithms, and how they relate to the Python dictionary implementation; and, finally, the usual thoughts and ponderings from both myself and from Steve Holden of the Python Software Foundation. Come join us!

Like many Unix users, I long ago created a ~/bin/ directory in my home directory and added it to my PATH so that I could supplement the wonderfully rich set of basic Unix commands with some conveniences and shell scripts of my own devising.

The problem, of course, was the chance of collision. Because my shell script names tended to be short and pithy collections of lowercase characters, just like the default system commands, there was no telling when Linux would add a new command that would happen to have the same name as one of mine. This was actually not very likely on, say, a System V Revision 3 workstation in the 1980s, but the trouble became quite a bit more acute when I moved into the world of Debian. Red Hat never really worried me, because they packaged (comparatively) so little software. But Debian today supports a huge number of commands; my modest Ubuntu laptop shows several thousand available:

$ apt-file search -x '^/usr/bin/[^/]*$' | wc -l
21733

The solution was obviously to adjust my command names in such a way that they were still easy to type, but would never be chosen as system command names. For me, “easy to type” means not having to use the shift key, and very few characters turned out to be available, unshifted, on a modern keyboard. The lower-case letters are the very characters used in system commands; brackets, backslashes, the colon, the back-tick, and the single-tick all had a special meaning to the shell; and the slash and dot characters both mean something special in a filename. (The slash divides directory names from filenames, and thus cannot appear in a filename itself, while the dot means “hide this file from normal browsing” if it leads the name, and separates a file from its extension in many other cases.)

There was but one character left: the simple, modest comma.

A quick experiment revealed in a flash that the comma was exactly the character that I had been looking for! Every tool and shell that lay in arm's reach treated the comma as a perfectly normal and unobjectionable character in a filename. By simply prefixing each of my custom commands with a comma, they became completely distinct from system commands and thus free from any chance of a collision.

And, best of all, thanks to the magic of tab-completion, it became very easy to browse my entire collection of commands. When trying to remember which of my commands are available in my ~/bin/ directory on a given system, or when simply trying to remember what some of my commands are called, I simply type a comma followed by tab and my list of commands appears:

$ ,«tab»
,complete-scp        ,go-thpgp            ,range
,complete-ssh        ,gr                  ,svn-store-password
,coreoff             ,hss                 ,umount
,coreon              ,mount-thpgp
,find                ,mount-twt

I heartily recommend this techique to anyone with their own ~/bin/ directory who wants their command names kept clean, tidy, and completely orthogonal to any commands that the future might bring to your system. The approach has worked for me for something like a decade, so you should find it immensely robust. And, finally, it's just plain fun.

I am home from a relaxing vacation to the Midwest, and while I was gone last week my excellent publishing team released the July issue Python Magazine to the world. I am particularly pleased that two of the feature articles in this issue come from important segments of the Python world that we have not heard much from in previous issues.

First, IronPython, the .NET version of Python for Windows, is the topic of Jonathan Hartley's article about acceptance testing. He illustrates that, regardless of the language in which you write your .NET application, you can deploy simple strategies to make your application testable through a Python test harness, and thereby bring Python's strong flexibility as a testing language to bear on a product that you might be writing in another .NET language.

Second, Malthe Borch, a veteran of the Zope community, shares how he has written Chameleon, one of the fastest template language implementations available for Python. By processing each template and turning it into Python bytecode before it is ever used to render a single page, Malthe eliminates a huge amount of redundant processing as that same code is used over and over again. His library is a key ingredient in the new high-efficiency web frameworks appearing in the Zope world. His work might even (fingers crossed) become one of the components that the Plone community uses as they streamline their framework and move towards a lighter and more agile “Plone 4”.

Other technical topics covered are: the Hadoop map-reduce framework; the concept of hash functions, and how they apply to Python; and the new string formatting operator which Guido hopes will replace all of the percent-signs that currently litter our code. Wrap it all up with an editorial by Steve Holden about EuroPython 2009 and an editorial by me, and you have a complete issue! If you do not find Python Magazine sitting on your local newsstand, then I hope you will avail yourself of a subscription and, as always, let me know what topics you would like to see covered in future issues. Enjoy!

Well, drat.

Thanks to more than an hour of work today, I have a pretty list of a few dozen commands that make it easy for a WebFaction account holder to install the powerful lxml Python package for parsing HTML and XML under their hosting account. You can read Ian Bicking's wonderful blog post “lxml: an underappreciated web scraping library” for more information on why you want to be using lxml instead of any of its alternatives.

So, why do I say “drat”?

First, because I just tried out my instructions on another of my WebFaction accounts, and there the extra steps weren't even necessary; this other server of theirs already had lxml's dependencies installed! I suppose, had I been a bit more patient, that this support ticket that I glanced over this morning would have inspired me to ask WebFaction to install the libraries lxml needs on the server where I myself was working. But it felt like some sort of offense against symmetry to rely on something that WebFaction doesn't install everywhere, and I was perhaps just in too big of a hurry. Which, of course, cost more time in the end.

The other reason I say “drat” is because, now that I look at Ian's post again after all these months, I see that he has instructions for making the package install its own dratted copies of the system libraries it needs! Too bad that lxml's own installation instructions omit this crucial piece of information.

How typical, and how predictable. It turns out that I just needed to listen to Ian Bicking more carefully. How often we fail to do that, as individuals and as a Python community. Listen to Ian Bicking, everyone. Listen.