Several images disappeared from the Our Fresh World green-building web site because Google changed their Picasa API recently — and I must not be subscribed to the proper mailing list or blog to have been warned ahead of time. Where are incompatible Google API tweaks announced?

The site owner and his photographer use Picasa web albums to upload, edit, and maintain their image collection. They simply give special tags to their favorite photos, and my application code then knows that it is supposed to display those photos on the web site. I almost used Flickr for this application, both because I am an avid Flickr user myself and because I consider its web interface more usable. But, perhaps predictably, Picasa had the much stronger search API — whereas you can either ask Flickr for the photos in a particular set, or ask for all of someone's photos that share a particular tag, Picasa lets you can combine the two queries and ask for only the photos that are in a particular set and that also share a specific tag. And since search is what attaches pictures to this web site, Picasa was my choice.

All was going well, with each page getting populated by searches like:

http://picasaweb.google.com/data/feed/api/user/name
?kind=photo&tag=solar-power

Then I received an email from the site owner, complaining that many of the photographs had disappeared! After seeing some complaints in the Picasa forums about recent versions of the user interface treating certain “special characters” in tags as spaces instead, I suddenly wondered whether the hyphen in several of our tags (like the “solar-power” tag in the URL above) was the cause of our trouble. I adjusted my code so that this search became:

http://picasaweb.google.com/…?kind=photo&tag=solar+power

And, voilà, the images returned and were again visible! Does anyone know what forum or blog I should have been following to be informed of this critical change by Google? It is dismaying to have a site break in front of a customer when the very reason that I chose a Google product was because of their powerful API for integrating my application.

Thanks to more than an hour of work today, I have a pretty list of a few dozen commands that make it easy for a WebFaction account holder to install the powerful lxml Python package for parsing HTML and XML under their hosting account. You can read Ian Bicking's wonderful blog post “lxml: an underappreciated web scraping library” for more information on why you want to be using lxml instead of any of its alternatives.

So, why do I say “drat”?

First, because I just tried out my instructions on another of my WebFaction accounts, and there the extra steps weren't even necessary; this other server of theirs already had lxml's dependencies installed! I suppose, had I been a bit more patient, that this support ticket that I glanced over this morning would have inspired me to ask WebFaction to install the libraries lxml needs on the server where I myself was working. But it felt like some sort of offense against symmetry to rely on something that WebFaction doesn't install everywhere, and I was perhaps just in too big of a hurry. Which, of course, cost more time in the end.

The other reason I say “drat” is because, now that I look at Ian's post again after all these months, I see that he has instructions for making the package install its own dratted copies of the system libraries it needs! Too bad that lxml's own installation instructions omit this crucial piece of information.

How typical, and how predictable. It turns out that I just needed to listen to Ian Bicking more carefully. How often we fail to do that, as individuals and as a Python community. Listen to Ian Bicking, everyone. Listen.

Read more

1. Run the following script in your home directory. (You might want to use less to read the output.)
2. Ignore files whose date does not reflect your own activity.
3. List the oldest files in a blog post and discuss!

#!/usr/bin/env python
"""Print last-modified times of files beneath '.', oldest first."""
import os, os.path, time
paths = ( os.path.join(b,f) for (b,ds,fs) in os.walk('.') for f in fs )
for mtime, path in sorted( (os.lstat(p).st_mtime, p) for p in paths ):
    print time.strftime("%Y-%m-%d", time.localtime(mtime)), path

Only include files whose last-modified time is a date on which you really touched the file. The file's time should neither result from an error (a few files beneath my own home directory have an incorrect date of 1970-01-01), nor from unpacking someone else's archive that has old files inside of it. For example, I myself have excluded the following pair of nearly 17-year-old files because their dates reflect their age inside of the Python 3.0 source archive, instead of the actual moment last month when they became part of my home directory:

1992-03-02 ./src/Python-3.0/Demo/scripts/wh.py
1992-03-02 ./src/Python-3.0/Tools/scripts/dutree.doc

But there is no requirement that the actual content of each file you list be your own. Whether you wrote the file yourself long ago, or downloaded it from some ancient and forgotten FTP site, you have a story to share!

Within the rules given above, here are the oldest files beneath my own home directory:

Read more

The rise and fall of the popularity of “My Shirt” over the four weeks from 5 October through 1 November.

And by drilling down into the list of “Referrers” beneath the graph, I was even able to discover the source of its brief popularity! As Halloween approached, people were doing hundreds of Google Image searches for “waldo costume,” “where's waldo costume,” and “where's waldo shirt,” which brought them straight to my image. As you can see in the above graph, the swelling interest did not peak until Halloween itself, after which the photo plummeted back to its more typical popularity of one or two dozen views per day.

All sorts of gems are hidden in the statistics, waiting to be discovered. For example, it was very satisfying to learn that Yahoo! Image Search considers my wedding photograph A Grandfather in Attendance to be the most important “grandfather” image on the entire web. Behind every statistic is a story about how people find, and why they wind up visiting, each of my photos. Hopefully the fun of watching my old photographs will not distract me from taking some new ones!

Term paper mill. Need $100 by Friday to keep the lights on? No sweat, if you’re a writer. Plenty of kids need ten pages on Hamlet by Thursday... more»

The article, entitled “Term Paper Artist”, alternates between hilarity and poignancy as its author shares his adventures writing hundreds of term papers for hire. But near the end, his tone suddenly becomes serious as he turns to the question of why so many students are unable to write term papers of their own. He thinks that the reason is important enough to stand alone in his article as a one-sentence paragraph. Here is his preceding paragraph, and then the zinger:

It's not that I never felt a little skeevy writing papers. Mostly it was a game, and a way to subsidize my more interesting writing. Also, I've developed a few ideas of my own over the years. I don't have the academic credentials of composition experts, but I doubt many experts spent most of a decade writing between one and five term papers a day on virtually every subject. I know something they don't know; I know why students don't understand thesis statements, argumentative writing, or proper citations.

It's because students have never read term papers.

That is his diagnosis: students never see what term papers are supposed to look like, and so they have no idea how to produce them.

As he continued on, ridiculing the idea that students can produce something of which they are never once shown a good example, I realized that his argument was exactly the same as the one I made in my recent post Reading Code: A Computer Science Curriculum: that the production of any kind of literature, whether an essay in college or an elaborate routine in a computer program, is an essentially imitative act. Without being shown excellent examples from the genre they are expected to produce, students are left in the dark about what, exactly, they are trying to generate — and, more often than not, will fail. They are never even given the opportunity to demonstrate whether they do, in fact, lack the capacity to create, because they are never shown the goal towards which they are supposed to be striving.

Because users might become confused now that PyEphem is spread across three web sites — the home page is here at rhodesmill.org, releases are posted over at the Python Package Index, and, again, the development project is now hosted at Launchpad — I have completely redesigned the PyEphem home page with the goal of making the three-site distinction clear, coherent, and easy to navigate. The new home page and documentation are generated by the wonderful Sphinx documentation engine, and I am still thrilled about how pretty my code samples look (check out the one on the PyEphem home page!) now that Sphinx is coloring them in with the renowned Pygments system.

I have simultaneously released a new version of PyEphem that includes the new Sphinx-based documentation, along with several important fixes to the software itself. From now on, rather than cluttering my own blog with every minor version of PyEphem that I might release, fans of the software should visit its News and announcements page on Launchpad and subscribe themselves to its Atom/RSS feed. You will still see the project mentioned here whenever a technical or scientific issue becomes interesting enough for me to write about; but the audience of astronomers and hobbyists who just need to know when the next version is released should not have to wade through my blog to do so!

My users have already begun transferring their questions and problems to Lauchpad, and I look forward to offering much greater accountability through a fully public development process.

United States Constitution

One can spend several minutes just staring at the words so basic to our national life, and pondering the significance of their relative sizes! To make my own contribution to the burgeoning world of Wordle documents, I created a program to extract the memorial messages from the Marshall Booth Guest Book on legacy.com, and then submitted the result to Wordle. After several tries, and after experimenting with the color options, I came up with something I find quite satisfactory:

Marshall Booth memorial

I am sure that Wordle documents will look rather formulaic once everyone has used them to generate Christmas cards one or two years in a row. But they are without question of much greater visual interest than any other tag cloud I have ever seen, and are therefore a big step forward simply by making word frequency something worth staring at.

It would be fun to submit novels, or theological treatises, or each of the books of Paul, to Wordle and then see whether students of English literature or Biblical exegesis could identify the original document simply by which words appeared the most often. I think there would be interesting surprises! Could people tell apart the five acts of Hamlet?

I have started using the GraphViz application, which accepts a list of nodes and arrows, and figures out how to attractively arrange them in a diagram. For example, you can very nearly produce this output:

by supplying this rather modest input file to GraphViz (most of whose length comes from my wanting particular colors)::

 digraph Application {
    rankdir=LR;
    node [shape=box,style=filled,fillcolor="#C0D0C0"];
    subgraph clusterClient {
       label="Client"; style=filled; bgcolor="#D0C0A0";
       "Browser";
    };
    subgraph clusterServer {
       label="Server"; style=filled; bgcolor="#D0C0A0";
       "App";
       "Database" [shape=DatabaseShape,peripheries=0];
    };
    "Browser" -> "App" [label="HTTP"];
    "App" -> "Database" [label="SQL"];
 }

I used the words “very nearly” because, in fact, GraphViz only knows how to draw simple shapes like rectangles, and is ignorant of the standard cylinder-shaped database symbol that I have used here by asking for a DatabaseShape. Submitting the above code to GraphViz will, normally, produce three nodes that are all rectangles. To teach it about the database shape, I had to write some PostScript.

Read more