Several images disappeared from the Our Fresh World green-building web site because Google changed their Picasa API recently — and I must not be subscribed to the proper mailing list or blog to have been warned ahead of time. Where are incompatible Google API tweaks announced?

The site owner and his photographer use Picasa web albums to upload, edit, and maintain their image collection. They simply give special tags to their favorite photos, and my application code then knows that it is supposed to display those photos on the web site. I almost used Flickr for this application, both because I am an avid Flickr user myself and because I consider its web interface more usable. But, perhaps predictably, Picasa had the much stronger search API — whereas you can either ask Flickr for the photos in a particular set, or ask for all of someone's photos that share a particular tag, Picasa lets you can combine the two queries and ask for only the photos that are in a particular set and that also share a specific tag. And since search is what attaches pictures to this web site, Picasa was my choice.

All was going well, with each page getting populated by searches like:

http://picasaweb.google.com/data/feed/api/user/name
?kind=photo&tag=solar-power

Then I received an email from the site owner, complaining that many of the photographs had disappeared! After seeing some complaints in the Picasa forums about recent versions of the user interface treating certain “special characters” in tags as spaces instead, I suddenly wondered whether the hyphen in several of our tags (like the “solar-power” tag in the URL above) was the cause of our trouble. I adjusted my code so that this search became:

http://picasaweb.google.com/…?kind=photo&tag=solar+power

And, voilà, the images returned and were again visible! Does anyone know what forum or blog I should have been following to be informed of this critical change by Google? It is dismaying to have a site break in front of a customer when the very reason that I chose a Google product was because of their powerful API for integrating my application.

Thanks to more than an hour of work today, I have a pretty list of a few dozen commands that make it easy for a WebFaction account holder to install the powerful lxml Python package for parsing HTML and XML under their hosting account. You can read Ian Bicking's wonderful blog post “lxml: an underappreciated web scraping library” for more information on why you want to be using lxml instead of any of its alternatives.

So, why do I say “drat”?

First, because I just tried out my instructions on another of my WebFaction accounts, and there the extra steps weren't even necessary; this other server of theirs already had lxml's dependencies installed! I suppose, had I been a bit more patient, that this support ticket that I glanced over this morning would have inspired me to ask WebFaction to install the libraries lxml needs on the server where I myself was working. But it felt like some sort of offense against symmetry to rely on something that WebFaction doesn't install everywhere, and I was perhaps just in too big of a hurry. Which, of course, cost more time in the end.

The other reason I say “drat” is because, now that I look at Ian's post again after all these months, I see that he has instructions for making the package install its own dratted copies of the system libraries it needs! Too bad that lxml's own installation instructions omit this crucial piece of information.

How typical, and how predictable. It turns out that I just needed to listen to Ian Bicking more carefully. How often we fail to do that, as individuals and as a Python community. Listen to Ian Bicking, everyone. Listen.

In the meantime, here are some successful and unsuccessful ways of installing lxml under your WebFaction account. Consider the following to be a set of choose-your-own adventure scenarios.

• If the WebFaction host your account lives on already has libxml and libxslt installed, then installation is simple:

  $ easy_install lxml
  Searching for lxml
  Reading http://pypi.python.org/simple/lxml/
  ...
  Finished processing dependencies for lxml
  

• If your WebFaction host lacks libxml, but you listen to Ian Bicking and download the source code yourself, then your install will succeed:

  $ wget http://pypi.python.org/.../lxml-2.2.2.tar.gz
  $ tar xfz lxml-2.2.2.tar.gz
  $ cd lxml-2.2.2
  $ STATIC_DEPS=true python setup.py install
  ...
  Finished processing dependencies for lxml==2.2.2

• If your WebFaction host lacks libxml, but you listen to Ian Bicking, but you rely on easy_install to fetch the package, then your install will fail because it tries building inside of a temporary directory that, on WebFaction, you apparently cannot access:

  $ STATIC_DEPS=true easy_install lxml
  Searching for lxml
  Reading http://pypi.python.org/simple/lxml/
  ...
  Running "./configure --without-python --disable-dependency-tracking
   --disable-shared --prefix=/tmp/easy_install-81ufo5/lxml-2.2.2/buil
  d/tmp/libxml2" in build/tmp/libxml2-2.7.3
  error: Permission denied

• If your WebFaction host lacks libxml, and you fail to listen to Ian Bicking, then you can at least install lxml and its dependencies manually using the following commands, as I worked out this morning. The trick is that instead of trying to tell setup.py where you have installed the libraries by using CC= at the beginning of the command line or something like that, you need to make sure that the special command xslt-config is on your path somewhere:

  $ cd ~
  $ mkdir usr
  $ mkdir usr/src
  $ cd usr/src
  $ wget ftp://xmlsoft.org/.../libxml2-2.7.3.tar.gz
  $ wget ftp://xmlsoft.org/.../libxslt-1.1.24.tar.gz
  $ tar xfz libxml2-2.7.3.tar.gz
  $ tar xfz libxslt-1.1.24.tar.gz
  $ cd libxml2-2.7.3
  $ ./configure --prefix ~/usr
  $ make install
  $ cd ..
  $ cd libxslt-1.1.24
  $ ./configure --prefix ~/usr
  $ make install
  $ cd ..
  $ PATH=$HOME/usr/bin:$PATH
  $ wget http://pypi.python.org/.../lxml-2.2.2.tar.gz
  $ tar xfz lxml-2.2.2.tar.gz
  $ cd lxml-2.2.2
  $ python setup.py install
  ...
  Finished processing dependencies for lxml==2.2.2

But, as I mentioned, Ian's technique is faster. :-)

1. Run the following script in your home directory. (You might want to use less to read the output.)
2. Ignore files whose date does not reflect your own activity.
3. List the oldest files in a blog post and discuss!

#!/usr/bin/env python
"""Print last-modified times of files beneath '.', oldest first."""
import os, os.path, time
paths = ( os.path.join(b,f) for (b,ds,fs) in os.walk('.') for f in fs )
for mtime, path in sorted( (os.lstat(p).st_mtime, p) for p in paths ):
    print time.strftime("%Y-%m-%d", time.localtime(mtime)), path

Only include files whose last-modified time is a date on which you really touched the file. The file's time should neither result from an error (a few files beneath my own home directory have an incorrect date of 1970-01-01), nor from unpacking someone else's archive that has old files inside of it. For example, I myself have excluded the following pair of nearly 17-year-old files because their dates reflect their age inside of the Python 3.0 source archive, instead of the actual moment last month when they became part of my home directory:

1992-03-02 ./src/Python-3.0/Demo/scripts/wh.py
1992-03-02 ./src/Python-3.0/Tools/scripts/dutree.doc

But there is no requirement that the actual content of each file you list be your own. Whether you wrote the file yourself long ago, or downloaded it from some ancient and forgotten FTP site, you have a story to share!

Within the rules given above, here are the oldest files beneath my own home directory:

# My oldest five files!
# (The links return their content.)

1989-05-17 ./archive/unixpc/cee/crobots/BCRMAD.R
1989-05-17 ./archive/unixpc/cee/crobots/BCRONE.R
1990-01-18 ./archive/unixpc/ref/train.gz
1990-01-18 ./archive/unixpc/ref/xmas.gz
1990-05-18 ./archive/unixpc/save/Rhodes/treasure

You can see that these files were moved, long ago, into an archive directory for files that are no longer part of an active project. These files all date from the era when my home directory was hosted on the Unix PC which my father, a Bell Laboratories engineer, brought home in the 1980s as our personal computer.

We should start with the zipped files from January 1990, since they actually contain even older content, from December 1987 —more than twenty years ago! My father received them as Christmas greetings from fellow engineers at the Labs. You can view xmas right in your browser, since it is a simple ASCII-art holiday greeting (the name “Merrimack Valley” at the bottom refers to the particular Bell Labs location at which my father worked). Viewing train is more difficult, since it contains a VT-100 animation that will display much too quickly if you dump the file to a modern terminal. Instead, download the file and use this Python program to display it at a more traditional speed; you might want to light a candle and play some Christmas music while the animation is displaying:

#!/usr/bin/env python
"""Display the 'train' file, slowly."""
import sys, time
for c in open('train').read():
    sys.stdout.write(c)
    sys.stdout.flush()
    time.sleep(1.0/9600.0) # (drat, really sleeps 0.01s)

Though Dad probably retained the files only through the Christmas season, I was fascinated by watching them scroll silently across the black screen in glowing, green, phosphoric characters that seemed to leave trails behind them like shooting stars, and so I kept the files long after many subsequent Christmases had come and gone.

The other three files are all plain-text files, and are all of my own making. The BCRMAD.R and BCRONE.R programs were experiments in the old crobots game, where you wrote small C-language programs to control robots that drove around on the screen and shot at each other. The names seem to refer to the fact that the first file simulates a “mad bull” that drives straight at targets shooting as fast as it can, while the second program always takes “one shot” at a target then flees to another location on the screen. This would have been one of my very first forays into C programming — perhaps my very first — and so the algorithms are, I must admit, not exactly pinnacles of sophistication.

The last file is named treasure, and it is, indeed, a real treasure of a file to have discovered this New Year's morning! It is my long-lost account, carefully annotated with the venerable mm macro package (as you can see from the troff include directive on the first line), of the day in early 1990 when I discovered buried treasure that my father had hidden as a child on my grandparent's property! I actually saw the “treasure” last week (an ancient coffee can with old toys inside) sitting on a shelf when I visited my grandmother on Christmas Day. I will have to pull it down from the shelf and put some pictures on Flickr, along with excerpts from this long-lost story of its discovery!

In the meantime: what are the oldest files under your home directory?

The rise and fall of the popularity of “My Shirt” over the four weeks from 5 October through 1 November.

And by drilling down into the list of “Referrers” beneath the graph, I was even able to discover the source of its brief popularity! As Halloween approached, people were doing hundreds of Google Image searches for “waldo costume”, “where's waldo costume”, and “where's waldo shirt”, which brought them straight to my image. As you can see in the above graph, the swelling interest did not peak until Halloween itself, after which the photo plummeted back to its more typical popularity of one or two dozen views per day.

All sorts of gems are hidden in the statistics, waiting to be discovered. For example, it was very satisfying to learn that Yahoo! Image Search considers my wedding photograph A Grandfather in Attendance to be the most important “grandfather” image on the entire web. Behind every statistic is a story about how people find, and why they wind up visiting, each of my photos. Hopefully the fun of watching my old photographs will not distract me from taking some new ones!

Term paper mill. Need $100 by Friday to keep the lights on? No sweat, if you’re a writer. Plenty of kids need ten pages on Hamlet by Thursday... more»

The article, entitled “Term Paper Artist”, alternates between hilarity and poignancy as its author shares his adventures writing hundreds of term papers for hire. But near the end, his tone suddenly becomes serious as he turns to the question of why so many students are unable to write term papers of their own. He thinks that the reason is important enough to stand alone in his article as a one-sentence paragraph. Here is his preceding paragraph, and then the zinger:

It's not that I never felt a little skeevy writing papers. Mostly it was a game, and a way to subsidize my more interesting writing. Also, I've developed a few ideas of my own over the years. I don't have the academic credentials of composition experts, but I doubt many experts spent most of a decade writing between one and five term papers a day on virtually every subject. I know something they don't know; I know why students don't understand thesis statements, argumentative writing, or proper citations.

It's because students have never read term papers.

That is his diagnosis: students never see what term papers are supposed to look like, and so they have no idea how to produce them.

As he continued on, ridiculing the idea that students can produce something of which they are never once shown a good example, I realized that his argument was exactly the same as the one I made in my recent post Reading Code: A Computer Science Curriculum: that the production of any kind of literature, whether an essay in college or an elaborate routine in a computer program, is an essentially imitative act. Without being shown excellent examples from the genre they are expected to produce, students are left in the dark about what, exactly, they are trying to generate — and, more often than not, will fail. They are never even given the opportunity to demonstrate whether they do, in fact, lack the capacity to create, because they are never shown the goal towards which they are supposed to be striving.

Because users might become confused now that PyEphem is spread across three web sites — the home page is here at rhodesmill.org, releases are posted over at the Python Package Index, and, again, the development project is now hosted at Launchpad — I have completely redesigned the PyEphem home page with the goal of making the three-site distinction clear, coherent, and easy to navigate. The new home page and documentation are generated by the wonderful Sphinx documentation engine, and I am still thrilled about how pretty my code samples look (check out the one on the PyEphem home page!) now that Sphinx is coloring them in with the renowned Pygments system.

I have simultaneously released a new version of PyEphem that includes the new Sphinx-based documentation, along with several important fixes to the software itself. From now on, rather than cluttering my own blog with every minor version of PyEphem that I might release, fans of the software should visit its News and announcements page on Launchpad and subscribe themselves to its Atom/RSS feed. You will still see the project mentioned here whenever a technical or scientific issue becomes interesting enough for me to write about; but the audience of astronomers and hobbyists who just need to know when the next version is released should not have to wade through my blog to do so!

My users have already begun transferring their questions and problems to Lauchpad, and I look forward to offering much greater accountability through a fully public development process.