I am enjoying my first weeks of using the Tomato Firmware. I purchased a Linksys WRT54GL wireless router because of its admirable support for third-party firmware like Tomato, which replaces the traditional Linksys setup screens with an alterative system with many more configuration options. I can also connect directly to Tomato over SSH and use it as a very small Linux system! This opens endless possibilities for writing fancy firewall rules and running small embedded applications right at the border of my home network.

The Tomato firmware uses a small DNS server named dnsmasq to answer the steady stream of domain name requests from my home computers. It converts domain names that I type, like rhodesmill.org or google.com, into the low-level IP addresses with which computers identify each other.

But I also like using hostnames for the machines sitting right in my home, even though they do not have “real names” out on the Internet. I recommend placing local hostnames inside of a top-level domain that is local to your own network. Choose a suffix that differs from all of the top-level domains that exist out on the Internet — avoid .com, .net, or .uk, for example, in favor of something like .home or .myhouse instead. How, I wondered, could I add extra host names to dnsmasq?

After my first glance through the dnsmasq documentation, I thought that a series of address options might be the best way to provide my local hosts with names. So I visited its Tomato setup screen, which is under the DHCP / DNS section of the Advanced configuration options, and entered something like this into the “Custom Configuration” file:

local-ttl=1
address=/mail.home/192.168.1.2
address=/gamebox.home/192.168.1.3
address=/printer.home/192.168.1.5

The local-ttl option is necessary because the default value, zero, encourages the host(1) command on my Linux machine to issue a warning message for every host I look up.

This setup seemed to work fine, and made it possible to use hostnames like gamebox.home when connecting from one machine to another on my home network. But I noticed delays when creating SSH connections between home machines, as well as errors in my system logs, and, as a result, discovered that this configuration was only working in the forward direction: dnsmasq knew that gamebox.home was a name for the IP address 192.168.1.3, but it could not answer the reverse question, “What name belongs to the IP address 192.168.1.3?”

I returned to the dnsmasq documentation, read more thoroughly, and learned that it publishes both forward and reverse names for hosts that it finds in the /etc/hosts file. While the Tomato firmware does not seem to support an editable /etc/hosts file whose contents will reappear when the Linksys router reboots, it does provide an auxiliary flash-based filesystem. So I was able to solve my problem in three steps:

• I visited the JFFS2 screen in dnsmasq, under the Administration section, and enabled it. This creates a writable flash-based filesystem that gets mounted at /jffs every time the router boots up.
• I created a /jffs/hosts file by connecting to Tomato with SSH and using vi to edit and save the file. It looks like a normal /etc/hosts file:

  192.168.1.2 mail.home
  192.168.1.3 gamebox.home
  192.168.1.5 printer.home
  

• Finally, I removed all of the host names from the dnsmasq configuration itself, and simply provided the path to my new hosts file instead:

  local-ttl=1
  addn-hosts=/jffs/hosts
  

With this improvement, both forward and reverse name lookups now work perfectly! To add or remove a host name in the future, I can simply re-edit the file. Though I once dreaded the inflexibility of small embedded appliances, Tomato has finally convinced me to replace the large, noisy Linux box at the edge of my network with something far more tidy and sleek.

My efforts were, in the end, successful, and you can see the result — my first two screencasts — in my previous blog entry, which I posted earlier this week.

In the hope that my toil can benefit others, let me outline the details of the process that I have worked out for creating, editing, and posting Linux screencasts. For the impatient, here are the three most important things I learned:

• Linux tools have great difficulty keeping audio and video from gradually going out of sync over several minutes. To avoid problems, always run recordmydesktop with its “on-the-fly encoding” option, and never let “ffmpeg” anywhere near your audio! This not only means that you have to use “mencoder” instead when converting between video formats, but even within mencoder you must avoid the “lavc” audio module, since its code seems to have the same problems.
• Do not attempt to directly edit the resulting Ogg/Theora video! Instead, convert first to the Digial Video (DV) format, and perform your editing there. Be prepared for the fact that DV files are enormous, and that DV pixels are not square.
• Finally, convert to something like AVI for submission to Google Video. Avoid submitting a raw Ogg/Theora file, since even though Google can decode it, their decoders will waste valuable screen space by placing an empty border around the result.

For those interested in more details, I have more to share. Keep reading!

Recording the screencast

In order to record a screencast using Linux, use recordmydesktop. I can’t tell you how to get your sound card working with it, because — much to my surprise — when I plugged my old 1990s-vintage sound card microphone into my sound card, it simply worked and my voice was right there in the recording. So I was at least spared an epic battle with my audio configuration.

The recordmydesktop tool produces video in the open Ogg/Theora format, and the results look very sharp. It names its first output file out.ogv and, if you run it again, next uses the names out.ogv.1, out.ogv.2, and so forth. This makes it easy to sort through the clips that you have produced after making several attempts at a particular scene. There are four options that I found helpful when using recordmydesktop:

• Ask for the video to be encoded during recording, rather than at the end:

  recordmydesktop --on-the-fly-encoding

  This uses more CPU during the presentation, but was necessary — at least on my machine — to produce audio and video that stayed in sync. Without it, the audio and video started to diverge visibly within a minute or so, and by minute three or four you would hear me talking about a completely different slide than the one showing on the screen.

• Ask for a full screen-shot to be made every frame:

   --full-shots

  I needed this because the presentation software that I use, KeyJnote, uses OpenGL to produce extremely smooth transitions between my PDF slides, and only full screen-shots capture the contents of OpenGL accelerated windows. (If you’re curious: I created my presentation slides with OpenOffice Impress, and then exported to PDF.)

• Choose a recording area and frame rate appropriate for the Digital Video format we will use for editing:

   -fps 29.97 -width 720 -height 528

  You might have thought that, since DV frames are 720×480 pixels in size, you should use those dimensions when running recordmydesktop. But it turns out that, while pixels on a typical computer display are square, DV pixels are noticeably taller — so our 528 vertical pixels of captured video will just barely fill the 480 vertical lines of each DV frame. I should thank Jukka Aho’s guide to video resolution, without which this aspect ratio problem might have left me very confused.

• Finally, I ask for the recording area to be offset a bit from the upper-left corner of the screen:

   -x 20 -y 40

  This offset allows me to hide the borders around my KeyJnote window, or any other window — such as an xterm — that I place in the area of my screen that is being recorded. Before doing my first real screen captures, I start up recordmydesktop, and use the box that it draws around its recording area to position my other windows so that only their content shows.

I noticed that recordmydesktop often does not record exactly the area that I specify. Instead, it seems to have a fondness for rounding the height and width to multiples of 16. But all that matters is that the recording area is close to the right size, since, again, I use the handy border that it draws around the area — and not the exact numbers I have specified — to position the windows I’m recording.

To make positioning windows in the recorded area easier, I tend to make their content slightly oversized. When running KeyJnote, for example, I use:

keyjnote -f -g 724x532 python-before-eggs.pdf

This gives me a few pixels of leeway within which to position the window without its inner edges showing in the recording. Similarly, I create my xterms with quite wide margins using the -b option, then drag the corners around until I like how much text is showing in the presentation area.

xterm -fa inconsolata -fs 16 -b 32

The fact that I only record an area 720 pixels wide leaves some space available toward the right side of my monitor, where I can place things like a narrow text editor with notes about what I want to say, which tends to make my screencast smoother than if I am making up words as I go along.

Editing the presentation

After recording my first presentation, I tried every single movie editor that I could find for Linux, and finally concluded that none of them could edit Ogg/Theora videos. They either failed to recognize the file format, or would crash during editing, or produce garbled output. The format does seem to be rather obscure outside the Free Software world, so I tried converting my video to more popular formats before editing it — and found that many of the same problems still arose!

I have concluded that the movie editors available under Linux are simply not prepared for the challenges of editing highly compressed video formats. When dealing with compressed video, merely displaying a given frame when the user clicks on it can be a lot of work — one must first rewind to the most recent keyframe, then apply the series of changes encoded in the subsequent frames. Obviously, even the simplest editing can make complete hash of the existing sequence of keyframes and updates. After doing even simple edits, I found that I could no longer even click somewhere on my recording and have a coherent image come up in the movie editor.

After hours of experimentation and research, I have adopted the Digital Video (DV) format for all of my editing. The DV format was designed with editing in mind: each frame is independently compressed. This makes it a very easy format for a program to handle as you select frames, cut out segments of video, and paste them back in somewhere else.

Although the DV format can only support a few fixed screen sizes, whose dimensions and aspect ratios were determined by the ancient properties of broadcast television, its NTSC size of 720×480 is plenty large enough to capture the detail in a typical screencast — particularly if you are going to be mixing it down to the size supported by a video-sharing site like YouTube or Google Video.

As I mentioned at the beginning of this article, always use mencoder from the MPlayer project for converting video. To create an editable DV file from the out.ogv produced by recordmydesktop, I use the command:

mencoder out.ogv -ovc libdv -oac pcm \
 -vf scale=720:480 -o editme.dv

This command also goes ahead and converts the square pixels off of your screen into pixels with the proper shape for the DV format.

Do be prepared for the fact that the DV format results in recordings that are absolutely enormous. Plan on the file consuming about a gigabyte for every four minutes of video, and simply consider this the cost of having every single frame randomly accessible during the editing process.

Which video editor should you use? I lack the time to produce a full review of each option. Instead, I will heartily recommend that you use Cinelerra, for three reasons:

• Cinelerra is a “non-linear” video editor, which means that it shows you a time-line picture of your presentation, including your audio track. It’s the audio track that’s especially important! Each of your words shows up as a visible bump in the volume, which makes it really easy to put sentences back together again if, while recording, you made a mistake, paused, and then repeated the last few words again to get them correct.
• Cinelerra does not actually alter your source video file. Each time you save your project, it just saves a tiny project.xml file listing the pieces of video and audio that you’ve clipped together on its timeline. This means that it’s not wasting effort trying to modify your multi-gigabyte DV file; instead, it’s just remembering which frames of the DV file should be rendered after all of your cutting and pasting are complete.
• Because Cinelerra doesn’t save any information inside of your video file, you can actually just delete the DV file when you’re done using Cinelerra! If you want the ability to do more editing in the future, just keep around your original Ogg/Theora file and the project.xml generated from Cinelerra, and you can always re-generate the DV file (with the command given above) and then re-run Cinelerra.

One hint: when Cinelerra starts, not only will its timeline window appear, but also a window for recording video off of a live camera. Close the recording window, which won’t help you during editing, and use the “Windows” pull-down menu to open a “Compositor” window instead. It is the Compositor window that will let you see your presentation’s video as you edit.

When you are done editing and are ready to have Cinelerra “render” the result to a new movie file, you are given a choice of output formats. I found that saving to yet another Ogg/Theora file gave the best results.

Mixing back down

Finally, before you submit your screencast to Google Video, or YouTube, or wherever, there are three final goals to be accomplished:

• Convert it into a popular format that the hosting service will appreciate. Though Google Video can now process Ogg/Theora files, it seems to put ugly margins around them. I found that submitting the same data as an AVI file produced much more attractive results. (Not that Google Video has terribly good image quality either way!)
• Adjust the resolution. Typically, you will want to reduce it from 720 pixels across to something more modest.
• Finally, in my case the volume always needs to be adjusted; my microphone is very quiet. So, I ask for the volume to be increased so that my voice hovers around half of full volume. Listen to your result, and change this number to suit your own taste.

All three of these final goals are accomplished with a single command line:

mencoder cinelerra.ogg -vf scale=640:480 \
 -af volnorm=1:0.5 -ovc lavc -oac twolame \
 -o final.avi

The final.avi file is now ready to be posted on the Web!

I hope this helps someone through the frustrations of preparing a screencast themselves. I myself will be looking for more screencast topics in the near future, so that I can use all of this advice again myself. Good luck!

The audience seemed most interested in the last section of the talk, where I discuss three techiques for debugging problems with Python’s import statement; fast-forward to around 3:00 if you want to catch that part by itself.

Next, Jeremy Jones spoke about eggs, Noah Gift introduced virtualenv, and, finally, I got back up to talk about buildout. This was probably my own favorite among the recent presentations I have given, and it’s the one I’ve worked hardest to adapt to a competent screencast:

I have prepared a supplement to the above screencast that gives additional hints and tips about using buildout, as well as a link to the source code of the module that I use as my example.

Finally, if you’re ready to see something a little less polished — something that instead of being a screencast is actually a film of me talking in front of a live audience, and gesturing and jumping around — I filled a vacant lightning-talk slot at our February PyAtl meeting with an impromptu introduction to Kinetic Style Sheets (KSS), using an example application that was still sitting on my laptop after at a Georgia Tech developer’s luncheon earlier that week:

Now I can finally turn my attention to preparing for my upcoming talk at PyCon 2008 in Chicago! I will be talking about the basic “adapter” design pattern, and how a framework like Zope 3 can facilitate its use. Stay tuned for more information!

PyEphem now also includes a small database of world cities, so that people living near one can simply call ephem.city(’Boston’) to get their longitude, latitude, and elevation, rather than having to look up the numbers themselves.

I have started using the GraphViz application, which accepts a list of nodes and arrows, and figures out how to attractively arrange them in a diagram. For example, you can very nearly produce this output:

by supplying this rather modest input file to GraphViz (most of whose length comes from my wanting particular colors)::

 digraph Application {
    rankdir=LR;
    node [shape=box,style=filled,fillcolor="#C0D0C0"];
    subgraph clusterClient {
       label=”Client”; style=filled; bgcolor=”#D0C0A0″;
       “Browser”;
    };
    subgraph clusterServer {
       label=”Server”; style=filled; bgcolor=”#D0C0A0″;
       “App”;
       “Database” [shape=DatabaseShape,peripheries=0];
    };
    “Browser” -> “App” [label="HTTP"];
    “App” -> “Database” [label="SQL"];
 }

I used the words “very nearly” because, in fact, GraphViz only knows how to draw simple shapes like rectangles, and is ignorant of the standard cylinder-shaped database symbol that I have used here by asking for a DatabaseShape. Submitting the above code to GraphViz will, normally, produce three nodes that are all rectangles. To teach it about the database shape, I had to write some PostScript.

The PostScript language is a beautifully compact version of the venerable FORTH language, and operates more or less like a Hewlett-Packard reverse Polish notation calculator hooked up to a sleeker version of the turtle from Logo. A program typically places numbers on the stack, perhaps invokes some mathematical operators on them, and finally invokes a graphics operation which uses those numbers as coordinates. For example, consider this code:

x1 x0 sub 2 div y0 lineto

This means, “place the value of the variable x1 on the stack, then the value of x0, and then subtract to remove them from the stack and leave their difference there instead; then place the number 2 on the stack and divide so that only half of the previous result remains; then place the value of y0 on the stack; and, finally, invoke the command lineto which removes the bottom two numbers from the stack and interprets them as the coordinates of a point to which it then draws a line.” In Python the same expression would be:

lineto((x1 - x0) / 2, y0)

Even small PostScript programs like my DatabaseShape.ps tend to wind up using a dozen or more different coordinates, so I always have to work first with pencil and paper to sketch the shape and define the variable names for my coordinates before then turning to my keyboard to write the code. It makes me remember my childhood, drawing things on graph paper before writing a BASIC program to draw the same shape on the screen.

One annoyance with using a user-defined shape like this with GraphViz is that one is restricted to using only its PostScript output mode, which means having to use an additional command to turn the PostScript into PDF or PNG or some other format. For example, if the example GraphViz code that I quoted above is placed in a sample.dot file, then to generate a PDF of the graph one must perform two steps:

$ dot -Tps2 -l DatabaseShape.ps sample.dot -o sample.ps
$ ps2pdf sample.ps sample.pdf

But since all the system diagrams that I need to produce at work will need to have databases in them, I’m just happy that I’ve worked out how to display the shape at all.

But because I am also really enjoying my work with the new Python web framework Grok, I decided to make it the centerpiece of my presentation and illustrate the more general principles of web development by showing how Grok works. I also gave the talk a week later at an Atlanta Plone gathering, where I knew the interest would be more on what makes Grok unique.

When I was asked for a copy of my slides, I realized that my presentation style — which involves showing several slides, moving to my editor and web browser for some actual programming, then diving back into my slides — leaves the PDF of the presentation perhaps not making as much sense by itself as it might otherwise:

Click here to download the PDF of my presentation

Hence, this post. Here I take the reader through the aspects of my talk that failed to make it on to the slides, showing both the command-line operations and the coding that illustrated and made sense of the presentation itself.

Creating a Grok Instance

After some initial slides introducing the idea of “convention over configuration,” my presentation introduces Grok itself — making good use, I think, of the symbolic value of his club — and then announces, “Let’s create a Grok instance!” This is the point at which I bring up a command prompt and show the audience how easy grokproject makes it to start a new project:

$ grokproject MyApp
$ ./MyApp/bin/zopectl fg

After zopectl has finished starting up, I point my web browser at http://localhost:8080/, and then suffer a silent moment of frustration about the fact the web page that appears is not simply the welcome message from the new Grok app. Instead, the “admin interface” comes up, and insists on being fed an application instance name before allowing me to proceed. I choose lotr as the name, then show the audience that http://localhost:8080/lotr is where the new application lives.

Before continuing with my slides, I visit the directory MyApp/src/MyApp and give the audience brief glimpses of both app.py and the app_templates directory. This introduces three items which will be critical as the slides resume: the MyApp object, the Index view, and the index.pt page template. With these three items pointed out, I am ready to proceed with my slides and discuss how the object, view, and page template together form Grok’s “Threefold Way” of making objects available on the web.

The Actual Example

Only a few slides later, the time arrives for a more sophisticated example. The presentation exaggerates when it instructs me to “go add some models and further views to your application.” I do not, in fact, make the audience watch me write code, but present them with a finished and working example that I prepare before the presentation. You can create it for yourself by putting this file in place of the one grokproject created:

• app.py

and then putting these four files in app_templates:

• index.pt
• contents.pt
• battleindex.pt
• characterindex.pt

The resulting application, though tiny, was probably too complex for my target audience — many eyes, I noticed, glazed over when seeing Zope Page Templates for the first time — but four views is really barely enough to illustrate both that you can have multiple views on the same object (MyApp has both an /index and a /contents), and that several objects can have views with the same name (Battle and Character have different /index views). The example also shows clearly that if you declare four views, you need to write four page templates to go with them.

I emphasize for the audience that most of the 56 lines of app.py are concerned with creating the violence-themed object model; only the fifteen lines at the bottom are interesting Grok code, creating four very slender views. This is where I point out “convention over configuration” at work: all of the page templates get discovered without having to be named, and grok.name() only has to be employed when we want the view’s URL name to differ from the name of the class.

After demonstrating that the application does, in fact, work, we return to the slides and start examining the relationship between page templates and their views.

Moving to a “push” model

In the presentation I distinguish between “muscular views” and “muscular templates”; perhaps I should have used instead the more conventional idea of templates that “pull” data versus those to which data are “pushed.”

When I reach the slide that reminds me to, “show them your anemic View classes,” I return to my editor and point out that, in the example application, each view has almost no purpose but to associate a particular object class with a particular page template. There is very little information in a declaration like:

class CharacterIndex(grok.View):
    grok.context(Character)
    grok.name('index')

Whereas there is plenty going on in a template like characterindex.pt; every TAL reference it makes is either to the raw context object itself, or an object it reaches through the context! Take a look:

<h1>The Character:
<span tal:content="context/name">Name</span></h1>
This character fought in several battles.
<ul>
  <li tal:repeat="battle context/battles">
    <a tal:attributes="href python: view.url(battle)"
       tal:content="battle/name">Name</a>
  </li>
</ul>

The slides then continue, pointing out that this makes for very quick coding at the cost of several disadvantages. When the fourth and final intermission arrives (“go make your CharacterIndex View more muscular”), I return to the window where my editor has app.py open and rewrite the CharacterIndex to look more like this:

class CharacterIndex(grok.View):
    grok.context(Character)
    grok.name('index')
    def namespace(self):
        return {
            'character_name': self.context.name,
            'battles': [
                { 'name': battle.name,
                   'url': self.url(battle) }
                for battle in self.context.battles
                ],
            }

Note what this does: it takes all of the work that the page template was doing in order to fetch the character’s name, and the names and URLs of the battles in which the character fought, and performs that access in pure Python and in exactly once place. The values in the dictionary that is being passed — character_name and battle_info — are, if you will examine them carefully, composed of nothing but primitive Python types! They consist solely of lists and dictionaries containing strings. They contain no objects for the template to browse at its leisure; instead, every bit of pertinent data in our objects is reduced to harmless data structures which cannot be used to affect the original objects in any way. This means that the characterindex.pt template can be rewritten both more safely, and also more simply:

<html><body>
  <h1>The Character:
  <span tal:content="character_name">Name</span></h1>
  This character fought in several battles.
  <ul>
    <li tal:repeat="battle battles">
      <a tal:attributes="href battle/url"
         tal:content="battle/name">Name</a>
    </li>
  </ul>
  <a href="index">Return home</a>
</body></html>

And after showing the audience that the rewritten view and the rewritten page template work together, I return to my slides and — finally! — conclude my presentation.

The presentation seemed to weigh in at around forty minutes. Next time, I think I will tackle fewer concepts at once. But I do like making lots of use of the picture of the cave man, and his club.

error: Python was built with version 7.1 of Visual Studio, and extensions need to be built with the same version of the compiler, but it isn’t installed.

And, as I myself do not have Visual Studio on the small Windows machine that I deign to own for the sake of my photo printer, I have never been able to offer my users much help. But earlier this year, a helpful PyEphem user named Jeff Kowalczyk emailed me a link to Philip von Weitershausen’s post “Cheap binary Windows eggs”, which describes a method for building Python extensions using a freely available compiler.

Aside from the difficulty that the Python setuptools ignored the pydistutils.cfg file he recommended, making it necessary for me to name the compiler on the command line, his instructions produced a PyEphem windows egg with only a few steps:

• I installed Python for Windows.
• I installed the setuptools package using its Windows .exe installer (scroll down the page to find it).
• I installed the MinGW compiler.
• I installed LibArchive for Windows.
• I added to my Windows Path:

  C:\Python25;C:\MingW\bin;C:\Program Files\GnuWin32\bin

• I downloaded the PyEphem source archive.
• I brought up a command window and ran:

  C:\dev> bsdtar -xf pyephem-3.7.2.1.tar.gz
  C:\dev> cd pyephem-3.7.2.1
  C:\dev\pyephem-3.7.2.1> python setup.py build --compiler=mingw32
  C:\dev\pyephem-3.7.2.1> python setup.py bdist_egg
  

• I uploaded the egg in the dist directory to the Cheese Shop.

Windows users will now find my Windows binary eggs on the Cheese Shop PyEphem entry!

Always keep in mind that behind a small triumph like this one for a Free Software developer like myself stands a great mountain-mass of work by hundreds of others in the community. In this case one helpful user lead to one informative blog post which lead to a solid and available tool — MinGW — that is a descendant of the GCC compiler Richard Stallman himself started writing over twenty years ago! All of which allowed the delivery, to a new (and closed) platform, of my Python interface to astronomy routines which Elwood Charles Downey has been maintaining for more than a quarter-century.

Online chess relieves the game of two horrors for me, of which I had not even been aware until I noticed them during this first online game because of their absence: the horror of a waiting opponent, and the curse of having to wait for them in turn! It turns out that, for a novice like myself who can take upwards of a half-hour to even begin to appreciate the complexity of a given position, to play a live game is only to be rushed through a series of bad decisions. But now, over morning coffee, I can ponder the board for as long as I wish, and can therefore begin — just begin — to glimpse the beauty of the complex possibilies that each move offers. And then, my move complete, I can go about my day without being fixed, inactive, in a chair while my opponent weighs his decision.

I will have to thank Ilan for telling me about this — maybe even by playing him in person sometime, like he originally asked!