But then, eventually, one has to write the actual review.

And so, a full four months after that friendly email from Packt Publishing, it is time that I sit down and put together some thoughts about Carlos de la Guardia's first book, Grok 1.0 Web Development. Carlos is a long-time veteran of the Zope and Plone communities, and Grok, of course, is the web framework that places a simple and agile convention-driven engine atop the otherwise notoriously XML-ridden Zope application framework. Grok is an important project, because it packages the technology of Python's oldest and most experienced community of web developers in a way that makes it easy to extend and use.

First, one notices that the book does exhibit the kind of typos for which Packt are well known; but not so many, in this case, as to distract me from the text. The book tackles increasingly complicated topics in roughly the same order as every book in the web development genre: first installation, then views, models, forms, and so forth. But even here one notices an initial difference! Because this is the land of Zope, the search engine chapter appears quite early in the book — in Chapter 6, in fact. While some web frameworks can only support search through the installation of a special database extension, or perhaps through an entirely separate third-party application, Grok comes with search already integrated into its object database. This lets the concept be introduced early as a natural part of the framework, rather than being a difficult exercise in product integration best left for after the user has learned deployment.

Not that third-party products are bad! In fact, I am happy to see that in many cases Carlos teaches his readers to use them. Barely twenty pages into the book, the new developer is being taught about using virtualenv to maintain an installation of Python packages; and near the end, Carlos teaches deployment using the modern and sleek mod_wsgi module. Some books feel myopically focused on one software community; this one, instead, feels expansive, a place where best practices from all across the Python ecosystem meet.

It is particularly notable that at one point Carlos steps outside of Zope entirely and, for an entire chapter, describes how Grok integrates support for relational databases through SQLAlchemy, Python's most outstanding ORM. This both reflects credit upon the flexibility of Grok — whose abstractions must be working extremely well to build atop an entirely different database model, one that is not even native to Zope — and it evidences Carlos's determination to teach a very wide set of skills in his book. In fact, he even goes in the other direction at one point, and teaches the interested developer how to use the ZODB without Grok in case they want to access it from another application!

The book tends to be quite clear in what it explains, but it sometimes feels as though a few more side trips in its examples would be welcome — it was not always clear to me whether a beginner, after plowing through an example, would understand what alternatives Carlos faced at each step, and why Carlos chose to build the solutions in the way he did.

Though the book does cover testing, it is not test driven — testing is confined to its own chapter near the end of the book, rather than motivating the design of each sample application. A more interesting omission is that the book says nothing about web services, despite the fact that XML-RPC and REST are two things that Grok does particularly well! A chapter on building web APIs would be a great subject for a second edition of this book, especially if it then proceeded to show how easily Grok can support the Ajax design pattern.

It did strike me as decidedly old-fashioned that Carlos repeats the weary canard that the Zope template language's awkwardness and verbosity are worth it, because it allows your templates to remain valid HTML, because your designers might, at any moment, want to open up and tweak a raw template in their browsers! This, of course, only works if your templates are each an entire page — if you fail to avail yourself, in other words, of any of the power of the macros, slots, and viewlets that Carlos goes on to describe later. And, in fact, many of his later templates are indeed mere un-styled HTML snippets that no designer in their right mind would view in isolation. All of which leads to the question of why Genshi, another template language popular with Grok developers, does not warrant any mention in the book — especially since the book in so many other cases is careful to mention more widely deployed Python technologies that Grok is able to use.

Finally, I can attest that a concept I never managed to learn when I was active with Grok — the tangle of concepts that surround viewlets, viewlet managers, layers, and skins — became quite clear as I sat down and read Chapter 8 in close detail. On the one topic which I could truly approach as a newcomer, therefore, I found Carlos's approach to be very direct and understandable.

Grok has good online documentation, but unless you are already an experienced developer you will have difficulty putting the pieces together into a story that runs smoothly from model to view to deployment. If you need help getting the whole picture of how Grok apps fit together, you will certainly want this book. You can purchase it from either Amazon or directly from its publisher, Packt Publishing.

I am home from a relaxing vacation to the Midwest, and while I was gone last week my excellent publishing team released the July issue Python Magazine to the world. I am particularly pleased that two of the feature articles in this issue come from important segments of the Python world that we have not heard much from in previous issues.

First, IronPython, the .NET version of Python for Windows, is the topic of Jonathan Hartley's article about acceptance testing. He illustrates that, regardless of the language in which you write your .NET application, you can deploy simple strategies to make your application testable through a Python test harness, and thereby bring Python's strong flexibility as a testing language to bear on a product that you might be writing in another .NET language.

Second, Malthe Borch, a veteran of the Zope community, shares how he has written Chameleon, one of the fastest template language implementations available for Python. By processing each template and turning it into Python bytecode before it is ever used to render a single page, Malthe eliminates a huge amount of redundant processing as that same code is used over and over again. His library is a key ingredient in the new high-efficiency web frameworks appearing in the Zope world. His work might even (fingers crossed) become one of the components that the Plone community uses as they streamline their framework and move towards a lighter and more agile “Plone 4”.

Other technical topics covered are: the Hadoop map-reduce framework; the concept of hash functions, and how they apply to Python; and the new string formatting operator which Guido hopes will replace all of the percent-signs that currently litter our code. Wrap it all up with an editorial by Steve Holden about EuroPython 2009 and an editorial by me, and you have a complete issue! If you do not find Python Magazine sitting on your local newsstand, then I hope you will avail yourself of a subscription and, as always, let me know what topics you would like to see covered in future issues. Enjoy!

The answer is that, while the necessary changes were surprisingly easy, they took lots of time to figure out because I did not find them documented in any one place. I offer the following notes to assist any other adventurers who want to experiment with porting their extension modules to 3.0. These notes might also suggest useful additions to the official documentation.

But, first, I need to issue three cautions. To develop under 3.0, you may have to forego several Python tools that you probably thought you could no longer do without. The world of 3.0 is a windswept and icy landscape from which the glaciers have just receded, and you will find the stone tools rather primitive when compared to the comforts of civilization that you enjoy under Python 2. To wit:

• I cannot find virtualenv for 3.0, which is a disaster. This means that you have to create a separate Python 3.0 install, built with a different --prefix option to ./configure, for each development environment you want to create on your box.
• I cannot find a version of the setuptools available for 3.0. This means limiting your setup.py instructions to the primitive vocabulary of the distutils package. For example, I find myself unable to run the PyEphem test suite at this late hour because I have been running it for so long with:

  $ python setup.py test

  that I am not sure how to get it running otherwise.
• Should you succeed in porting your extension module, it is not at all clear how to distribute it. I had expected either a new PyPI to spring into being — since every package will need an entirely different version under 3.0 — or for a sophisticated scheme to appear for registering one pyephem.tar.gz as the Python 2 version and another pyephem.tar.gz for 3.0. But while the most recent version of your package can mark itself as 2-compatible or 3-compatible (or both) using classifiers, there is no way to have two “most recent” versions of a package. Are we supposed to start distributing a single tar.gz that includes the source code for both Python series, and that selects the right code by detecting the interpreter version at the top of the setup.py file?

So if you make the effort to port your code right now, you might find that the shiny new version of your module is all dressed up, but has no place to go. If you experiment with the following steps, though, you will at least be ready when an official distribution channel does appear for releasing your package into the wilds of 3.0.

Four Steps To 3.0

Yes, four steps were all that were necessary to convert my quite complex extension module to Python 3.0!

1. Use PyModule_Create(). The old mechanism that I had been using to initialize my extension module, the rather clunkily-named Py_InitModule3(), happily does not even exist in the Python 3.0 header files. Instead, call the PyModule_Create() function which you can find described in the Module Initialization section of the Extending and Embedding document. And be sure to keep its return value: unlike in older Pythons, you now have to return the module object it creates as the return value from your module initialization function.

2. Adjust all Python object headers. Each type object in my code started with a macro to set up the common fields that all Python objects share. This was then followed by the ob_size field, which in my code always is always zero, and then the type name:

   /* For Python 2 */
   
   static PyTypeObject BinaryStarType = {
        PyObject_HEAD_INIT(NULL)
        0,                   /* ob_size */
        "ephem.BinaryStar",  /* tp_name */
        ...
   

   Though the Python 3.0 documentation still shows this as the way to create types, this technique will now completely fail. (The bug indicating that the documentation gets this wrong has, as its most recent comment, the helpful note “I'm lowering the priority so it doesn't block the release.”) Anyway, the solution is simple: the first two lines in the struct shown above simply have to be combined into a single macro call:

   /* For Python 3.0 */
   
   static PyTypeObject BinaryStarType = {
        PyVarObject_HEAD_INIT(NULL, 0)
        "ephem.BinaryStar",  /* tp_name */
        ...
   

   With this change, my objects are now operating fine.

3. Use plain “static.” PyEphem inherits code from an era when it was popular to use variations on the static keyword so that Python could work around problems with various troublesome C compilers. This filled my code with staticforward declarations like the one you can see at the top of the old Python 2.2 Defining New Types page. It turns out that for well-behaved compilers these were always simply synonyms for the static keyword, which is what you must replace them with when porting your code to 3.0.
4. Upgrade to Unicode. Python 3.0 makes a clean and sharp distinction between strings, which are sequences of Unicode characters, and byte arrays, which can represent anything. To reflect this sea change down at the C level, the decision was made to eliminate everything from the C API whose name started with PyString. The obvious compiler errors that result from this provide a clear signal to programmers that they have to decide, everywhere that they had been using an old-style Python string, whether they should now represent that data with the PyUnicode or the PyBytes type. This was a brilliant decision; the transition could have been a nightmare had it remained possible for old code to compile without having been properly converted!

   When migrating the PyEphem code base, I found that most of the Unicode transition was very easy. Everywhere that my code handled or created a string object, I simply changed the prefix of the function to PyUnicode and everything worked:

    PyString_Check      ... becomes ... PyUnicode_Check
    PyString_FromString ... becomes ... PyUnicode_FromString
    PyString_FromFormat ... becomes ... PyUnicode_FromFormat
   

   Well, okay, the trick does not work everywhere; this one was harder to guess:

    PyString_Size ... becomes ... PyUnicode_GET_SIZE
   

   The situations that require real thought are the places where my code needs to convert a Python string into the sort of simple ASCII character array that the underlying C library can absorb. At the moment, my code is leaning heavily on a pitiful PyUnicode_AsString() routine that I wrote just to get things working; in the morning I will have to look into doing this more correctly, including catching the error if a fancy Unicode character is present that cannot properly be converted.

Overall, I am very impressed with how quickly I was able to get my extension module compiling and running under Python 3.0. The procedure was simple — I just tried, over and over again, to build the module with:

$ python3.0 setup.py build

and then tackled the compiler errors that resulted. Once every last warning had been addressed, the module started up and operated without a single further complaint. This calls for celebration! I'm going to bed.

Download slides as PDF

They had asked me to attend so that I could present the Using Grok to Walk Like a Duck talk that I gave at PyCon 2008 back in March. They changed the title, I suppose, to better highlight why it would be of interest to the Plone community; but the change actually helped me to rethink the presentation. I wound up using only the first half of my PyCon slides. For the second half of the talk, which at PyCon had consisted of a crazy sequence of hints and tips about using adapters in your own applications, I instead did a much more successful series of slides about how adapters are actually used in Zope 3 to suit up objects for presentation on the web. I think this made the idea more concrete, and thus much easier to understand for people seeing adapters for the first time.

The talk was well enough received that I should perhaps think seriously about finding further opportunities to share Zope 3 technologies with the Plone community.

Click here to download the PDF of my presentation

Creating a Grok Instance

$ grokproject MyApp
$ ./MyApp/bin/zopectl fg

The Actual Example

• app.py

• index.pt
• contents.pt
• battleindex.pt
• characterindex.pt

Moving to a “push” model

class CharacterIndex(grok.View):
    grok.context(Character)
    grok.name('index')

<h1>The Character:
<span tal:content="context/name">Name</span></h1>
This character fought in several battles.
<ul>
  <li tal:repeat="battle context/battles">
    <a tal:attributes="href python: view.url(battle)"
       tal:content="battle/name">Name</a>
  </li>
</ul>

class CharacterIndex(grok.View):
    grok.context(Character)
    grok.name('index')
    def namespace(self):
        return {
            'character_name': self.context.name,
            'battles': [
                { 'name': battle.name,
                   'url': self.url(battle) }
                for battle in self.context.battles
                ],
            }

<html><body>
  <h1>The Character:
  <span tal:content="character_name">Name</span></h1>
  This character fought in several battles.
  <ul>
    <li tal:repeat="battle battles">
      <a tal:attributes="href battle/url"
         tal:content="battle/name">Name</a>
    </li>
  </ul>
  <a href="index">Return home</a>
</body></html>