But then, eventually, one has to write the actual review.

And so, a full four months after that friendly email from Packt Publishing, it is time that I sit down and put together some thoughts about Carlos de la Guardia's first book, Grok 1.0 Web Development. Carlos is a long-time veteran of the Zope and Plone communities, and Grok, of course, is the web framework that places a simple and agile convention-driven engine atop the otherwise notoriously XML-ridden Zope application framework. Grok is an important project, because it packages the technology of Python's oldest and most experienced community of web developers in a way that makes it easy to extend and use.

First, one notices that the book does exhibit the kind of typos for which Packt are well known; but not so many, in this case, as to distract me from the text. The book tackles increasingly complicated topics in roughly the same order as every book in the web development genre: first installation, then views, models, forms, and so forth. But even here one notices an initial difference! Because this is the land of Zope, the search engine chapter appears quite early in the book — in Chapter 6, in fact. While some web frameworks can only support search through the installation of a special database extension, or perhaps through an entirely separate third-party application, Grok comes with search already integrated into its object database. This lets the concept be introduced early as a natural part of the framework, rather than being a difficult exercise in product integration best left for after the user has learned deployment.

Not that third-party products are bad! In fact, I am happy to see that in many cases Carlos teaches his readers to use them. Barely twenty pages into the book, the new developer is being taught about using virtualenv to maintain an installation of Python packages; and near the end, Carlos teaches deployment using the modern and sleek mod_wsgi module. Some books feel myopically focused on one software community; this one, instead, feels expansive, a place where best practices from all across the Python ecosystem meet.

It is particularly notable that at one point Carlos steps outside of Zope entirely and, for an entire chapter, describes how Grok integrates support for relational databases through SQLAlchemy, Python's most outstanding ORM. This both reflects credit upon the flexibility of Grok — whose abstractions must be working extremely well to build atop an entirely different database model, one that is not even native to Zope — and it evidences Carlos's determination to teach a very wide set of skills in his book. In fact, he even goes in the other direction at one point, and teaches the interested developer how to use the ZODB without Grok in case they want to access it from another application!

The book tends to be quite clear in what it explains, but it sometimes feels as though a few more side trips in its examples would be welcome — it was not always clear to me whether a beginner, after plowing through an example, would understand what alternatives Carlos faced at each step, and why Carlos chose to build the solutions in the way he did.

Though the book does cover testing, it is not test driven — testing is confined to its own chapter near the end of the book, rather than motivating the design of each sample application. A more interesting omission is that the book says nothing about web services, despite the fact that XML-RPC and REST are two things that Grok does particularly well! A chapter on building web APIs would be a great subject for a second edition of this book, especially if it then proceeded to show how easily Grok can support the Ajax design pattern.

It did strike me as decidedly old-fashioned that Carlos repeats the weary canard that the Zope template language's awkwardness and verbosity are worth it, because it allows your templates to remain valid HTML, because your designers might, at any moment, want to open up and tweak a raw template in their browsers! This, of course, only works if your templates are each an entire page — if you fail to avail yourself, in other words, of any of the power of the macros, slots, and viewlets that Carlos goes on to describe later. And, in fact, many of his later templates are indeed mere un-styled HTML snippets that no designer in their right mind would view in isolation. All of which leads to the question of why Genshi, another template language popular with Grok developers, does not warrant any mention in the book — especially since the book in so many other cases is careful to mention more widely deployed Python technologies that Grok is able to use.

Finally, I can attest that a concept I never managed to learn when I was active with Grok — the tangle of concepts that surround viewlets, viewlet managers, layers, and skins — became quite clear as I sat down and read Chapter 8 in close detail. On the one topic which I could truly approach as a newcomer, therefore, I found Carlos's approach to be very direct and understandable.

Grok has good online documentation, but unless you are already an experienced developer you will have difficulty putting the pieces together into a story that runs smoothly from model to view to deployment. If you need help getting the whole picture of how Grok apps fit together, you will certainly want this book. You can purchase it from either Amazon or directly from its publisher, Packt Publishing.

I would like some advice from Zope and Plone folks about how to create forms that are not only easy for other developers to specialize, but which allow several specializations to be composed together. While I have used zope.formlib and z3c.form before for simple tasks, I have not yet been able to tell whether they support these more advanced kinds of operations.

Some background: I am doing some work on GetPaid for Plone with the generous funding of Derek Richardson who, if his dreams had not carried him away from grad school at the end of the Spring semester, would have tackled this same work as part of the 2009 Google Summer of Code.

The current mechanisms that GetPaid provides for customizing its checkout process are very primitive, and my task is to improve them. That is why I have been thinking about customizing forms.

Plugging in payment processors

When a Plone user has filled their shopping cart with goods and presses the “Checkout” button, GetPaid begins stepping them through a checkout process of several forms that must all be satisfied before, finally, the user's credit card will be charged and their purchase recorded. GetPaid's current definition of a “Payment Processor” requires only that it provide three functions authorize(), capture(), and refund() that talk on GetPaid's behalf to an online merchant account.

This design works fine for traditional payment processors that are comfortable letting Plone handle the credit card information, but these days site owners usually want to delegate the dangerous task of handling card numbers to an “off-site” payment site. And these off-site payment processors turn out to be but one example of the many ways in which GetPaid expansion modules want to customize (or hijack?) the GetPaid checkout wizard. Consider the following scenarios:

• 
  Something like Google Checkout is comfortable taking the whole checkout process out of your hands; the “Checkout” button at the bottom of your shopping cart is replaced with a branded “Google Checkout” button that sends the user right to Google. Currently in Getpaid, that button can only be customized by providing completely new page templates for the shopping cart that duplicate everything already there, but which have a Google image on the checkout button.
• Services like PxPay (at least the way we use them) let the GetPaid checkout process take care of collecting the customer's address and shipping information, but then need the user redirected off of the Plone site. Again, entire views get replaced just to provide an alternate submit button.
• Many site owners want to allow several payment processors, with the user choosing which one gets used since the user might already have an account with Google Checkout or be previously registered with PayPal. This suggests the need for a “meta” payment processor which can be given a list of payment processors and, when the user reaches the first checkout step for which those payment processors differ, presents the user with a screen from which they select the processor that they like.
• Some sites have specialized information that they need to collect from users: they want to ask for extra address information, or extra shipping options for certain products, or need more contact information from the user when they are checking out.

So, currently, in pretty much every one of the above situations, the answer to, “How do I tweak the GetPaid checkout process?” is, “Override the entire view or module that you need to customize, by providing your own copy that looks exactly the same as the default one, except for the one thing you need changed.”

If you are curious, I have created a diagram showing which payment processors have to override which parts of GetPaid.

Not only does this wanton overriding mean that modules have to meticulously track the changes that GetPaid makes to the forms they are overriding, but it means that customizations are inherently not composable: you can't put two of them together.

The problem of composability is a big one. Imagine that you live in Argentina and want to install a module that adjusts GetPaid's mailing address form to match the way postal addresses are formatted locally, but that you also want to use an off-site payment processor that needs to insert its custom “Checkout” button on the bottom of the mailing address form. Since the two modules will provide competing versions of the entire mailing address form, only one of them can “win”, and the customization that the other module wants to present will not be displayed.

Four crazy ideas

How might checkout process customizations become operations that could be combined and composed, instead of each one of them being a fragile, blanket replacement of an entire part of GetPaid? Here are four ideas that I have thought through today as I sat with my trusty pencil and yellow pad:

• Plug-in points. Currently, GetPaid just has those three functions mentioned above (authorize(), capture(), and refund()) where payment processors can plug into its checkout logic. One way of solving the problems we are encountering is simply to provide a lot more plug-in points where GetPaid's default behavior can be adjusted or subverted. It looks like this would have to include separate views for every button, for the body of every major form, and for the transitions between forms. It is doable within the time I have left for this project, and serves as my default, conservative option if nothing more spectacular can happen right now. But it's very brittle: every time in the future that someone thinks of a new way of adjusting or tweaking GetPaid, they have to ask us to create a new plug-in point.
• Request-level wrapper. It might be possible (I mention this technique only for the sake of completeness, to spur our imaginations) to implement an off-site change processor as a wrapper around GetPaid that interacted at the bare level of incoming HTTP requests and outgoing HTML pages or redirections. If you have chosen Google Checkout, a special wrapper would be installed that would rewrite the shopping cart “Checkout” button with its own. If you need another field added to the final checkout page, you would create a wrapper that adds the field when it sees the HTML for that form going out, and that intercepts the additional field on its way back in. Such wrappers would at least be composable: you could have several active at once, that each made their own modification to a single page so that the user could see an address form that was formatted for Argentina and that had the correct checkout button on the bottom.
• Wrap a form that provided an API. What do forms “look like” to the code that calls them? If we imagine a form library where a Form is separate from the FormRenderer that introspects and traverses the properties of each form to turn it into HTML, then each GetPaid extension could be a wrapper around the GetPaid form object(s) that passes through most requests intact, leaving most parts of most forms alone, but that intercepts certain requests and returns different answers than GetPaid would return. The Argentina module would wrap the default GetPaid shipping form and, when traversed for its sub-forms and fields, would return a series of fields specific to Argentina. The off-site payment processor wrapper would just adjust the properties of the checkout button before returning them. And, these two wrappers would be composable! If you wrapped them, in any order, around the default GetPaid checkout forms, then you could enjoy both customizations.
• Forms as modifiable objects. What if forms — or even a small collection of forms, like the GetPaid checkout process — were objects that you could adjust and edit and modify before asking them to be rendered? Then, each GetPaid customization could be written as a small routine that takes, as its argument, a description of the entire checkout process, and that modifies that description by tweaking, adding, and removing everything that it needs to. Before rendering a view for the user, GetPaid would construct a copy of the checkout process, and then pass it, in turn, to each module. The Argentina module would reach in and tweak some address fields. The off-site processor would reach in and adjust the checkout button sitting below them. And the result could then be rendered. The one problem, when we compare this approach to the previous scenario, would be how to intercept new data that would be coming in as a result of added form fields. I suppose the form descriptions would have to include callback functions for processing them, and that extension modules would just add their own callbacks.

Is there anything here worth trying?

These are just the ideas that have come to me today, as I've scribbled and sketched and sat down to re-read all of the design patterns in the Gang of Four Book looking for inspiration. (You might recognize the third option above as something like the “decorator” design pattern!) Does zope.formlib or z3c.form or anything else like that in the Zope universe provide the kind of services I would need from a form library to implement the ideas outlined above?

If you're a Zope or Plone person with an inkling of an answer, then please let me know!

I am home from a relaxing vacation to the Midwest, and while I was gone last week my excellent publishing team released the July issue Python Magazine to the world. I am particularly pleased that two of the feature articles in this issue come from important segments of the Python world that we have not heard much from in previous issues.

First, IronPython, the .NET version of Python for Windows, is the topic of Jonathan Hartley's article about acceptance testing. He illustrates that, regardless of the language in which you write your .NET application, you can deploy simple strategies to make your application testable through a Python test harness, and thereby bring Python's strong flexibility as a testing language to bear on a product that you might be writing in another .NET language.

Second, Malthe Borch, a veteran of the Zope community, shares how he has written Chameleon, one of the fastest template language implementations available for Python. By processing each template and turning it into Python bytecode before it is ever used to render a single page, Malthe eliminates a huge amount of redundant processing as that same code is used over and over again. His library is a key ingredient in the new high-efficiency web frameworks appearing in the Zope world. His work might even (fingers crossed) become one of the components that the Plone community uses as they streamline their framework and move towards a lighter and more agile “Plone 4”.

Other technical topics covered are: the Hadoop map-reduce framework; the concept of hash functions, and how they apply to Python; and the new string formatting operator which Guido hopes will replace all of the percent-signs that currently litter our code. Wrap it all up with an editorial by Steve Holden about EuroPython 2009 and an editorial by me, and you have a complete issue! If you do not find Python Magazine sitting on your local newsstand, then I hope you will avail yourself of a subscription and, as always, let me know what topics you would like to see covered in future issues. Enjoy!

Further hypothesis: this relationship is non-linear. My guess is that frameworks which take, say, four seconds to restart probably get no more IRC attention than frameworks which take two seconds to restart, so adding mere seconds of delay to your framework's startup routine is not going to generate thriving community. There is, instead, a threshold — probably around ten or twelve seconds — above which the delay feels so long that the developer can, in good conscience towards their employer, switch over into IRC during a restart without feeling like they hurt their productivity.

The answer is that, while the necessary changes were surprisingly easy, they took lots of time to figure out because I did not find them documented in any one place. I offer the following notes to assist any other adventurers who want to experiment with porting their extension modules to 3.0. These notes might also suggest useful additions to the official documentation.

But, first, I need to issue three cautions. To develop under 3.0, you may have to forego several Python tools that you probably thought you could no longer do without. The world of 3.0 is a windswept and icy landscape from which the glaciers have just receded, and you will find the stone tools rather primitive when compared to the comforts of civilization that you enjoy under Python 2. To wit:

• I cannot find virtualenv for 3.0, which is a disaster. This means that you have to create a separate Python 3.0 install, built with a different --prefix option to ./configure, for each development environment you want to create on your box.
• I cannot find a version of the setuptools available for 3.0. This means limiting your setup.py instructions to the primitive vocabulary of the distutils package. For example, I find myself unable to run the PyEphem test suite at this late hour because I have been running it for so long with:

  $ python setup.py test

  that I am not sure how to get it running otherwise.
• Should you succeed in porting your extension module, it is not at all clear how to distribute it. I had expected either a new PyPI to spring into being — since every package will need an entirely different version under 3.0 — or for a sophisticated scheme to appear for registering one pyephem.tar.gz as the Python 2 version and another pyephem.tar.gz for 3.0. But while the most recent version of your package can mark itself as 2-compatible or 3-compatible (or both) using classifiers, there is no way to have two “most recent” versions of a package. Are we supposed to start distributing a single tar.gz that includes the source code for both Python series, and that selects the right code by detecting the interpreter version at the top of the setup.py file?

So if you make the effort to port your code right now, you might find that the shiny new version of your module is all dressed up, but has no place to go. If you experiment with the following steps, though, you will at least be ready when an official distribution channel does appear for releasing your package into the wilds of 3.0.

Four Steps To 3.0

Yes, four steps were all that were necessary to convert my quite complex extension module to Python 3.0!

1. Use PyModule_Create(). The old mechanism that I had been using to initialize my extension module, the rather clunkily-named Py_InitModule3(), happily does not even exist in the Python 3.0 header files. Instead, call the PyModule_Create() function which you can find described in the Module Initialization section of the Extending and Embedding document. And be sure to keep its return value: unlike in older Pythons, you now have to return the module object it creates as the return value from your module initialization function.

2. Adjust all Python object headers. Each type object in my code started with a macro to set up the common fields that all Python objects share. This was then followed by the ob_size field, which in my code always is always zero, and then the type name:

   /* For Python 2 */
   
   static PyTypeObject BinaryStarType = {
        PyObject_HEAD_INIT(NULL)
        0,                   /* ob_size */
        "ephem.BinaryStar",  /* tp_name */
        ...
   

   Though the Python 3.0 documentation still shows this as the way to create types, this technique will now completely fail. (The bug indicating that the documentation gets this wrong has, as its most recent comment, the helpful note “I'm lowering the priority so it doesn't block the release.”) Anyway, the solution is simple: the first two lines in the struct shown above simply have to be combined into a single macro call:

   /* For Python 3.0 */
   
   static PyTypeObject BinaryStarType = {
        PyVarObject_HEAD_INIT(NULL, 0)
        "ephem.BinaryStar",  /* tp_name */
        ...
   

   With this change, my objects are now operating fine.

3. Use plain “static.” PyEphem inherits code from an era when it was popular to use variations on the static keyword so that Python could work around problems with various troublesome C compilers. This filled my code with staticforward declarations like the one you can see at the top of the old Python 2.2 Defining New Types page. It turns out that for well-behaved compilers these were always simply synonyms for the static keyword, which is what you must replace them with when porting your code to 3.0.
4. Upgrade to Unicode. Python 3.0 makes a clean and sharp distinction between strings, which are sequences of Unicode characters, and byte arrays, which can represent anything. To reflect this sea change down at the C level, the decision was made to eliminate everything from the C API whose name started with PyString. The obvious compiler errors that result from this provide a clear signal to programmers that they have to decide, everywhere that they had been using an old-style Python string, whether they should now represent that data with the PyUnicode or the PyBytes type. This was a brilliant decision; the transition could have been a nightmare had it remained possible for old code to compile without having been properly converted!

   When migrating the PyEphem code base, I found that most of the Unicode transition was very easy. Everywhere that my code handled or created a string object, I simply changed the prefix of the function to PyUnicode and everything worked:

    PyString_Check      ... becomes ... PyUnicode_Check
    PyString_FromString ... becomes ... PyUnicode_FromString
    PyString_FromFormat ... becomes ... PyUnicode_FromFormat
   

   Well, okay, the trick does not work everywhere; this one was harder to guess:

    PyString_Size ... becomes ... PyUnicode_GET_SIZE
   

   The situations that require real thought are the places where my code needs to convert a Python string into the sort of simple ASCII character array that the underlying C library can absorb. At the moment, my code is leaning heavily on a pitiful PyUnicode_AsString() routine that I wrote just to get things working; in the morning I will have to look into doing this more correctly, including catching the error if a fancy Unicode character is present that cannot properly be converted.

Overall, I am very impressed with how quickly I was able to get my extension module compiling and running under Python 3.0. The procedure was simple — I just tried, over and over again, to build the module with:

$ python3.0 setup.py build

and then tackled the compiler errors that resulted. Once every last warning had been addressed, the module started up and operated without a single further complaint. This calls for celebration! I'm going to bed.

Download slides as PDF

They had asked me to attend so that I could present the Using Grok to Walk Like a Duck talk that I gave at PyCon 2008 back in March. They changed the title, I suppose, to better highlight why it would be of interest to the Plone community; but the change actually helped me to rethink the presentation. I wound up using only the first half of my PyCon slides. For the second half of the talk, which at PyCon had consisted of a crazy sequence of hints and tips about using adapters in your own applications, I instead did a much more successful series of slides about how adapters are actually used in Zope 3 to suit up objects for presentation on the web. I think this made the idea more concrete, and thus much easier to understand for people seeing adapters for the first time.

The talk was well enough received that I should perhaps think seriously about finding further opportunities to share Zope 3 technologies with the Plone community.

Click here to download the PDF of my presentation

Creating a Grok Instance

$ grokproject MyApp
$ ./MyApp/bin/zopectl fg

The Actual Example

• app.py

• index.pt
• contents.pt
• battleindex.pt
• characterindex.pt

Moving to a “push” model

class CharacterIndex(grok.View):
    grok.context(Character)
    grok.name('index')

<h1>The Character:
<span tal:content="context/name">Name</span></h1>
This character fought in several battles.
<ul>
  <li tal:repeat="battle context/battles">
    <a tal:attributes="href python: view.url(battle)"
       tal:content="battle/name">Name</a>
  </li>
</ul>

class CharacterIndex(grok.View):
    grok.context(Character)
    grok.name('index')
    def namespace(self):
        return {
            'character_name': self.context.name,
            'battles': [
                { 'name': battle.name,
                   'url': self.url(battle) }
                for battle in self.context.battles
                ],
            }

<html><body>
  <h1>The Character:
  <span tal:content="character_name">Name</span></h1>
  This character fought in several battles.
  <ul>
    <li tal:repeat="battle battles">
      <a tal:attributes="href battle/url"
         tal:content="battle/name">Name</a>
    </li>
  </ul>
  <a href="index">Return home</a>
</body></html>