I would like some advice from Zope and Plone folks about how to create forms that are not only easy for other developers to specialize, but which allow several specializations to be composed together. While I have used zope.formlib and z3c.form before for simple tasks, I have not yet been able to tell whether they support these more advanced kinds of operations.

Some background: I am doing some work on GetPaid for Plone with the generous funding of Derek Richardson who, if his dreams had not carried him away from grad school at the end of the Spring semester, would have tackled this same work as part of the 2009 Google Summer of Code.

The current mechanisms that GetPaid provides for customizing its checkout process are very primitive, and my task is to improve them. That is why I have been thinking about customizing forms.

Plugging in payment processors

When a Plone user has filled their shopping cart with goods and presses the “Checkout” button, GetPaid begins stepping them through a checkout process of several forms that must all be satisfied before, finally, the user's credit card will be charged and their purchase recorded. GetPaid's current definition of a “Payment Processor” requires only that it provide three functions authorize(), capture(), and refund() that talk on GetPaid's behalf to an online merchant account.

This design works fine for traditional payment processors that are comfortable letting Plone handle the credit card information, but these days site owners usually want to delegate the dangerous task of handling card numbers to an “off-site” payment site. And these off-site payment processors turn out to be but one example of the many ways in which GetPaid expansion modules want to customize (or hijack?) the GetPaid checkout wizard. Consider the following scenarios:

• 
  Something like Google Checkout is comfortable taking the whole checkout process out of your hands; the “Checkout” button at the bottom of your shopping cart is replaced with a branded “Google Checkout” button that sends the user right to Google. Currently in Getpaid, that button can only be customized by providing completely new page templates for the shopping cart that duplicate everything already there, but which have a Google image on the checkout button.
• Services like PxPay (at least the way we use them) let the GetPaid checkout process take care of collecting the customer's address and shipping information, but then need the user redirected off of the Plone site. Again, entire views get replaced just to provide an alternate submit button.
• Many site owners want to allow several payment processors, with the user choosing which one gets used since the user might already have an account with Google Checkout or be previously registered with PayPal. This suggests the need for a “meta” payment processor which can be given a list of payment processors and, when the user reaches the first checkout step for which those payment processors differ, presents the user with a screen from which they select the processor that they like.
• Some sites have specialized information that they need to collect from users: they want to ask for extra address information, or extra shipping options for certain products, or need more contact information from the user when they are checking out.

So, currently, in pretty much every one of the above situations, the answer to, “How do I tweak the GetPaid checkout process?” is, “Override the entire view or module that you need to customize, by providing your own copy that looks exactly the same as the default one, except for the one thing you need changed.”

If you are curious, I have created a diagram showing which payment processors have to override which parts of GetPaid.

Not only does this wanton overriding mean that modules have to meticulously track the changes that GetPaid makes to the forms they are overriding, but it means that customizations are inherently not composable: you can't put two of them together.

The problem of composability is a big one. Imagine that you live in Argentina and want to install a module that adjusts GetPaid's mailing address form to match the way postal addresses are formatted locally, but that you also want to use an off-site payment processor that needs to insert its custom “Checkout” button on the bottom of the mailing address form. Since the two modules will provide competing versions of the entire mailing address form, only one of them can “win”, and the customization that the other module wants to present will not be displayed.

Four crazy ideas

How might checkout process customizations become operations that could be combined and composed, instead of each one of them being a fragile, blanket replacement of an entire part of GetPaid? Here are four ideas that I have thought through today as I sat with my trusty pencil and yellow pad:

• Plug-in points. Currently, GetPaid just has those three functions mentioned above (authorize(), capture(), and refund()) where payment processors can plug into its checkout logic. One way of solving the problems we are encountering is simply to provide a lot more plug-in points where GetPaid's default behavior can be adjusted or subverted. It looks like this would have to include separate views for every button, for the body of every major form, and for the transitions between forms. It is doable within the time I have left for this project, and serves as my default, conservative option if nothing more spectacular can happen right now. But it's very brittle: every time in the future that someone thinks of a new way of adjusting or tweaking GetPaid, they have to ask us to create a new plug-in point.
• Request-level wrapper. It might be possible (I mention this technique only for the sake of completeness, to spur our imaginations) to implement an off-site change processor as a wrapper around GetPaid that interacted at the bare level of incoming HTTP requests and outgoing HTML pages or redirections. If you have chosen Google Checkout, a special wrapper would be installed that would rewrite the shopping cart “Checkout” button with its own. If you need another field added to the final checkout page, you would create a wrapper that adds the field when it sees the HTML for that form going out, and that intercepts the additional field on its way back in. Such wrappers would at least be composable: you could have several active at once, that each made their own modification to a single page so that the user could see an address form that was formatted for Argentina and that had the correct checkout button on the bottom.
• Wrap a form that provided an API. What do forms “look like” to the code that calls them? If we imagine a form library where a Form is separate from the FormRenderer that introspects and traverses the properties of each form to turn it into HTML, then each GetPaid extension could be a wrapper around the GetPaid form object(s) that passes through most requests intact, leaving most parts of most forms alone, but that intercepts certain requests and returns different answers than GetPaid would return. The Argentina module would wrap the default GetPaid shipping form and, when traversed for its sub-forms and fields, would return a series of fields specific to Argentina. The off-site payment processor wrapper would just adjust the properties of the checkout button before returning them. And, these two wrappers would be composable! If you wrapped them, in any order, around the default GetPaid checkout forms, then you could enjoy both customizations.
• Forms as modifiable objects. What if forms — or even a small collection of forms, like the GetPaid checkout process — were objects that you could adjust and edit and modify before asking them to be rendered? Then, each GetPaid customization could be written as a small routine that takes, as its argument, a description of the entire checkout process, and that modifies that description by tweaking, adding, and removing everything that it needs to. Before rendering a view for the user, GetPaid would construct a copy of the checkout process, and then pass it, in turn, to each module. The Argentina module would reach in and tweak some address fields. The off-site processor would reach in and adjust the checkout button sitting below them. And the result could then be rendered. The one problem, when we compare this approach to the previous scenario, would be how to intercept new data that would be coming in as a result of added form fields. I suppose the form descriptions would have to include callback functions for processing them, and that extension modules would just add their own callbacks.

Is there anything here worth trying?

These are just the ideas that have come to me today, as I've scribbled and sketched and sat down to re-read all of the design patterns in the Gang of Four Book looking for inspiration. (You might recognize the third option above as something like the “decorator” design pattern!) Does zope.formlib or z3c.form or anything else like that in the Zope universe provide the kind of services I would need from a form library to implement the ideas outlined above?

If you're a Zope or Plone person with an inkling of an answer, then please let me know!