While the first twenty revisions of my branch deal with simple Python 3.0 mechanics, I must congratulate Mr. Schüttel for most of the improvements in the subsequent dozen revisions. We not only exchanged emails all day as I produced one revision after another that I needed him to test, but he then joined me on IRC tonight until 3:30am in his time zone as we worked out the last problems.

The issues were all related to localization under OS X. The astronomy library underlying PyEphem used the C-language functions sscanf() and atof() to turn strings into numbers, and it turns out that these functions are very sensitive to locale under the specific combination of OS X and Python 3.0! Because of his locale, the functions wanted a comma instead of a period to separate whole numbers from their decimal fraction (so that π would have to be input like 3,141 rather than like 3.141). They also wanted month name abbreviations to be in German rather than English, ruining my test cases that check planet positions against Naval Observatory tables which use English month abbreviations like Jan, Feb, and Mar with very little regard for how the months would be spelled in German. We are still mystified by the combination of Python version and operation system that were necessary to cause this problem:

Python 2.6 under Linux: tests passed
Python 3.0 under Linux: tests passed
Python 2.6 under OS X: tests passed
Python 3.0 under OS X: broken: sscanf()/atof() change with locale

But the tests worked fine if he put LANG=C on the command line.

To work around this problem, I discovered a wonderful PyOS_ascii_strtod() function in Python's C library that avoided all of the problems that I was having with localization, and so I gradually rewrote the astronomy routines to use that function instead. Fixing the problems with month names was easier; instead of trying to make Python convert month abbreviations by passing the '%b' conversion character to time.strptime(), I simply converted months to integers myself and then passed the integers in with the simpler '%m' format character. It was only late at night that we finally tracked down every routine that was misbehaving.

The final puzzle was how to release my new software. The Python Package Index does not yet allow a project to offer separate source code archives for both the Python 2.x and the 3.0 version of a project. There might be some clever way of storing both versions of the source code in the same .tar.gz file, but the packages I see on PyPI that already support both 2.x and 3.0 contain enough #ifdef statements to convince me that I want to keep my branches separate!

I was saved by a peculiarity of my project. Though it provides the Python package named ephem, the actual project on PyPI has always been named “PyEphem” instead. The product has to have the “Py” in front, you see, to distinguish it both from the original text-screen ephem command and the modern XEphem graphical application, and my first instinct had been to name the project after the product. But as I gained experience with PyPI, it seemed more and more awkward that programmers who wanted their programs to be able to:

import ephem

had to remember to type something different when installing:

easy_install pyephem

Since I had been wanting to switch PyPI names anyway, I suddenly saw my chance! I have now released the new Python 3.0 version of PyEphem under the actual package name ephem and will continue to maintain the 2.x version of the package separately under the old pyephem name. Obviously, this solution only works because of my project's unique circumstances; I have no idea what other projects should do who also want to come out with their new Python 3.0 versions this weekend.

Fans of PyEphem should rest assured that when I develop new features, I will be adding them to both versions of PyEphem for probably at least the next decade. I have absolutely no intention of abandoning or slowing development of the Python 2.x version of the library; I simply wanted the library available to users of the new platform. As always, feel free to email me with bugs, suggestions, and new features — and, enjoy using PyEphem and Python!

The answer is that, while the necessary changes were surprisingly easy, they took lots of time to figure out because I did not find them documented in any one place. I offer the following notes to assist any other adventurers who want to experiment with porting their extension modules to 3.0. These notes might also suggest useful additions to the official documentation.

But, first, I need to issue three cautions. To develop under 3.0, you may have to forego several Python tools that you probably thought you could no longer do without. The world of 3.0 is a windswept and icy landscape from which the glaciers have just receded, and you will find the stone tools rather primitive when compared to the comforts of civilization that you enjoy under Python 2. To wit:

• I cannot find virtualenv for 3.0, which is a disaster. This means that you have to create a separate Python 3.0 install, built with a different --prefix option to ./configure, for each development environment you want to create on your box.
• I cannot find a version of the setuptools available for 3.0. This means limiting your setup.py instructions to the primitive vocabulary of the distutils package. For example, I find myself unable to run the PyEphem test suite at this late hour because I have been running it for so long with:

  $ python setup.py test

  that I am not sure how to get it running otherwise.
• Should you succeed in porting your extension module, it is not at all clear how to distribute it. I had expected either a new PyPI to spring into being — since every package will need an entirely different version under 3.0 — or for a sophisticated scheme to appear for registering one pyephem.tar.gz as the Python 2 version and another pyephem.tar.gz for 3.0. But while the most recent version of your package can mark itself as 2-compatible or 3-compatible (or both) using classifiers, there is no way to have two “most recent” versions of a package. Are we supposed to start distributing a single tar.gz that includes the source code for both Python series, and that selects the right code by detecting the interpreter version at the top of the setup.py file?

So if you make the effort to port your code right now, you might find that the shiny new version of your module is all dressed up, but has no place to go. If you experiment with the following steps, though, you will at least be ready when an official distribution channel does appear for releasing your package into the wilds of 3.0.

Four Steps To 3.0

Yes, four steps were all that were necessary to convert my quite complex extension module to Python 3.0!

1. Use PyModule_Create(). The old mechanism that I had been using to initialize my extension module, the rather clunkily-named Py_InitModule3(), happily does not even exist in the Python 3.0 header files. Instead, call the PyModule_Create() function which you can find described in the Module Initialization section of the Extending and Embedding document. And be sure to keep its return value: unlike in older Pythons, you now have to return the module object it creates as the return value from your module initialization function.

2. Adjust all Python object headers. Each type object in my code started with a macro to set up the common fields that all Python objects share. This was then followed by the ob_size field, which in my code always is always zero, and then the type name:

   /* For Python 2 */
   
   static PyTypeObject BinaryStarType = {
        PyObject_HEAD_INIT(NULL)
        0,                   /* ob_size */
        "ephem.BinaryStar",  /* tp_name */
        ...
   

   Though the Python 3.0 documentation still shows this as the way to create types, this technique will now completely fail. (The bug indicating that the documentation gets this wrong has, as its most recent comment, the helpful note “I'm lowering the priority so it doesn't block the release.”) Anyway, the solution is simple: the first two lines in the struct shown above simply have to be combined into a single macro call:

   /* For Python 3.0 */
   
   static PyTypeObject BinaryStarType = {
        PyVarObject_HEAD_INIT(NULL, 0)
        "ephem.BinaryStar",  /* tp_name */
        ...
   

   With this change, my objects are now operating fine.

3. Use plain “static.” PyEphem inherits code from an era when it was popular to use variations on the static keyword so that Python could work around problems with various troublesome C compilers. This filled my code with staticforward declarations like the one you can see at the top of the old Python 2.2 Defining New Types page. It turns out that for well-behaved compilers these were always simply synonyms for the static keyword, which is what you must replace them with when porting your code to 3.0.
4. Upgrade to Unicode. Python 3.0 makes a clean and sharp distinction between strings, which are sequences of Unicode characters, and byte arrays, which can represent anything. To reflect this sea change down at the C level, the decision was made to eliminate everything from the C API whose name started with PyString. The obvious compiler errors that result from this provide a clear signal to programmers that they have to decide, everywhere that they had been using an old-style Python string, whether they should now represent that data with the PyUnicode or the PyBytes type. This was a brilliant decision; the transition could have been a nightmare had it remained possible for old code to compile without having been properly converted!

   When migrating the PyEphem code base, I found that most of the Unicode transition was very easy. Everywhere that my code handled or created a string object, I simply changed the prefix of the function to PyUnicode and everything worked:

    PyString_Check      ... becomes ... PyUnicode_Check
    PyString_FromString ... becomes ... PyUnicode_FromString
    PyString_FromFormat ... becomes ... PyUnicode_FromFormat
   

   Well, okay, the trick does not work everywhere; this one was harder to guess:

    PyString_Size ... becomes ... PyUnicode_GET_SIZE
   

   The situations that require real thought are the places where my code needs to convert a Python string into the sort of simple ASCII character array that the underlying C library can absorb. At the moment, my code is leaning heavily on a pitiful PyUnicode_AsString() routine that I wrote just to get things working; in the morning I will have to look into doing this more correctly, including catching the error if a fancy Unicode character is present that cannot properly be converted.

Overall, I am very impressed with how quickly I was able to get my extension module compiling and running under Python 3.0. The procedure was simple — I just tried, over and over again, to build the module with:

$ python3.0 setup.py build

and then tackled the compiler errors that resulted. Once every last warning had been addressed, the module started up and operated without a single further complaint. This calls for celebration! I'm going to bed.

Because users might become confused now that PyEphem is spread across three web sites — the home page is here at rhodesmill.org, releases are posted over at the Python Package Index, and, again, the development project is now hosted at Launchpad — I have completely redesigned the PyEphem home page with the goal of making the three-site distinction clear, coherent, and easy to navigate. The new home page and documentation are generated by the wonderful Sphinx documentation engine, and I am still thrilled about how pretty my code samples look (check out the one on the PyEphem home page!) now that Sphinx is coloring them in with the renowned Pygments system.

I have simultaneously released a new version of PyEphem that includes the new Sphinx-based documentation, along with several important fixes to the software itself. From now on, rather than cluttering my own blog with every minor version of PyEphem that I might release, fans of the software should visit its News and announcements page on Launchpad and subscribe themselves to its Atom/RSS feed. You will still see the project mentioned here whenever a technical or scientific issue becomes interesting enough for me to write about; but the audience of astronomers and hobbyists who just need to know when the next version is released should not have to wade through my blog to do so!

My users have already begun transferring their questions and problems to Lauchpad, and I look forward to offering much greater accountability through a fully public development process.

error: Python was built with version 7.1 of Visual Studio, and extensions need to be built with the same version of the compiler, but it isn't installed.

• I installed Python for Windows.
• I installed the setuptools package using its Windows .exe installer (scroll down the page to find it).
• I installed the MinGW compiler.
• I installed LibArchive for Windows.
• I added to my Windows Path:

  C:\Python25;C:\MingW\bin;C:\Program Files\GnuWin32\bin

• I downloaded the PyEphem source archive.
• I brought up a command window and ran:

  C:\dev> bsdtar -xf pyephem-3.7.2.1.tar.gz
  C:\dev> cd pyephem-3.7.2.1
  C:\dev\pyephem-3.7.2.1> python setup.py build --compiler=mingw32
  C:\dev\pyephem-3.7.2.1> python setup.py bdist_egg
  

• I uploaded the egg in the dist directory to the Cheese Shop.