Ubuntu Python: raise an exception, import 190 modules
February 25th, 2010
Imagine my surprise, while writing my first PEP 302 compliant import hook this afternoon, to carefully watch “sys.modules” for the results of my import but see it suddenly grow by nearly two hundred modules! What on earth had I done wrong? Some quick experiments revealed that my only sin was having the temerity to raise an exception. Let's try raising a simple NameError:
>>> import sys >>> len(sys.modules) 35 >>> foo ... NameError: name 'foo' is not defined >>> len(sys.modules) 225
That's 190 extra modules — merely importing them takes around 60 ms on my laptop! Where are they all coming from? And how could an exception cause so many imports, including such illustrious modules as “email”, “mimetools”, and “xml”?
After reading Ubuntu's “sitecustomize.py” file and all of its consequences, the situation became clear. Their apport crash-reporting subsystem instruments Python with an exception hook that, when invoked, discovers that my system says “enabled=0” in my “/etc/default/apport” file and so it undertakes no special crash logging. But, on the way to loading the routine that performs this simple check, it performs two quite flagrant and unnecessary imports, pulling in both “apt” (that brings with it 83 packages) and “apport” (an additional 107 packages).
The solution? I have removed the “python-apport” package, along with the “ubuntuone-client” suite that depends on it. After the uninstall, exceptions are — wonderfully enough — not causing a single import of a new module! Now, finally, I can continue writing my import hook in peace.
Posted:
Thursday, February 25th, 2010 at 11:16 pm
Categories: Computing, Python
You can leave a response, or trackback from your own site.
Subscribe
Wow, that is crazy! do you know f that has been reported as a bug ’cause it should.
February 26th, 2010 at 2:50 amThanks for the tip :)
Would be cool to report a bug about this… seems pretty bad to waste this memory for a turned off feature.
February 26th, 2010 at 3:19 amI just did this on my Ubuntu box and got the same result. That is really ridiculous. That’s a new one.
February 26th, 2010 at 3:56 amHave you submitted a bug report to Launchpad?
February 26th, 2010 at 4:22 amOther inefficiencies I noticed in relation to auto bug reporting:
redhat slow sigquit
ubuntu slow sigquit
February 26th, 2010 at 4:25 amDid you fill a bug report on launchpad?
I can fill it for you if you don’t.
Great work, btw
February 26th, 2010 at 4:52 amYou also filed a bug report about this, so the developers can fix this in the next version, right?
February 26th, 2010 at 5:08 amI hope you submitted a bugreport for ubuntu? Seems sensible to first check and then import.
February 26th, 2010 at 5:30 amHedious! Please submit a bug report on Launchpad!
February 26th, 2010 at 5:44 amI’ve raised a bug on launchpad with a patch to move the check to avoid unnecessary imports: https://bugs.edge.launchpad.net/ubuntu/+source/apport/+bug/528355
February 26th, 2010 at 5:54 amSomeone took the time to report this in launchpad:
https://bugs.launchpad.net/ubuntu/+source/apport/+bug/528355
Next time you find something like this, it would be of great benefit if you could report it too :)
February 26th, 2010 at 6:10 amDid you create a bug report? If not then please do it now!
February 26th, 2010 at 6:53 amWell, that will teach me to not blog right before bedtime! Or, it will teach me to leave blog comments unmoderated for the first few hours if I do. :-) Because none of you could see each other’s comments, you took time commenting when you might otherwise have spent those two minutes some other way. Sorry about that!
No, I had not created a bug report; I was so happy at nearly midnight to have found the problem and written it up that I went to bed without even considering it.
And, Stuart — Thank you for making the bug report! We will all be watching eagerly to see if the version of the package in Ubuntu 10.04 lacks this problem.
February 26th, 2010 at 7:22 amDid you try setting up a virtualenv with no site packages? That might help, no?
February 26th, 2010 at 7:38 amYou should look into creating a clean environment with virtualenv with the –no-site-packages flag. That way you can eliminate any side effects from packages that are installed to your main Python installation.
February 26th, 2010 at 7:51 amBTW, I tried it in both mere python and dreampie shells…
February 26th, 2010 at 8:05 amYepp, vanilla python shell cats just as described, while dreampile is a bit different:
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "copyright", "credits" or "license()" for more information.
DreamPie 1.0
>>> import sys
>>> len(sys.modules)
0: 112
>>> import foo
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
import foo
ImportError: No module named foo
>>> len(sys.modules)
1: 112
>>>
Upon reading this I checked out my
February 26th, 2010 at 8:08 amsitecustomize.pyand found out that I’ve solved this problem ages ago by just commenting all of it out. This is fine since it’s actually symlinked to/etc/pythonX.Y/sitecustomize.pywhich can safely be edited by sysadmins (and will be preserved).Jordan, Dave — But I like my Ubuntu site packages! Well, normally, when they’re not importing things crazily behind my back. :-)
The Ubuntu project has compiled things like NumPy, PyCrypto, lxml, psycopg2, and — of course — the famously nefarious Python Imaging Library, so that I don’t have to. That’s their job. I stopped worrying about “-dev” packages and “./configure” scripts ages ago, and hopefully will keep doing so if we can keep the Ubuntu Python install livable.
Floris — Thanks for the tip! I had no idea that “sitecustomize.py” is an admin-editable /etc file! It is so nice to know, in the face of a problem like this, that even here the Debian philosophy has won: they, apparently, did leave us the ancient recourse of editing a config file to opt out of their settings. I am impressed.
February 26th, 2010 at 8:31 amI am running LinuxMint Helena, based on Ubuntu 9.10.
It doesn’t import any modules on an exception.
February 26th, 2010 at 10:12 amBrandon,
and — of course — the famously nefarious Python Imaging Library
I just started using PIL. Why is it “famously nefarious”? Is there something I need to be careful about?
February 26th, 2010 at 11:15 am[...] 原文地址： http://rhodesmill.org/brandon/2010/ubuntu-exception-190-modules/ [...]
February 26th, 2010 at 11:34 amI don’t know if this is actually a big problem. Are you raising exceptions on high-throughput scripts regularly? Try…Except doesn’t have this issue so it’s only when an actual exception is raised that these modules are imported. Usually scripts that run under high load don’t have a comparatively high number of exceptions :-)
>>> import sys
February 26th, 2010 at 12:00 pm>>> try:
… foo
… except:
… pass
…
>>> len(sys.modules)
35
>>>
Adam — I agree that this is not a huge problem. Not only do caught exceptions fail to trigger the hook, but the cost of the imports is only borne by the first exception in a given session anyway; all subsequent exceptions will find the modules already loaded. The only purpose of my blog post was to point out a curiosity that made it difficult to play with import hooks in the presence of exceptions, that’s all.
David — The PIL is nefarious in the sense that it is very difficult to compile with all of its features, since it needs so many “dev” packages to be installed in order to find the headers and libraries that it needs. The Plone project, which needs the PIL, always found its compilation to be one of the greatest pain points in getting Plone installed and running.
February 26th, 2010 at 12:12 pmPIL is nefarious [...] very difficult to compile
Thanks! I was worried there was some sort of gotcha deep in using PIL.
Fedora, Ubuntu had PIL in their package repositories but I did have to build from source on OSX. Already built Gnuplot and ImageMagick from source so didn’t notice any dependency problems when I built PIL.
February 26th, 2010 at 12:26 pmI had some memory leak issues with PIL a while back.
I’m now using ImageMagick with much less issues, albeit with ugly SubProcess calls.
I heard GraphicsMagick was in a better state than IM. No issues with IM so far.
February 26th, 2010 at 1:38 pmYet another case of package management screwing up the packages they manage.
Of course all the Ubuntu fans here say “report a bug in Launchpad!” instead of “gee sorry maybe we shouldn’t screw with software we don’t understand!”
February 26th, 2010 at 3:55 pminstead of mumbling about a bug report, i’ll say Nice One! :)
February 26th, 2010 at 4:37 pmA patch of 57452 lines is applied to python shipped by ubuntu:
http://archive.ubuntu.com/ubuntu/pool/main/p/python2.6/python2.6_2.6.4-6ubuntu1.diff.gz
To do python development you probably better build yourself a clean python :)
February 27th, 2010 at 8:44 amHey man, thanks for all your help at PyCon! Especially with regards to odd environments for WebFaction.
PIL has been a thorn in my side for some time now as far as installation goes. Dang, but I am addicted to its API wonderfullness, but installing it… must it be so painful?
On OSX 10.6 Snow Leopard, it is especially painful (except for recently but that could be because of my custom envrionment).
Folks, I recommend for all devs that with Ubuntu or OSX that you do NOT use your system python for serious work. Chris reccently told me he had no troubles with the system python, but I required 32 bit cooperation in things like PIL and Pyglet (for ctypes bindings).
System python and MacPorts Python were NOT my friends.
Though hast been warned :)
February 27th, 2010 at 6:23 pm[...] • История о том, как Python был “причесан” в дебрях Ubuntu и что из этого вышло. [...]
March 1st, 2010 at 1:30 amAn update: And less than a week after my blog post, it appears that Ubuntu has now fixed the problem:
http://bazaar.launchpad.net/~apport-hackers/apport/trunk/revision/1714
Whatever one may think of distributions shipping with alterations to the system Python — and I am not yet convinced that it ruins the system Python, though I must admit that I had no idea how many changes Debian and Ubuntu made until Troll pointed it out — I think it reflects very well on their project that Ubuntu made the change quickly and without a single argument.
March 2nd, 2010 at 7:57 am[...] there’s a fatal flaw in the original blog post to which limi refers. Yes, on the desktop, Ubuntu imports 190 packages when an exception is raised. [...]
March 2nd, 2010 at 8:54 amOh, please, this is the global exception handler. If your program has let an exception propagate to the global exception handler, the program will exit anyway, so what do you care?
March 2nd, 2010 at 12:31 pm>>> try:
... foo
... except NameError:
... pass
...
>>> import sys
>>> len(sys.modules)
34
>>>
Nona — I “cared” because, as I stated clearly in the post, I happened to be writing and debugging a new import hook at the Python prompt. When I made mistakes along the way, my import hook would stumble and raise an exception. Surely I am allowed some natural curiosity about the fact that, when operating properly, my hook would import one or two modules, but when it threw an exception it somehow managed to import more than a hundred? :-)
March 2nd, 2010 at 1:33 pm[...] Let's Discuss the Matter Further ? Ubuntu Python: raise an exception, import 190 modules (tags: python ubuntu) [...]
March 14th, 2010 at 7:03 am