[PEAK] Re: setuptools test command generates exception after interpreter shutdown

[P.J. Eby]

At 11:00 PM 5/3/2010 -0400, Nikolaus Rath wrote:
>Working:
>
>$ python -S -c 'import sys; print sys.path'
>['', '/usr/lib/python2.6/', '/usr/lib/python2.6/plat-linux2', 
>'/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', 
>'/usr/lib/python2.6/lib-dynload'
>
>Non-working:
>
>$ python -S -c 'import sys; print sys.path'
>['', '/usr/lib/python2.6/', '/usr/lib/python2.6/plat-linux2', 
>'/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', 
>'/usr/lib/python2.6/lib-dynload']

No, print this from the test case code -- I want the path at the time 
of the test running.


>   /home/nikratio/tmp/tests/t3_fs_api.py(12)test_getattr_root()
>-> log.info('Woops!')
>   /usr/lib/python2.6/logging/__init__.py(1030)info()
>-> self._log(INFO, msg, args, **kwargs)
>   /usr/lib/python2.6/logging/__init__.py(1142)_log()
>-> record = self.makeRecord(self.name, level, fn, lno, msg, args, 
>exc_info, func, extra)
>   /usr/lib/python2.6/logging/__init__.py(1117)makeRecord()
>-> rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
>   /usr/lib/python2.6/logging/__init__.py(272)__init__()
>-> from multiprocessing import current_process
> > /usr/lib/python2.6/multiprocessing/__init__.py(46)<module>()
>-> __version__ = '0.70a1'
>
>
>I think that it really is the logging module.

Yep.  This was changed in the 2.6 maintenance branch because it was 
causing crashes under Google App Engine:

http://svn.python.org/view/python/branches/release26-maint/Lib/logging/__init__.py?r1=75229&r2=75425

So, that's probably the difference between the two machines - i.e., 
the dot release of Python.


> >>  And only with setuptools it produces exceptions.
> >
> > You're testing the wrong thing.  What you want to be testing is the
> > difference between the environments where it works with setuptools,
> > and where it doesn't work with setuptools.  All that you've shown is
> > that setuptools is *related* to the problem; there's still the
> > possibility of a plugin interfering.
>
>Well, the way it looks to me is that in this constellation the use of
>setuptools seems to produce bogus error messages in the multiprocessing
>module. Why is it so important what loads the multiprocessing module?
>Can't we just assume that I send the test case with an additional
>'import multiprocessing' line?

It was relevant because it might have been due to e.g. a plugin not 
properly initializing or tearing down multiprocessing, or leaving 
something around to be garbage collected later.

Now, to narrow it down further, we'll have to remove stuff from the 
setuptools code execution path to find out what part of setuptools is 
relevant.  I would start with removing code from the test command's 
'run()' method, especially the running of other commands such as 
egg_info, and the saving and restoring of sys.path and 
sys.modules.  My guess at this point is that it may be the 
sys.modules restoration that triggers it.

Actually, there's a quick way to test that: add 'import 
multiprocessing, logging' to your setup.py (not the test code).  If 
that makes it work, then the sys.modules restoration is probably the 
culprit.  If it still doesn't work, then it's almost certainly a 
multiprocessing/logging bug.