[04:10:23] ** mike8 has joined us
[06:07:53] [connected at Fri Feb 15 06:07:53 2008]
[06:07:53] <> *** Looking up your hostname...
[06:07:53] <> *** Checking ident
[06:07:53] <> *** No identd (auth) response
[06:07:53] <> *** Found your hostname
[06:07:53] <> *** Your host is clarke.freenode.net[clarke.freenode.net/6667], running version hyperion-1.0.2b
[06:07:55] [I have joined #peak]
[06:07:55] ** clarke.freenode.net set the topic to does anyone remember the real topic ? - not really ...
[06:41:07] ** tav has left IRC (Read error: 110 (Connection timed out))
[09:50:41] ** spy has joined us
[10:10:38] ** spy has left us
[11:00:28] ** zooko has joined us
[11:00:52] <zooko> Howdy folks.  Does someone know how to use pkg_resources to import v0.2.4 of a package even if both 0.2.4 and 0.2.3 are installed?
[11:53:13] ** mike8 has left IRC ("Leaving")
[11:53:31] ** pje has joined us
[12:22:00] ** apoirier has joined us
[13:33:50] ** apoirier has left IRC ("KVIrc 3.2.6 Anomalies http://www.kvirc.net/")
[13:46:58] <coderanger> zooko: require('foo==0.2.4')
[14:39:16] <zooko> coderanger: Thanks.
[14:39:28] <zooko> I've learned that this would work only if foo 0.2.4 were installed with the -m option.
[14:39:44] <zooko> But I suspect that if I install foo 0.2.4 with the -m option then a simple "import foo" might not work at all.  :-(
[14:40:04] <zooko> So for the v0.8.0 of http://allmydata.org , I'm just going to require people to manually delete older incompatible versions of dependenices.
[14:42:03] <coderanger> uh, no
[14:42:22] <coderanger> Also, if you have deps, you should have them listed as such in setup.py
[14:42:46] <coderanger> you can always require things, as long as the metadata is intact
[14:42:59] <coderanger> if you install the pure module, not eggs, then you can't use things like require
[14:43:09] <coderanger> or any other dist-related functions
[14:43:31] <zooko> just  a sec..
[14:43:36] ** mike8 has joined us
[14:44:34] <zooko> So as per this file:
[14:45:26] <zooko> http://allmydata.org/trac/tahoe/browser/_auto_deps.py
[14:45:40] <zooko> We do install_requires  foolscap >= 0.2.4
[14:45:50] <zooko> Then we do pkg_resources.require("foolscap >= 0.2.4")
[14:46:01] <zooko> but, then it raises an exception
[14:46:12] <zooko> Because foolscap v0.2.3 is installed.
[14:46:20] <coderanger> this won't even remotely work
[14:46:24] <zooko> Even though both foolscap v0.2.3 and foolscap v0.2.4 are installed as eggs.
[14:46:41] <coderanger> first things first
[14:46:51] <coderanger> dist name and module name are unrelated
[14:47:01] <coderanger> second, __import__ doesn't work the way you think
[14:47:03] <zooko> This person has the same experience: http://entitycrisis.blogspot.com/2008/01/many-versions-of-one-python-package.html
[14:47:12] <zooko> What are you talking about dist name and module name?
[14:47:32] <coderanger> you can't just try to import the dist name
[14:48:04] <zooko> I don't understand.
[14:48:19] <coderanger> dist name != module name
[14:48:29] <zooko> What is a dist name?
[14:48:30] <coderanger> pkg_resources operates on distributions
[14:48:44] <coderanger> the "name" argument to setup()
[14:48:49] <zooko> A distribution is what I call a "package", right?
[14:49:02] <coderanger> No, a package is a collection of module
[14:49:47] <zooko> So, are you saying that one of the names in my list of requirements in that file is the wrong name?
[14:50:09] <coderanger> no, just that your second case for if the require() fails needs to be deleted
[14:50:13] <coderanger> as it won
[14:50:19] <coderanger> won't work in multiple ways
[14:50:24] <coderanger> second issue
[14:50:30] <zooko> You mean __import__(name)?
[14:50:33] <zooko> What's wrong with it?
[14:50:46] <coderanger> please check the version of foolscap is really "0.2.4"
[14:51:15] <coderanger> I don't feel like explaining exactly how the python importer works, suffice to say that __import__ doesn't work the same way as import
[14:51:21] <coderanger> its a much simpler mechanism
[14:51:45] <coderanger> it seems foolscap uses "0.2.4+" between releases
[14:51:48] <zooko> I'm pretty sure that what I have there is correct.
[14:51:54] <coderanger> Its not
[14:51:56] <zooko> I'll check the version number of foolscap.
[14:52:14] <zooko> I've tested it.  It does what I want.
[14:52:24] <coderanger> No, it doesn't
[14:52:32] <zooko> If you want me to believe you, you'll have to give me more information.
[14:52:44] <coderanger> fine
[14:52:58] <coderanger> for one, import doesn't process . in a module name the same way as import
[14:53:09] <coderanger> two __import__ return the module
[14:53:23] <coderanger> it doesn't add it to the global namespace like import
[14:53:42] <zooko> So actually, we don't need it to -- the purpose of that __import__ is to raise an exception if it can't import the thing.
[14:53:48] * zooko tries it on "zope.interface".
[14:54:15] <coderanger> on zope.interface you are just importing "zope"
[14:54:19] <zooko> It worked, but possibly only because zope.interface is installed in eggy form here.
[14:54:26] <zooko> Not in this case.
[14:54:31] <zooko> >>> __import__("zope.interface")<module 'zope' from '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/zope.interface-3.4.1-py2.5-macosx-10.3-fat.egg/zope/__init__.pyc'
[14:54:34] <zooko> I don't have zope installed.
[14:54:40] * zooko tries it with a non-eggy zope.interface.
[14:55:16] <coderanger> how 'bout that
[14:55:38] <coderanger> now, any further disbelief?
[14:55:44] <pje> zooko, the __import__('zope.interface') returned the 'zope' module
[14:55:57] <pje> but it does import 'zope.interface' (i.e., load it into sys.modules)
[14:56:01] <zooko> Right, once I uninstalled the eggy one and installed it without eggy, I got:
[14:56:06] <zooko> HACK wonwin-mcbrootles-computer:~/playground/ZopeInterface/zope.interface-3.4.0$ python -c 'x = __import__("zope.interface");print x'<module 'zope' (built-in)
[14:56:11] <zooko> Howdy, pje.
[14:56:24] <pje> coderanger is correct that require() doesn't necessarily take the same info as import
[14:56:37] <pje> however, it's certainly possible to have them be the same name.
[14:56:37] <coderanger> you can use __import__ to do import, but the __ version is a lower-level construct
[14:56:45] <zooko> Um, so actually it pretty much suffices for this purpose that __import__() is importing zope.
[14:56:57] <pje> yes
[14:57:05] <pje> now, regarding require() and conflicts between versions
[14:57:08] * zooko reads back to see what was coderanger's next point.
[14:57:12] <pje> you can have a default version (i.e., no -m)
[14:57:34] <pje> however, to use a non-default version, you have to use a setuptools-generated script.
[14:58:03] <zooko> pje: I don't understand what you men about a setuptools-generated script.
[14:58:10] <pje> that is, scripts defined in your project's setup.py can use non-default versions, as long as they were in the setup.py's install_requires
[14:58:16] <zooko> The goal is to import foolscap 0.2.4 if possible.
[14:58:35] <zooko> Defined by entry_points, "console scripts"?
[14:58:36] <pje> What's the script that's using this?
[14:59:01] <pje> Here's the thing: if you aren't using a setuptools-generated script, then you can ONLY access the default version.
[14:59:14] <zooko> Hm.
[14:59:22] <zooko> http://allmydata.org/trac/tahoe/browser/src/allmydata/scripts
[14:59:40] <zooko> But I would also like for it to be imported when it isn't being executed from a script at all.
[15:00:19] <pje> well, then foolscap 0.2.4 will have to be the default version.
[15:00:30] <zooko> Like this fellow -- http://allmydata.org/trac/tahoe/browser/src/allmydata/scripts -- and like David Reid,
[15:00:44] <pje> Or there has to be *no* default version (i.e. all installed w/eggs and -m)
[15:00:50] <zooko> I thought that pkg_resources("foolscap >= 0.2.4");import foolscap would suffice to import the needed version.
[15:00:58] <zooko> In fact, coderanger seemed to think that would work, too.
[15:01:01] <pje> it will -- unless there's a conflicting default version.
[15:01:13] <zooko> I wonder how the foolscap 0.2.3 egg got to be the default.
[15:01:23] <pje> by somebody easy_install-ing it.
[15:01:34] <pje> (or "setup.py install"ing)
[15:01:38] <coderanger> zooko: If you are calling from within setuptools land it all works. This is dependent on your code, not theirs.
[15:01:46] <zooko> It was my Makefile/setup.py that installed it.
[15:02:10] <pje> if you install something without -m, a default version gets set (for that thing and all its dependencies)
[15:02:34] <zooko> Why didn't the subsequent installation of foolscap 0.2.4 without the -m make that become the default version?
[15:02:41] <pje> it should have.
[15:03:10] <zooko> Hmmmm.
[15:03:27] <zooko> So the situation is that the previous release of allmydata.org -- v0.7.0 -- installed foolscap 0.2.3 into ./support/.
[15:03:42] <zooko> And now the imminent next release of allmydata.org -- v0.8.0 -- installed foolscap 0.2.4 into ./support.
[15:03:57] <zooko> And then I did pkg_resources.require("foolscap >= 0.2.4") and got a version conflict error.
[15:04:35] <pje> The only way to deal with that is to make sure that 0.2.4 is required by your install_requies
[15:04:38] <pje> requires, I mean
[15:04:41] <zooko> Nice to see you again, by the way.  You've been absent from IRC and distutils-sig for a while.
[15:04:53] <zooko> And only one blog post IIRC...
[15:05:07] <zooko> So I already do install_requires foolscap >= 0.2.4.
[15:05:17] <zooko> I think the problem is that the foolscap 0.2.3 .egg is on my PYTHONPATH.
[15:07:39] <zooko> pje: please see http://mail.python.org/pipermail/distutils-sig/2008-January/008724.html if you haven't already.
[15:15:11] <zooko> So here is a complete, minimal example: 
[15:15:12] <zooko> PYTHONPATH="/cygdrive/c/playground/allmydata/tahoe/support/lib/python2.5/site-packages/foolscap-0.2.3-py2.5.egg:/cygdrive/c/playground/allmydata/tahoe/support/lib/python2.5/site-packages/foolscap-0.2.4-py2.5.egg" python -c'import pkg_resources;pkg_resources.require("foolscap >= 0.2.4");import foolscap;print foolscap;print foolscap.__version__'
[15:15:12] <zooko> Traceback (most recent call last):
[15:15:12] <zooko>   File "<string>", line 1, in <module>
[15:15:14] <zooko>   File "/usr/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/pkg_resources.py", line 626, in require
[15:15:19] <zooko>   File "/usr/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/pkg_resources.py", line 528, in resolve
[15:15:21] <zooko> pkg_resources.VersionConflict: (foolscap 0.2.3 (/cygdrive/c/playground/allmydata/tahoe/support/lib/python2.5/site-packages/foolscap-0.2.3-py2.5.egg), Requirement.parse('foolscap>=0.2.4'))
[15:15:45] <zooko> This is what you expected since the foolscap 0.2.3 egg appears earlier on the PYTHONPATH?
[15:17:15] ** tav has joined us
[15:22:36] ** eikke has joined us
[15:22:47] <eikke> anyone around?
[15:23:29] <coderanger> indeed
[15:24:26] <eikke> can I do support questions on setuptools/pkg_resources here, or is this some other peak? :)
[15:37:10] <zooko> Ask.  I will do my best.
[15:37:21] <zooko> While I'm waiting for coderanger or pje to answer *my* support questions.  ;-)
[15:37:30] <eikke> heh :)
[15:37:52] <eikke> wel, I have an egg, which has an EGG-INFO dir and a package dir in its root (its zipped)
[15:38:05] <pje> zooko, your example fails as expected, based on what I already told you.
[15:38:25] <eikke> now I'd need to get the file/dirlist of everything in this package dir from within the packages __init__.py
[15:38:26] <pje> it is not a script, and 0.2.3 is the default version due to the PYTHONPATH.
[15:38:56] <zooko> pje: thanks.
[15:39:09] <eikke> I thought I could use pkg_resources.resource_listdir('packagename', ''), but that returns [], whilst pkg_resources.resource_isdir('packagename', 'somesubdir') does return true
[15:40:05] <eikke> I 'fixed' this by doing something like dist = pkg_resources.get_distribution(__name__); l = distribution.resource_listdir(__name__) in the __init__.py, which works, but it feels kinda hackish
[15:40:40] <eikke> (note this is not my package, it's one I got to repackage as a single egg)
[15:41:29] <pje> er, that last statement confuses me.
[15:41:46] <pje> resource_listdir() takes a *module* name, btw.
[15:41:57] <eikke> pje: the original code uses os.listdir
[15:42:16] <eikke> a module containing files? :-/
[15:42:31] <pje> it can be a module that's a package.  :)
[15:42:41] <pje> I just mean, it's not the name of a distribution or project.
[15:43:02] <pje> Meanwhile, this "repackage as a single egg" thing isn't making any sense to me.
[15:43:02] <eikke> thats not what i used :)
[15:43:31] <eikke> pje: original code uses os.listdir(__path__), so can't be used as a zipped egg
[15:44:02] <pje> So you are working on a patch to some project?
[15:44:08] <pje> Or forking your own version of it?
[15:44:28] <eikke> whether its a patch or a fork depends on my management ;)
[15:53:52] <pje> ok, so "resource_listdir(__name__, '')" *should* work in the __init__.py
[15:54:05] <pje> I don't really get why it's not.
[15:56:58] <pje> I just tried it on a bunch of projects on my PC, and using '' as the resource name works fine with resource_listdir
[16:06:29] ** coderanger has left IRC ()
[16:06:42] ** coderanger has joined us
[16:57:58] <eikke> pje: resource_listdir of the pkg_resources package itself of of some object?
[16:58:16] <eikke> must say this is on windows, althoug i guess that shouldnt matter
[16:58:18] <pje> of pkg_resources
[16:58:21] <pje> I'm on windows too
[16:58:34] <eikke> hmh, i'll try again
[17:07:51] <eikke> so, say I got an egg pysnmp-4.1.9a-py2.4.egg (zipped) in site-packages
[17:08:10] <eikke> I do require('pysnmp'), egg zip is found and added to sys.path
[17:08:39] <eikke> resource_listdir('pysnmp', '') returns []
[17:09:31] <eikke> in the egg zip, there's a folder 'pysnmp', which contains a folder 'v4' and a file __init__.py -> how to list these?
[18:37:39] <pje> hrm, it looks like there is a problem with zipped eggs, after all.
[18:37:51] <pje> the ones I was trying were directory eggs
[18:38:07] <eikke> ah, that just uses os.listdir :)
[18:44:02] ** mike8 has left IRC ("Ex-Chat")
[18:45:01] <pje> ok, found the bug...  it only affects the empty string as a resource name
[18:45:12] <pje> it's a trailing-/ problem, basically
[18:46:31] <pje> Yep, fixed.
[18:48:26] <eikke> cool :)
[18:53:52] <pje> okay, you can update with "easy_install setuptools==dev06" now.
[18:54:14] <pje> thanks for the bug report
[18:54:27] <eikke> no problem, thanks for fixing so quickly :)
[18:56:03] <eikke> I'm patching distutils/setuptools so I can cross-compile win32 eggs on a linux host using mingw, btw
[18:56:22] <eikke> proof-of-concept works, but code is a mess, rewriting
[20:36:21] ** pje has left IRC ("Client exiting")