[03:40:34] ** bitmonk has joined us [03:42:56] ** bitmonk has left IRC (Client Quit) [05:28:33] [connected at Fri Dec 21 05:28:33 2007] [05:28:33] <> *** Looking up your hostname... [05:28:33] <> *** Checking ident [05:28:33] <> *** No identd (auth) response [05:28:33] <> *** Found your hostname [05:28:33] <> *** Your host is clarke.freenode.net[clarke.freenode.net/6667], running version hyperion-1.0.2b [05:28:35] [I have joined #peak] [05:28:35] ** clarke.freenode.net set the topic to does anyone remember the real topic ? [12:22:09] zookoaway is now known as zooko [15:05:25] ** bitmonk has joined us [15:27:32] ** bitmonk has left IRC () [15:28:10] ** bitmonk has joined us [15:34:31] ** BlueAidan_work has joined us [15:45:51] ** BlueAidan_work has left IRC ("Leaving.") [15:54:33] ** pje has joined us [16:03:45] pje is now known as pje_mtg [16:43:12] I have a question about setuptools. I've attempted to configure my setup.py to bootstrap setuptools from a local file: http://allmydata.org/trac/tahoe/browser/setup.py#L36 -- but for some reason setuptools writes "Downloading file:misc\dependencies\setuptools-0.6c7-py2.5.egg" and then writes "Downloading http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c7-py2.5.egg". -- [16:43:12] http://allmydata.org/buildbot/builders/windows/builds/54/steps/compile/logs/stdio [16:43:12] [16:43:24] Why does it attempt to download the same file from pypi.python.org that it just examined locally? [16:43:37] I'll be back in a few minutes -- time to pick up my son from First Grade. [16:44:09] (It is snowing and my wife is afraid to drive.) [17:24:08] pje_mtg is now known as pje [17:24:51] back [17:25:22] Run it with -v [17:25:34] "setup.py -v build_ext -i" [17:25:41] or better yet, -vv [17:26:18] pje: will do. [17:27:00] offhand, though, it would appear that it's a different setup.py using setuptools that's causing the download [17:27:18] Notice the different "File setup.py" vs "File ./setup.py" [17:28:30] Oh. [17:28:39] Hopefully -vv will tell me more. [17:28:58] Oh, I'll bet it is something that tahoe setup_requires. [17:29:16] like setuptools_darcs, perhaps? [17:29:50] Yep. [17:30:05] I just reproduced it on my local box by removing setuptools_darcs egg from the cwd. [17:30:20] Yeah, the traceback is consistent with installing a setup_requires argument [17:30:29] Hm... Does this mean that things which are setup_require'd can be satisfied merely by including a .egg locally. Nice. [17:30:36] Yes [17:30:47] Now if only I didn't have to specify a python version number in the egg, then I could have just one egg. [17:31:16] I've already tried satisfying setup_requires by bundling a source .tar.gz -- no dice. [17:32:21] You can specify the filenames in the 'dependency_links' keyword to setup() [17:32:34] and then it will unpack and build the source files [17:32:52] Now, as to why setuptools_darcs needs to download and install setuptools a second time, I don't know. [17:33:03] That is an interesting question. [17:33:28] It should already be imported and on sys.path [17:33:31] We already use dependency_links to source tarballs, but I think that doesn't work for setup_requires, and definitely not for setuptools itself. [17:33:35] Although I'll double-check that. [17:33:43] It *should* work for setup_requires. [17:33:54] Okay, I'll experiment. [17:34:04] There is lots of extra code in there to make it work, anyway. :) [17:34:25] http://allmydata.org/trac/tahoe/browser/misc/dependencies [17:34:32] This is currently what we ship, in terms of bundled packages. [17:34:34] (e.g. that ensures dependency_links is set up, and that it gets merged with config file settings for --find-links) [17:34:47] http://allmydata.org/trac/tahoe/browser/calcdeps.py [17:34:54] This is where we declare that we install_require each of those. [17:35:32] Well, putting them in dependency_links should allow them to be installed, irrespective of how they get require'd [17:35:59] http://allmydata.org/trac/tahoe/browser/setup.py [17:36:02] Here's our setup.py. [17:36:13] * zooko reads it to see how the result of calcdeps gets added to dependency_links. [17:36:24] Looks right. [17:36:54] Btw, I notice that you have specified "zip_safe=False" on some of your files, but you should not be using this to control installation, when you can just set your config files to never install any eggs zipped. [17:37:16] That way, people who want zipped eggs will not be forced to force zipping of your eggs [17:37:24] They can override it and force zipping.\ [17:37:38] Yes, but that's not nice to other people. [17:37:51] It's always safe for them to force *un*zipping, but not safe to force zipping [17:38:02] Oh, because somethings really can't be zipped. [17:38:03] Hm. [17:38:05] So you are making it harder for people who want/need things zipped. [17:38:06] Right [17:38:33] that's why it's called "zip_*safe*" :) [17:38:52] The default is to zip things that are zip safe, but you can safely specify that you want to unzip everything, instead. :) [17:39:15] I see. [17:39:22] Anyway, I don't see any obvious reason why setuptools_darcs is looking for another copy of setuptools [17:39:28] So why does setuptools_darcs downlo.. [17:39:34] That was what I was just wondering, too. [17:39:35] perhaps you can do a bit more digging [17:39:40] on your end [17:39:49] meanwhile, I need some food; just got out of a long meeting [17:39:55] Well, [17:40:00] Okay, talk to you later. [17:40:11] I just saw that setuptools_darcs uses the ez_setup trick at the beginning of its setup.py. [17:40:19] It says use_setuptools(). [17:40:40] yes, but that version should already be available [17:40:45] So when tahoe's setup.py setup_requires setuptools_darcs, then setuptools_darcs's setup.py does use_setuptools(). [17:40:48] Okay, I'll dig into that. [17:40:49] so it shouldn't download again [17:41:02] I'm running a patched ez_setup.py, so it might be my bug. [17:41:07] Thanks for the help. [17:42:05] Ah, yes, I think this is a bug caused by my patch. [17:42:44] My patch (as we discussed on the mailing list) checks the version number of setuptools without importing it, so that when someone runs ./setup.py then setup.py uses ez_setup, then ez_setup can upgrade the version of setuptools currently installed to a newer version. [17:43:13] You pointed out on the mailing list that this does not help for the case that someone has executed easy_install allmydata-tahoe, in which case it is too late for ez_setup to upgrade the installed version of setuptools. [17:43:38] So the way ez_setup checks the version number of the installed setuptools, in my ez_setup.py, is os.popen3("%s -c \"import setuptools;print setuptools.__version__\"" % (sys.executable,) [17:44:00] Which does not detect the setuptools .egg in the cwd that the tahoe setup.py just installed. [17:44:27] This is the kind of reason why I haven't submitted my patch yet -- I wanted to do one full release cycle using it first. [17:45:42] So how does setup.py find that egg in the CWD, I wonder, to know that its setup_requires has been satisfied... [17:45:43] * zooko digs. [17:52:52] Hm. setuptools source code defies my text-oriented programming. [17:53:12] But I'll bet that if I do something like PYTHONPATH=$PYTHONPATH:*.egg then it'll work... [17:53:48] At least on Unix. :-/ Hm. [17:59:40] There: os.popen3("%s -c \"import os,sys; sys.path.extend([x for x in os.listdir('.') if x.endswith('.egg')] ; import setuptools;print setuptools.__version__\"" % (sys.executable,) [18:00:28] Er, just do this: [18:00:37] if 'pkg_resources' in sys.modules: [18:00:53] import pkg_resources # and check the version the setuptools way [18:00:54] else: [18:01:05] # do whatever random crazy checks you want :) [18:01:17] That's what you already suggested on the mailing list. [18:01:25] Yes, and are you doing that? [18:01:45] I did for a while. [18:01:56] But the import of pkg_resources prevents me from upgrading setuptools later. [18:02:00] If I recall correctly. [18:02:11] If 'pkg_resources' is in sys.modules, it's *already* too late. [18:02:28] because you're being run by easy_install in that case [18:02:53] Ok. So at some point, per our mailing list discussions, I was using the code that you suggested. [18:02:56] * zooko checks the list archives. [18:03:15] Right, so go back to it, that will fix *this* problem [18:03:19] I'm not sure precisely why I stopped, other than that if x: do_it_the_nice_way() else: do_it_the_random_crazy_way() seems worse to me than just [18:03:31] do_it_the_random_crazy_way(), as long as the random crazy way works. [18:03:46] I see. [18:04:11] That would fix this problem, and then in the case that pkg_resources was not already imported, then it would do it the random crazy way. [18:04:23] However, I think that my current random crazy way will also fix this problem. [18:04:29] which is why I said to do it that way in the first place [18:05:08] So if I did it that way, then in the case that pkg_resources was not *already* in sys.modules, but the setuptools egg was already in the CWD, then my ez_setup.py would decide to go ahead and install a new setuptools, right? [18:15:42] yes [18:15:47] (sorry, was on phone) [18:16:56] No problem. [18:16:59] I appreciate your help. [18:17:58] I can, however, tell you an *new* crazy way to check the version, that I might use myself at some later time. :) [18:17:59] I'm trying to figure out how setup.py (which uses ez_setup.py) finds and loads the ./setuptools*.egg. [18:18:06] Okay, let's hear it! [18:18:31] temp = new.module('some_oddball_name') [18:18:37] temp.__path__ = sys.path[:] [18:18:49] sys.modules[temp.__name__] = temp [18:19:05] try: from some_oddball_name import pkg_resources [18:19:13] except ImportError: # not installed [18:19:36] else: # do lookup stuff, then delete sys.modules items [18:19:46] delete temp.__name__ from sys.modules? [18:19:54] where the module name startswith(temp.__name__) [18:20:21] because everything imported by pkg_resources is going to get duplicated under temp.__name__ [18:20:25] So I tried deleting 'pkg_resources' from sys.module before. [18:20:32] e.g oddball.os, oddball.sys, etc. [18:20:51] Yes, but this is different, because this will be a separate module object, and won't relate [18:21:08] Is the crazy thing that you have just suggested going to allow me to import setuptools-0.6a9, then decide that this isn't good enough and then install and start using setuptools-0.6c7 in the same Python interpreter? [18:21:09] it won't have been imported by anything *else* [18:21:19] Yes, it *should* [18:21:33] I have not tried it, but it should work, so I might use it myself later, if I can work out all the kinks. [18:21:52] .. except possibly if setuptools-0.6a9 did something too funny as a side-effect of being imported? [18:22:55] That's not the kink source I'm worried about [18:23:13] It's that not filtering sys.path before copying it to temp.__path__ means that various *other* modules will get duplicated [18:23:58] e.g. we might have oddball.os, oddball.re, oddball.StringIO [18:24:13] So... I'm still searching here. Is the ability of my setup.py to use a setuptools*.egg in the CWD implemented in ez_setup.py? [18:24:16] I guess it must be. [18:24:28] I don't know. [18:24:46] pje: add what's the potential source of kinks from those other oddball modules being duplicated? [18:24:59] dunno, that's why it's only potential :) [18:25:05] It just bothers me, is all. [18:25:14] You don't know how setup.py when using ez_setup.py manages to use a setuptools*.egg in the CWD? [18:25:30] From my inspection of source code, I wonder if it works only when that setuptool*.egg was just "downloaded" by ez_setup.py. [18:25:32] I do now - I looked at it. [18:25:45] I.e., if it is already there then it doesn't get inserted into sys.path(0,) and assigned to [18:25:49] It uses it if it's in the CWD [18:25:54] setuptools.bootstrap_install_from [18:26:06] See download_setuptools() [18:26:17] which checks to see if it's already in the target dir, and skips downloading it. [18:26:29] Oh I see -- download_setuptools() short-circuits the download. [18:26:34] Okay, thanks. [18:27:01] Oh, you know what, the "import it under an odd name" idea is unnecessary. [18:27:04] So earlier you were saying that I could use dependency-links to install dependencies, even setup_requires dependencies, from source tarballs, but it wouldn't be so easy to install from a setuptool ssource tarball. [18:27:19] If pkg_resources isn't in sys.modules to start with, then it should be safe to delete it after you import it! [18:27:25] I'd have to change ez_setup.py to detect and use the source tarballs. [18:27:35] pje: I think that I tried that. [18:27:37] Whereas if it *is* in sys.modules, it's already too late to do anything about it, no matter what. [18:27:45] I may have even submitted such a patch to distutils-sig. [18:27:56] Did you only delete it if it wasn't imported to start with? [18:29:24] http://mail.python.org/pipermail/distutils-sig/2007-October/008345.html [18:29:26] This was the one. [18:29:28] * zooko reads it. [18:29:50] This one always imports it and then always deletes it. [18:31:07] Right, which is why I didn't like it - that breaks the easy_install case [18:31:28] The one that I current have doesn't break the easy_install case and also works for the ./setup.py case. [18:32:21] That's fine if you want to do that, just understand that I'm not going to support (or include) a patch that invokes another process in order to find out what's happening in the current process. :) [18:32:22] So what do you think of the idea of Eggs which are not specific to a Python version number, because they contain only Python source? [18:32:37] Why not invoke another process? [18:32:39] I think I've already answered that on the mailing list - i.e., I don't. [18:32:49] Because it's not necessary. [18:33:02] Ok. [18:33:09] No matter, what you're not going to get around the fact that in the easy_install case, you're hosed. [18:33:33] The reason I was asking just now is that when I want to bundle dependent libs in order to make a self-contained collection, I could avoid -py2.3, -py2.4, -py2.5, -py2.6 files and have just one. [18:33:37] If you want to delete pkg_resources when it *hasn't* been imported, that's relatively safe, and doesn't add any new moving parts into the whole shebang. [18:33:42] I understand that my patch doesn't help in the easy_install case. [18:34:14] I don't yet understand why you object to invoking another process in order to detect the existing setuptools version without polluting the current Python interpreter. [18:34:50] Because you can find that out by importing and deleting and you're not dependent on what sys.executable is. [18:35:07] For example, effbot's exemaker sets sys.executable to itself -- which will totally not be what you want here. [18:35:16] Hm. [18:35:30] sys.executable is a minefield that you don't want to go near if you can avoid it. [18:35:44] I think that importing and deleting will prevent me from upgrading setuptools afterward, though. [18:35:49] Or have you thought of a way around that? [18:36:19] No, it won't prevent you. You just have to *only* import pkg_resources if it's not already imported, and delete it only if it wasn't already imported. [18:36:34] Nothing else will have a reference to it, if you delete your reference, and the one in sys.modules [18:36:54] So in the case that it hasn't already been imported, I import it, then do what -- pkg_resources.require('setuptools>=') ? [18:36:55] If you want to be absolutely sure, use pkg_resources.__dict__.clear() before deleting [18:36:56] Then delete it? [18:37:04] Yes. [18:37:18] Hm. I should have taken better notes about why I gave up on this approach last time... [18:37:21] I'll try that again. [18:37:54] And if the require() fails, go ahead and download the old way and proceed - this is only a change to the version check, nothing else. [18:39:04] But won't the call to require() import setuptools ? [18:39:32] Last time, I didn't do pkg_resources.require(), I did pkg_resources.parse_version(). [18:45:28] I don't understand why it is important to *not* import pkg_resources in the case that pkg_resources was already ther.e [19:04:37] By the way, I have a patch to Twisted to build twisted with setuptools. [19:04:41] It was pleasantly easy to do. [19:04:46] Unfortunately the Twisted people will not accept it as is. [19:04:51] So I have some more work to do. [19:30:07] zooko is now known as zookofamilytime [19:41:00] zookofamilytime, the require() does NOT import setuptools [19:41:39] and no, it's not important to not import pkg_resources if it's already there [19:42:00] it's only the not deleting it part that's important. [19:42:59] ** thinkeryzu has left IRC (Remote closed the connection) [20:39:18] ** thinkeryzu has joined us [22:00:41] zookofamilytime is now known as zooko [22:07:17] ** pje has left IRC ("Client exiting") [22:38:49] ** bitmonk has left IRC () [22:44:33] ** bitmonk has joined us [23:34:07] ** bitmonk has left IRC () [23:36:10] Folks: when using ez_setup to install (bootstrap) setuptools, I can pass a to_dir to tell it where to put the egg. [23:36:21] Does anyone know of a way to do likewise with setup_requires dependencies? [23:36:27] I'd rather not have them cluttering up the CWD.