[05:03:16] [connected at Wed Nov 1 05:03:16 2006] [05:03:16] <> *** Looking up your hostname... [05:03:16] <> *** Checking ident [05:03:16] <> *** Found your hostname [05:03:16] <> *** No identd (auth) response [05:03:16] <> *** Your host is sterling.freenode.net[freebsd.widexs.nl/6667], running version hyperion-1.0.2 [05:03:16] [I have joined #peak] [08:33:00] ** AVGVSTO has joined us [08:33:11] hi [08:36:19] ** AVGVSTO has left us [12:04:07] ** pje has joined us [17:13:33] ** pje has left IRC (Read error: 104 (Connection reset by peer)) [17:20:14] ** pje has joined us [19:06:07] ** exarkun has joined us [19:06:38] Mmm, not completely empty... :) [19:06:53] * exarkun is looking at namespace_packages. [19:07:15] ok [19:07:29] pje: Hi :) [19:07:30] "You see a maze of twisty passages, all different." [19:07:49] * exarkun gets out a pen and a piece of paper. [19:08:20] Specifically, I am trying out setuptools (for the first time from the developer perspective) for a package which includes a Twisted plugin [19:08:20] What is thy question, o seeker of truth and knowledge? :) [19:08:39] Perhaps this is impossible, though, given the way Twisted plugins work. [19:09:05] twisted.plugins is a namespace package, I guess. Or maybe not /exxaacctly/ a namespace package. [19:09:20] It doesn't register itself as one, at least. It does want modules from lots of different places to live in it, though. [19:09:49] yeah, the problem with that is that pkg_resources doesn't support having a namespace package as a child of a non-namespace package [19:10:07] Well, actually, I don't know if it's a *problem* [19:10:20] it's just not designed-for, if you know what I mean. [19:10:35] * exarkun nods [19:10:35] but I digress. What was your question? [19:12:09] Well... given what you've said so far, I guess my question is whether I should bother trying to make an egg (which doesn't sound doable at the moment), if there is a way setuptools can be useful without making an egg, or if I should just do a regular distutils thing? [19:13:02] Well, here's the thing... as long as twisted.__init__ and twisted.plugins.__init__ don't *do* anything [19:13:08] (and that includes importing anything) [19:13:12] They both do some things [19:13:27] Ah. That's potentially a problem. [19:13:41] But only for system-packaged scenarios [19:13:55] i.e., if you bdist_rpm twisted and your project, using a namespace package. [19:14:08] twisted.plugins.__init__ could probably be changed to not do anything, and maybe that'd even be a good idea. My main goal with this project was just to get a feel for setuptools and see how many of the use-cases I have it satisfies. [19:14:10] and setuptools. [19:14:24] * exarkun nods [19:14:43] Here's the thing... if you use setuptools to build different eggs that comprise a namespace package [19:14:49] like the plugin and Twisted [19:15:07] then, to be packaged using stuff like RPM etc., you can't have any conflicting files (like __init__.py) [19:15:15] Right, that makes sense [19:15:35] So, setuptools *omits* the __init__.py in that case, and gives each egg its own .pth file instead that fakes a dummy package in sys.modules. [19:16:06] It's a horrible kludge, but unfortunately the only way to do it in that scenario, unless you have some egg that's designated as "owning" the __init__.py, and there's no way to do that right now. [19:16:27] I was just about to ask about ownership :) That would be a good solution for my case, I think. [19:16:32] Planned future feature? [19:16:34] However... if you don't care about that one particular scenario, because you're using eggs, then the current stuff works. [19:16:51] because each egg has its own copy of __init__.py, and *all* of them get run. [19:16:57] Ahh, handy. [19:17:07] but in no particular order! [19:17:22] and of course sharing the same module object. [19:17:24] I think that implies the answer to my next question, which is whether this requires Twisted to be in an egg for the namespace package to work correctly. [19:17:31] (And it sounds like it does) [19:17:40] Not necessarily. [19:17:55] twisted.plugins could simply declare itself to be a namespace package. [19:18:32] One of those two things is necessary, though? [19:18:47] Hm. I'm trying to recall whether I auto-generate missing __init__.py files when installing in backward-compatibility mode... [19:19:07] Something has to declare the namespace package, as that's what triggers the __path__ munging. [19:19:28] If twisted.plugins just did it at runtime, its own __init__ would of course have already run. [19:19:51] * exarkun nods [19:20:02] And that's basically what the current contents of twisted/plugins/__init__.py are fow [19:20:08] s/fow/for/ [19:20:26] http://twistedmatrix.com/trac/browser/trunk/twisted/plugins/__init__.py [19:20:48] Twisted wouldn't need to be an egg or even depend on setuptools to do that, btw. [19:21:06] The only tricky bit is if you package a *plugin* as an RPM or some such. [19:21:30] I haven't done a lot with platform-specific packages, so I don't know much about the problems you run into in that area. [19:21:35] Hm... no, wait, that's not a problem either. [19:22:00] Because it just means that the *plugin's* twisted/plugins/__init__.py won't be installed. [19:22:07] And it should've been empty, anyway. [19:22:10] yep [19:22:30] which is a restriction that's already present in Twisted's plugin system [19:22:34] So, actually it sounds like everything would work just fine. [19:23:05] Except that Twisted itself should *not* declare twisted.plugins as a namespace package in its setup.py [19:23:11] (assuming you were using setuptools) [19:23:19] So maybe I should try changing it to do that and see what happens [19:23:34] Under setuptools 0.6, declared namespace packages get eagerly imported, which would not be good here. [19:23:39] Right now I just get "Distribution contains no modules or packages for namespace package 'twisted.plugins'" [19:24:13] Also, the twisted.__init__ and twisted.plugins.__init__ would not be installed in system-packaging mode. [19:24:34] Is that with declaring the namespace in your plugin, or in Twisted's setup? [19:25:00] That's with neither, I think, just a namespace_packages=['twisted.plugins'] in my package's setuptools-based setup.py [19:25:10] ok [19:25:30] Well, you still need to declare your packages or modules like in a regular setup script. [19:25:58] And you need to either put your module code in twisted/plugins/ under your setup, or do the package_directory dance. [19:26:00] Ah, well I have some other arguments to setup(), eg, packages=find_packages() [19:26:07] Aha, okay [19:26:11] * exarkun looks at package_directory docs [19:26:20] I think it's called that, it might be package_dirs [19:26:32] package_dir singular, looks like [19:26:44] ah, yes, that's it. [19:26:48] oh man this is familiar... not a pleasant data structure [19:27:02] You might need something like {'twisted.plugins':'.'} if you just want to dump everything in the current dir. [19:28:03] That will let you just put an __init__.py and your module. [19:28:15] But then find_packages() won't work; you'd just list 'twisted.plugins' and be done with it. [19:28:40] Unless of course there is stuff in the plugin that's under other package names, in which case forget everything I just said. [19:28:42] :) [19:29:26] There is, and I believe I see why that would be problematic :) [19:29:32] Another alternative to all of this, by the way, is to use entry points. [19:29:40] Because those don't require you to import from a particular place. [19:29:43] But now that you have reminded me of package_dir I wonder if I even want to be trying to use namespace_packages for this. [19:30:08] Yeah, namespace packages are really just for establishing a namespace. [19:30:17] For plugins, entry points are a heckuva lot more convenient. [19:30:42] I think they're next on my investigation list [19:30:57] Zope and PEAK use namespace packages in order to have separately distributed peak.foo.bar or zope.whiz.bang components. [19:31:12] A lot of the difficulty here is impedence mismatch between Twisted and setuptools, I think. [19:31:51] Well, "difficult" is relative. I gather that what you're currently doing can be made to work with setuptools. [19:32:03] Some parts might be inconvenient, depending on your source tree layout. [19:32:18] The simpler your source tree, the easier setuptools gets. [19:32:29] (Or the distutils for that matter) [19:32:38] Twisted's source tree is pretty simple. It's more or less exactly what the installed directory structure looks like. [19:32:52] And is that what your plugin looks like, too? [19:32:54] The only exception is third-party plugins, since they share twisted.plugins. [19:32:57] :) [19:33:22] Well, you know even the distutils supports quasi-namespace packaging in "installed directory structure" format. [19:33:26] As does setuptools. [19:33:35] Maybe I do, but possibly I don't. [19:33:40] you just have to make a twisted/plugins/ directory, or munge package_dirs. [19:33:42] (I'm still fighting with package_dir :P) [19:33:59] Maybe you should just make a twisted/plugins/ directory :) [19:34:14] i.e., just use "installed tree layout" [19:34:45] pje: Actually, that's what I have :) [19:34:53] so what's the problem, then? [19:35:07] How about I simplify this description... http://svn.twistedmatrix.com/cvs/sandbox/exarkun/merit/branches/setup/ [19:35:07] are you missing __init__.py's in twisted/ or twisted/plugins/? [19:35:15] pje: Probably mainly that I don't know distutils that well. [19:35:28] Hmm, yes, as a matter of fact. I added one in twisted/plugins/, but not twisted/. [19:35:40] need one there too [19:35:54] find_packages() doesn't know twisted's a package. [19:36:46] More impedence mismatch, though :/ Adding an __init__.py to either of those probably isn't a realistic solution, since it will probably break the Twisted plugin system for developers, and it did just break it for me, even for just running setup.py. [19:37:07] um, so do this: [19:37:07] merit.__version__ is defined based on an import from Twisted [19:37:15] Which no longer works once my fake twisted directory is a real Python package [19:37:22] packages = find_packages()+['twisted.plugins'] [19:37:27] Hmm, okay [19:37:29] and get rid of the __init__.py files. [19:37:54] With distutils, there's Always A Way(TM). :) [19:38:29] Wee, that does indeed package up all the files in a way which appears correct. [19:40:01] About halfway to where I am looking to be, I guess. I'll have to finish this later though. Thanks a lot for the hand-holding. [19:40:02] Now, to work correctly with this, twisted.plugins (the real one in Twisted) should declare the namespace, and it'll automatically get any egg-supplied paths added. [19:40:40] * exarkun nods [19:41:13] Hm... does your bdist_egg contain __init__.py files for twisted and twisted.plugins? [19:41:32] pje: nope [19:41:51] do you have namespace_packages=['twisted.plugins']? [19:42:00] nope :/ [19:42:38] although if I re-add that now, with my pkg_resources hack in twisted/plugins/__init__.py, the egg will still build [19:42:46] actually, that shouldn't matter... just listing twisted.plugins as a package should've made it end up with __init__.py files. [19:43:03] Hmm, it didn't have one in the .egg file [19:43:56] Which actually doesn't matter with Twisted's existing plugin system, but only because it does some extra weird stuff, which is probably incompatible with real namespace packages [19:44:59] Ah, I see the problem... [19:45:19] since there are no .py files in Twisted, the auto-__init__ generator decides not to explore further. :( [19:45:57] * pje updates his bug list for setuptools 0.6 [19:47:15] with this bug, ns pkg stuff isn't going to work for you. [19:47:34] I'll see if I can get it fixed this week. [19:48:44] That would be cool. This is just something I'm fiddling with, so I can put it down for a little while. Plenty of other stuff to do. :) [19:50:58] My suggestion, though, would be to look at entry points, since they allow you to easily import things without them having to be under twisted.plugins. [19:51:40] it can be as simple as "plugins = [ep.load() for ep in iter_entry_points('twisted.plugins')]" [19:51:49] I'll definitely do that, but the code which does the importing here is already written and part of Twisted. It's a good avenue of investigation for future Twisted developments, but not a useful short term solution. [19:51:55] to give you back modules or classes or functions or whatever. [19:52:44] do entry points have any notion of interfaces/protocols, or is the name the main/only way you differentiate things? [19:53:31] There's no builtin notion; it's assumed that you document the interface(s) required or supported, and of course you can enforce it in code. [19:54:02] the thing is, the entry point data is just that: data, so it doesn't require modules to be imported in order to query it. [19:54:15] * exarkun nods [19:54:19] Which basically puts it down to what you can represent in strings. [19:54:32] In order to query interfaces, you'd have to actually load the things. [19:54:48] Which you can do, if you want; it's just that the API doesn't load anything unless you ask for it. [19:55:03] * exarkun nods [19:57:03] the biggest difference between entry points and twisted's plugins seems to be how they're specified; entry points are explicit in setup.py (it seems?) whereas twisted plugins are discovered automatically in the filesystem. Is there something like find_packages(), but for entry points? [19:57:12] * pje reads the Twisted Plugins doc [19:57:16] I guess it wouldn't be hard to write, given a specific discovery policy, if it doesn't exist already. [19:57:40] Right. Of course, importing your own code during setup() is generally a bad idea, at least if you have any dependencies. [19:57:55] (Because they might not be installed yet when a user runs the setup.py) [19:58:02] * exarkun nods [19:58:15] But yeah, given a discovery policy it wouldn't be hard. [19:58:26] On the other hand, explicit registration is, well, explicit. :) [19:58:42] That does have its advantages. :) [19:59:12] But I just read the doc and I can go ahead and tell you now, that namespace packages aren't really going to work for twisted.plugins. [19:59:18] A long history of watching people do unpleasant things with distutils gives me a, perhaps irrational, wariness of doing anything in a setup.py file :) [19:59:34] Because setuptools has no way for you to find what *modules* are in the packages. [19:59:40] (without e.g. entry points) [19:59:48] Hmm, not sure I follow [20:00:05] Well, if you install a plugin in a zip file, how is Twisted going to know what modules are in there? [20:00:19] I thought there was some kind of name-based thing going on, but according to the docs you're searching directories. [20:00:42] So, unless you set zip_safe=False in setup.py for a plugin, it might be installed as a zipfile, and poof, there goes your plugin discovery. [20:01:12] Currently unimplemented, but the idea is that all the directory searching will be done through an abstract interface (which is implemented already) which can deal with regular filesystem directories or zipfiles full of python packages or what have you [20:01:29] Okay, well that would work. [20:01:47] The directory searching in the current system is primarily a developer convenience though. The recommended installation procedure is to copy your plugins into the twisted/plugins/ directory of the install of Twisted you're targetting. [20:01:57] That's not necessarily a hard requirement, it's mainly just an implementation detail of the current system. [20:02:32] The main thing that's desirable about it is just the performance characteristics. Loading plugins means scanning one directory, instead of crawling all over the place (unlike in the developer configuration). [20:03:02] Okay, here's what the idiomatic way to do what you're doing would be, in setuptools: [20:03:16] developers put their plugins wherever the heck they want (i.e. any module or package) [20:03:23] developers list entry point for plugin in setup.py [20:03:51] twisted.plugins does plugins = [ep.load() for ep in pkg_resources.iter_entry_points('twisted.plugins')] [20:04:01] Twisted is happy. :) [20:04:21] pkg_resources caches the loaded lists, btw, so subsequent scans do not cause re-reading of entry point lists [20:04:32] And Python of course caches any imports that occurred. :) [20:04:44] Only piece missing is on-disk cache of interface info [20:05:26] However, you could do that by getting ep.dist.location (directory or zipfile name) and using that to figure out where to cache. [20:05:28] Which is less important, I guess, since the main purpose of that is to avoid loading lots of Python modules, which entry points take care of. [20:05:41] ep.load() still loads 'em. [20:05:48] Ah [20:05:58] OTOH, you could just write one big-ass entry point cache with the interface info. [20:06:09] rather than directory-by-directory caches. [20:06:43] If you see an entry point in iter_entry_points that you don't recognize (e.g. its distro has a new version #), bammo. [20:06:44] Or the entry point could be for interface information which leads to something meatier which can be imported later? [20:06:53] You could do it that way. [20:07:05] You could also put the interface info in the name of the entry point, e.g.: [20:07:08] [twisted.plugins] [20:07:37] foo (twisted.interfaces.IFooBar, blah.baz.ISpammity) = some.module.with.a:PluginClass [20:08:05] Each 'ep' in iter_entry_points() has a 'name' attribute that you can inspect to parse stuff like that out. [20:08:21] In which case, you don't need a cache at all: the entry point names are their own cache. [20:08:47] The only visible character you can't have in an entrypoint name is '=' [20:09:05] Hmm. That brings up a point, I guess. I kinda dislike ini files as a storage mechanism for structured data. Is there another way to define entry points? [20:09:24] Sure, you can create them with the EntryPoint constructor. [20:09:43] (But of course they get serialized in .ini format) [20:09:56] EntryPoint(name, module_name, attrs=(), extras=(), dist=None) [20:10:31] so the above would be EntryPoint("foo (twisted.interfaces.IFooBar, blah.baz.ISpammity)", "some.module.with.a", ["PluginClass"]) [20:10:32] Okay, cool. [20:10:39] Er, almost cool :) [20:11:04] If it's the [twisted.plugins] part you dislike, you can do it like this: [20:11:12] could I subclass EntryPoint and add, say, an interfaces list to the constructor? [20:11:26] entry_points = {'twisted.plugins': 'foo (...) = ...'} [20:11:37] MyEntryPoint("foo", ["twisted.interfaces.IFoo", "twisted.interfaces.IBar"], "some.module", ["PluginClass"])? [20:11:55] Sure you could... but keep in mind that you'd need to be able to import that class when running setup.py.... [20:12:15] which is a tricky business, because then you have to have a dependency that's installed by the time setup() is executed... [20:12:28] And even setup_requires doesn't run soon enough for that. [20:12:55] * exarkun nods [20:13:02] gotcha [20:13:19] One of those args was named `extras' [20:13:32] Is that for arbitrary application-specific data, or does it have some specific meaning? [20:13:36] That's to specify dependencies that need to be met if you *use* the entry point. [20:13:43] ep.load() will try to resolve them [20:13:45] Ah [20:13:54] they are names of "extra" dependencies defined in install_requires [20:14:06] ep.load() will try to resolve them before actually importing the entry point. [20:14:11] as a feature request, how would adding **kw to EntryPoint.__init__ sound, as a mechanism for passing some structured data on to the application? [20:14:31] Actually, I was thinking I would do something like that in 0.7 [20:15:09] cool :) [20:15:10] The key requirement is that the data would have to be something that can be effectively squished into the 'name' [20:15:24] (in order to have inter-version compatibility) [20:15:53] Hmm, I hadn't thought about that. I guess EntryPoint stuff gets written to a file at some point? [20:15:56] And of course, the data would still have to be represented as strings. [20:16:04] Yeah, entry_points.txt in the egg metadata [20:16:15] hmm, tough [20:16:34] serialization is such a drag [20:16:48] That's why I kept it simple. [20:16:56] * exarkun nods [20:17:13] Well, that and the fact that setup() pretty much needs to only use arguments with "built-in" or stdlib types (including setuptools types) [20:17:28] right [20:17:40] This is a bootstrapping issue, rather than a matter of setuptools policy. [20:18:14] I've beat my own head up against it once or twice, which is why Chandler's EggTranslation system uses an .ini file to declare i18n resources [20:18:32] oops, okay, really need to take off now :) appreciate the help and discussion, and I'll probably be back at some point [20:18:34] Because any library we created to do it in a fancier way, would've run into the same bootstrap problem [20:18:45] np. l8rs [20:18:49] ** exarkun has left us [20:30:30] ** pje has left IRC ("Client exiting")