[02:13:37] [Global Notice] Good morning, all. The freenode network will be experiencing a significant late-scheduled outage later this morning. For information, please see http://freenode.net/news.shtml .... apologies for this inconvenience and thanks in advance for your patience and understanding. [04:02:26] [connected at Thu Jan 27 04:02:26 2005] [04:02:26] <> *** Looking up your hostname... [04:02:26] <> *** Checking ident [04:02:26] <> *** Found your hostname [04:02:57] <> *** No identd (auth) response [04:02:57] <> *** Your host is niven.freenode.net[niven.freenode.osuosl.org/6667], running version dancer-ircd-1.0.35 [04:02:57] [I have joined #peak] [04:02:57] ** niven.freenode.net set the topic to http://dirtsimple.org/2004/11/generic-functions-have-landed.html [05:03:00] ** vlado has joined us [07:05:01] ** vlado has left IRC (Read error: 104 (Connection reset by peer)) [07:05:51] ** tav_ has left IRC (niven.freenode.net irc.freenode.net) [07:05:51] ** etrepum has left IRC (niven.freenode.net irc.freenode.net) [07:05:51] ** Maniac has left IRC (niven.freenode.net irc.freenode.net) [07:06:40] ** vlado_ has joined us [07:06:40] ** tav_ has joined us [07:06:40] ** etrepum has joined us [07:06:40] ** Maniac has joined us [08:19:50] [Global Notice] Good morning, all. The freenode network may experience one or more significant outages in the next couple of hours. For information, please consult http://freenode.net/news.shtml .... thanks in advance for your patience, and thank you for using freenode! [09:12:45] ** rdmurray has joined us [09:21:42] tav_ is now known as tav [09:40:27] ** apoirier has joined us [09:54:27] ** [apoirie] has joined us [10:02:59] ** apoirier has left IRC (Read error: 60 (Operation timed out)) [10:07:15] ** gbay has joined us [11:32:33] ** pje has joined us [11:32:48] hello, pje [11:33:09] Howdy [11:34:01] your blog made me go buy GTD [11:34:09] well, yours and a few others. [11:34:19] reached critical mass and I went and bought it [11:34:23] Ah. Did you click through on my amazon affiliate link? ;) [11:34:35] no :< i bought at a local bookstore [11:34:41] Ah well. :) [11:35:34] i kinda dig it.. i'd like to implement it in python somehow [11:37:13] speaking of python, i just got flight itinerary for pycon! [11:38:14] * rdmurray reminds himself to think about going to pycon. [11:40:10] Think fast; early bird deadline is tomorrow. :) [11:41:18] Ah. Thanks for that info. [11:41:27] * rdmurray puts it on his todo list for today. [11:58:25] Cool; I just got back proofs of my Dr. Dobbs article on decorators [11:59:06] This'll be my first "professional" publication [12:08:22] excellent [12:09:41] PyCon is going to be good this year [12:09:48] not that it hasn't been every other year [12:46:40] ** [apoirie] has left IRC (Read error: 104 (Connection reset by peer)) [12:48:00] ** rdmurray has left IRC ("User disconnected") [12:54:04] ** apoirier has joined us [12:54:37] So Bob, you think you might want to sprint a bit on that binary plugin thing at PyCon? [13:00:14] yeah I'd love to [13:00:21] it's definitely something that needs to be done [13:00:42] Cool. The thing I'm stuck on on it is how to specify a "platform" sensibly [13:01:00] e.g. OS,processor,etc. [13:02:06] One way that you can punt on dependency metadata is just to let arbitrary code figure it out and say yes or no [13:03:21] I suppose you could always try importing the binaries and if they fail, it's the wrong platform. :) [13:03:48] Actually, checking whether the extensions match the platform extension would at least distinguish Win vs. Unix. :) [13:03:53] that is dangerous though, you could cause a link error and kill the process on the right platform but the wrong version and missing dependencies [13:04:16] True. [13:04:51] Thing is, for reasonable distribution of multiple platform versions, you're going to want to have platform info in the filename. [13:05:15] which can be done in the same way build dirs are done by distutils [13:05:39] Yeah, it doesn't put much platform info there though. [13:06:10] But I'm thinking that maybe there could be a system-local config file where you could list what platform strings (besides exact match) are OK for your machine. [13:06:23] I don't really like that [13:06:51] I don't think it would buy anything in practice, and it's too much of a burden on users [13:07:52] So you think exact match would be good enough? [13:07:57] I think that the first release should just run some arbitrary code and say yes or no.. the arbitrary code will have a bunch of useful functions in its namespace for figuring out what platform it is [13:08:23] as far as naming goes, distutils is basically good enough except for processor architecture [13:08:24] I was thinking first release would be exact match or regular expression match of filename platform info. :) [13:08:42] I.e. *really* dumb. :) [13:09:28] well allowing arbitrary code lets you get as tricky as you need to, the default could do just that [13:10:03] Well, filename checking can be done even before your application tries to download and install a plugin. [13:10:29] And if you have the plugin already, you don't need to even open it to see it's not ready for your platform. [13:11:01] Er, compatible with. [13:11:39] so this is also going to include a mechanism for downloading and installing plugins automatically? [13:12:00] Sure, as long as you're the one coding it. ;) [13:12:02] j/k [13:12:07] I'm just looking ahead. [13:12:21] it could also go the other way, on the server you can have a code snippet that is download and run that says "ok, you need to download this one" [13:12:29] True. [13:12:58] that's kinda how the pimp/packagemanager stuff works for MacPython [13:13:34] It just seems so "dirty", somehow... running arbitrary code I mean. [13:13:43] Even though you're gonna run a lot more of it later. :) [13:14:15] I felt the same way for a while but it's a good way to figure out what kind of "safe" things you can do [13:15:09] after you write a bunch of these dependency scripts you figure out "ok, we need a way to call a sysctl, a way to detect the OS version, a way to detect the presence of a library..." and you write safe ways of doing that later on [13:15:46] Maybe the "right" way to handle it is to wrap those platform checks as pseudo-plugins. [13:16:13] I mean, if you have a dependency list, you can say you need 'bsd-libc>=#.XX' [13:16:39] And 'x86-processor>=386' ;) [13:16:51] that doesn't really buy much [13:16:55] just more work [13:17:05] directx-version >= 9 [13:17:15] specific-registry-key = foo [13:17:39] oracle-installed = X [13:18:10] True. [13:18:13] so instead of writing one plugin you have to write 6, because the standard implementation isn't going to have the 5 domain specific meta-plugins you need [13:19:00] OTOH, you also have no way to share any of those checks between plugins either. [13:19:12] you have no practical way to share those checks anyway [13:19:27] The main reason I dislike arbitrary code is that I've seen lots of people do hideous things in setup.py files [13:19:33] oh me too [13:19:58] but a lot of the hideous things end up being abstracted eventually by someone and used in a nicer way [13:20:43] Yeah. I was just hoping we could narrow it down to a few things like file globs, environment vars, registry keys, and imports. [13:21:33] Hm. I suppose we could just include setup.py and have a 'platform_check' command. :) [13:21:50] apple supports 7 safe ways to determine dependencies, plus lets you run arbitrary scripts at various points in the installation process [13:22:05] http://developer.apple.com/documentation/DeveloperTools/Conceptual/SoftwareDistribution/Concepts/sd_spec_install_req.html [13:22:25] I think we'd need to implement about 12 or 13 dependency things [13:22:30] which would take a while [13:22:43] Ugh. [13:23:11] so it would probably be more interesting to work on the code that matters, and punt on dependency crap and let it use arbitrary code until someone is bored enough to write something that is safe [13:23:15] "safe" [13:24:39] What I was thinking is that if we used setup.py, we could actually ship the developer's original dependency info included. [13:25:10] But that might be overkill, I suppose. [13:25:25] I was just thinking that the dependencies on the target platform would be the same as on the build platform. [13:26:37] shipping setup.py is a bad idea [13:26:58] the setup.py for PyObjC does a lot of really scary things, for example [13:27:27] and that's not necessarily true [13:27:43] Okay. [13:27:48] So how much arbitrary code? [13:28:46] I guess pre-install (check deps) and post-install (add registry keys, or whatever it needs to do) arbitrary code is good enough [13:29:42] post-install could go away if we say "your plugin has to be smart enough to do post-install on first use if it needs to" [13:30:42] Hrm. I'm not sure about the whole post-install thing; what if there's more than one copy of the same plugin installed for different apps on the box? [13:31:02] that's a good point [13:31:28] post-install is rarely used for anything useful in practice, from what I've seen [13:31:43] and may not apply to plugins [13:34:42] Also, I'm not sure pre-install is the best term, either; I don't know if there's an "install" per se. [13:34:52] Although a given application might have such a concept, I guess. [13:35:31] well it's some kind of precondition, what it's called isn't so important [13:36:29] Heck, it seems to me that maybe the simple thing to do is just for things with such platform considerations to have safe checks at the beginning of their modules. [13:36:51] And then raise an error explaining the problem. [13:37:53] yeah but that's after you've downloaded the whole thing and put it somewhere [13:38:08] the precondition code could be separate from the plugin itself [13:39:44] it might recommend which package to actually download, or say no, you can't use this at all [13:40:04] ISTM that these things you're talking about are more for libraries than plugins. [13:40:20] what's the real difference? [13:40:36] A library is something you can meaningfully use with more than one application. [13:41:44] why make the difference? [13:42:10] say you have an application that has its own Python, like Chandler.. it's still going to need "libraries" to facilitate its "plugins" [13:42:26] I'm just separating use cases for better understanding of each use case. [13:42:33] I'm not saying there should be separate mechanisms. [13:43:30] But I am noting that libraries are more likely to have platform dependencies apart from just their included extensions being executable. [13:43:45] that is true [13:43:58] So, they need more platform check capabilities than just os version/processor/python version and import dependencies [13:44:01] but wouldn't normal "plugins" be largely of the pure python variety anyway? [13:44:07] Right. [13:44:18] ** lrl has joined us [13:44:21] I'm just noting that it's a continuum with libraries on the heavy checking end and plugins on the light end. [13:44:38] lrl is now known as asrenzo [13:44:52] absolutely true [13:46:37] ** asrenzo has left IRC (Remote closed the connection) [13:46:42] Oops; business call... [13:46:56] pje is now known as pje|phone [14:09:13] ** rdmurray has joined us [14:33:26] pje|phone is now known as pje [14:33:37] re [14:35:13] ** _pje has joined us [14:35:13] ** pje has left IRC (Read error: 54 (Connection reset by peer)) [14:46:41] _pje is now known as pje [14:51:09] ** apoirier has left IRC (Read error: 113 (No route to host)) [14:56:23] The pre-install checks you do with the Mac stuff, they're in the list of packages, right? It's not part of the download. [14:57:10] Or am I confused? [14:58:35] That's how most package systems work. I don't have any experience with the mac ones myself. [14:59:18] Hmm. I'm probably wrong saying "most", since I only have experience with one :) [15:03:52] I was talking about Bob's Python package list for the Mac; ISTR it has Python expressions embedded in the XML. [15:04:18] And I was wondering if we could just use that technique and just punt on automatic platform detection. [15:13:00] that's more or less what I was talking about [15:13:13] Okay, excellent. [15:13:13] yes it's in the package list [15:14:26] the package list probably shouldn't be so centralized though. You could point to http://foo/plugin.metaplugin, which will have the metadata about the binary plugins you download and the dependency checking scripts [15:14:43] so from that "metaplugin" description you can find out if you can use a plugin, and which one to download if you can [15:14:56] the metaplugin can also have checksums for the binaries [15:15:08] I like it. We could actually make that be the PyPI download URL. [15:15:10] and the metaplugin itself could be signed so that it could be verified to come from a particular author [15:15:23] Sounds like fun. [15:15:32] Is there an existing format like this that we can steal? [15:15:42] and PyPI could have a keyring for the authors or something.. [15:16:04] S/MIME ? [15:16:48] Heh. I meant for the metadata part. :) [15:16:58] oh, no, probably not [15:17:24] Well, if all else fails, ConfigParser to the rescue. :) [15:17:50] the format we use for pimp is the plist format, which is basically just a serialization for simple objects.. so it has no schema beyond how to serialize a list, dict, number, etc. [15:18:01] * pje nods [15:18:41] I think this file needs to be human editable, though; I don't see how you'd generate it all from distutils. [15:18:56] MIME wouldn't be bad, the email module should be good enough [15:19:56] ** sprout has joined us [15:20:15] * pje waves [15:20:24] * sprout bows [15:20:24] Bob, this is Ted Leung from OSAF [15:20:41] hey Ted [15:20:50] <-- Bob Ippolito [15:20:52] I mentioned some of our plugin discussion to him and he popped over to partake. [15:20:53] hi bob [15:21:03] well, more like learn. [15:21:11] and perhaps whine. [15:21:29] Btw, Ted, transcript of today's talk is at http://peak.telecommunity.com/irc-logs/2005-01-27.txt [15:21:31] so far we've only been talking about metadata associated with plugins [15:22:04] Mainly, platform requirements metadata [15:22:16] yes [15:22:31] For plugins and libraries with binary (non-pure Python) content [15:22:46] Plugin = library for a specific app [15:22:56] Plugins are less likely to include native code, libraries more likely. [15:23:00] ** bear_ has joined us [15:23:13] Hi Bear. [15:23:22] hi [15:23:31] Bob, Bear is Mike Taylor, also from OSAF, he works on Chandler build stuff [15:23:31] but of course pure python code still has requirements too (other code, particular set of Python versions, ...) [15:23:41] Right. [15:24:10] Distutils supports automatically naming binaries w/platform and python API version, so at least that basic info is available in the filename. [15:24:27] But for more esoteric platform detection like "Is DirectX 9 installed", more is needed. [15:25:03] distutils can also distinguish between platlib and purelib, so that the platform can be ignored if the code is all python [15:25:12] So far, our discussion has been around putting that info in an external file that lists the various versions of a package that are available (e.g. Mac version vs Win version) [15:25:49] And also putting any special requirements detection code into that external file so a smart download tool can avoid downloading before prerequisites are met. [15:26:10] And, putting the URL of that external metadata as the PyPI URL, so that it can be used to find the packages. [15:26:44] The big difference between what we're talking about and what already exists in distutils is that the emphasis is on apps rather than on the "Python installation". [15:26:58] I.e., there might be more than one Zope or Chandler install on a box, each with its own plugins and libraries. [15:27:08] that may or may not share a Python installation [15:27:26] Yeah, and different versions of Python, for that matter. [15:28:37] So, basically, we're talking about a Python equivalent to Java's .jar [15:28:49] Coupled with some distribution amenities, and distutils support for building them. [15:29:15] So that in principle, you can do this for anything that already has a setup.py today. [15:29:54] At this point we have buy-in from the authors of both py2app (Bob here) and py2exe (Thomas Heller) [15:29:59] similar to bdist_rpm, bdist_dumb, bdist_mpkg, ... [15:30:27] I'm sure that buy-in for cx_freeze would happen if Thomas and I did it :) [15:30:43] Is that the equivalent of py2app/py2exe for Linux? [15:31:03] it supports Linux and Windows, yes [15:31:07] Btw, bear mentioned earlier that Chandler's OS/X build philosophy isn't compatible with py2app, but the details were over my head. [15:31:33] did he do that here? [15:31:42] at the time py2app was a quickly moving target (this was 9 months ago or so) [15:31:43] No, on #chandler; sorry. [15:31:49] so I need to revisit py2app [15:32:04] well regardless, if you build a Mac app without py2app you will probably do it wrong ;) [15:32:18] it seems to be working :P [15:32:19] * pje grins [15:32:20] if you need any specific functionality you should let me know [15:32:27] seems to be working on your machines, sure [15:32:40] I bet I could construct a situation that breaks your application without touching the application itself, though [15:32:40] And people think I'm um, "confident", yeah, that's the word. :) [15:32:46] it's just a bundled app now - nothing fancy in the install at all [15:32:54] that's the right way to do it [15:33:13] but the process of bundling an application correctly is difficult [15:33:25] it has a manifest and all that, and soon it will even be placing runtime data in the proper spot instead of in the app dir [15:33:43] beyond collecting the Python bits, just with regard to Mach-O headers and the like, which often need to be rewritten [15:34:09] the current method uses a shell script for launching [15:34:17] yuck [15:34:21] * bear_ nods [15:34:27] very very brute-force [15:34:41] py2app uses a precompiled application stub that can detect certain classes of errors and present a nice dialog [15:34:45] * bear_ whispers inherited issues and looks innocent [15:35:16] yea - that is what drew me to it originally [15:35:18] the contents of the dialog can be determined by running an __error__ script, which can even send the user to a particular URL [15:35:19] Of course, Chandler doesn't have a setup.py at the moment, so it's kind of moot wrt py2app for the time being. [15:35:39] that's not true, you can create a setup.py for bundling purposes even if you don't typically use one [15:36:11] I have written scripts that generate a setup.py (or effectively do it) on the fly by way of drag+drop (py2applet) or by parsing an Xcode project file (in ReSTedit) [15:36:16] true - the real reason for not having a setup.py is the time/effort it would take to get it working when compared to the other more immediate todo items [15:37:07] I'm somewhat more interested in being able to manage Chandler's dependencies this way, than in Chandler's base itself. [15:37:37] well, most py2app setup scripts are trivial anyway.. probably much shorter than what you have now [15:37:45] (e.g. test it with different versions of a dependency) [15:37:46] do you guys have a release coming up soon? [15:38:05] milestone builds are every 2 or so weeks [15:38:10] also py2app's alias build mode is incredibly useful for development, because the application doesn't need to be rebuilt when you change source [15:38:57] well I'll take a look at Chandler at some point, hopefully soon, and see what it would take [15:38:59] currently the script runs thru a "manifest" list and builds the various pieces - very distutils like [15:39:34] something tells me I'll be looking at it sooner than later and dropping by for some clue-stick hits to the head [15:40:01] best way to get ahold of me about that sort of thing is email [15:40:25] you are on the distutils mailing list IIRC, so I can use that so others can benefit from my goofs [15:40:35] bear_, anything you've got now that runs as Python code can be made into a distutils command. [15:41:06] The basic idea is that a distutils Distribution object contains all the *data* to drive the process, and a distutils command does stuff with the data. [15:41:17] pje, I would agree with that but I kinda have a gut feeling that some of what chandler does to python and the third party libs may make that less than smooth [15:41:19] pje: that is entirely true, I moved a lot of PyObjC's functionality into build commands (to build libffi, documentation, run tests, ...) [15:41:43] not really, all you have to do is make the command's run() do whatever you're doing now [15:42:18] Also, I would hope that eventually Chandler won't "do" things to Python third party code. :) [15:42:27] * bear_ moves distutils up the todo list another notch [15:43:05] http://cvs.eby-sarna.com/setuptools/setuptools/command/test.py?rev=1.4&content-type=text/vnd.viewcvs-markup [15:43:11] some of it is unfortunately going to remain - the patches that fix things like libxml2 have not been accepted into libxml2's source code so we will always have to build our own for that [15:43:21] That's a simple distutils command extension that runs unit tests from a setup.py, for example. [15:43:46] bear_, okay, but that's not "doing" something to it, is it? [15:43:58] I mean, you're just distributing a patched version then. [15:44:24] Or are you saying that this is libxml2 as a part of Python itself? [15:44:36] it compiles the patched source, builds the python extensions with chandler's python and bundles it all up [15:44:52] When I use patched versions of stdlib modules I put them in a different place with a different name. [15:45:10] Thus avoiding conflict with any third-party code that might need the unpatched vesion. [15:45:13] chandler uses its own Python though, so that's not such a problem [15:45:14] version, even. :) [15:45:24] etrepum, it is for development. :) [15:45:33] perhaps [15:45:44] Even Zope corp stopped using a custom Python at least half a decade ago. :) [15:45:56] with py2app you can point it to whatever Python runtime you want to use in its Info.plist [15:46:06] ** debugger has joined us [15:46:16] hello [15:46:22] Hey. [15:46:46] So, back to plugins, the external metadata file would list platforms, dependency checks, and what else? [15:47:01] Signature, you already mentioned. [15:47:20] update url's? [15:47:42] canonical current version URL makes sense [15:47:44] I think the metadata URL *is* the update URL, right Bob? [15:48:04] Download urls are in the file for each platform and offered version. [15:48:19] So, to upgrade, you check the metadata file for whether there's a newer version listed inside. [15:48:29] Sort of like an Eclipse feature file. [15:48:49] (Maybe we should see if their format for that is worth stealing.) [15:49:08] that depends [15:49:14] Which depends? [15:49:17] there may be one metadata file per version [15:49:23] Ah. [15:49:35] its possible to do something like "Selecting Multiple Objects" from http://sqlobject.org/docs/SQLObject.html in PEAK? or this kind of thing should be encapsulated as a feature of our DM? [15:49:47] either one metadata file per version, or one metadata file that contains information for multiple versions [15:50:21] debugger, write a query method using the preloadState method [15:50:52] pje: so its my "or" section? :) [15:50:59] I was thinking of this as a publishing mechanism, where you're listing the stuff you have available. [15:51:14] debugger: yes. :) [15:51:48] debugger: Of course you can make it as arbitrarily parameterized as you like. [15:52:33] So, if I were publishing PEAK as a library plugin, when I release a new version I'd edit the PEAK publication file to list the new version's source and binary packages. [15:52:56] OTOH, since PyPI supports multiple versions of the same package, I suppose we could punt on all that and let PyPI handle it. [15:53:07] pje: sure hehe [15:53:09] I.e. a new download URL for each release [15:53:51] thats true, PyPI itself could be the canonical URL [15:54:26] At least for libraries; for plugins it's more likely that e.g. dev.zope.org or an equivalent community site for Chandler would be the source for plugins. [15:54:53] pje: what you think about the features of sqlobject? I mean, the AND() and startswith of expressions like: "AND(Address.q.personID == Person.q.id, Address.q.zip.startswith('504'))"? you rater do that in plain old SQL? [15:55:33] debugger, yes. peak.query has a prototype of something similar, but ultimately it'll be replaced with a query syntax based on Python listcomps. [15:56:39] Hm. I still think I like one file per library/plugin rather than per version. [15:57:04] It's got to have download info in it, and I don't see how you're going to generate that from setup.py, without just doing the same thing as you'd do by editing the file in the first place. [15:57:05] pje: hummm peak.query is on peak cvs? [15:57:23] debugger, see peak.query.algebra and peak.query.tests.test_algebra [15:57:43] ok thx! [15:58:33] What did you have in mind, Bob? [16:00:10] One file per library/plugin would probably be sufficient [16:00:30] it seems sensible enough [16:00:38] pje: whats the proper way of using the arguments in isqlconnection? an array of args or a arg per function argument, I mean, db("insert ... values(%s, %s)", "one", "two") or db("insert ... values(%s, %s)", ["one", "two"]) [16:00:59] though this could become a problem when we upgrade the format [16:01:38] f.ex. SuperPlugin 1.0-1.5 uses pyplugin metadata format 1.0, but then ports over to pyplugin 2.0 for SuperPlugin 2.0 [16:01:44] will this info live with the plugin or be collected in a central location on the user's computer? [16:01:51] debugger, it depends on the driver, alas. This is an area where PEAK doesn't protect you from driver-specificness. [16:01:59] That's what peak.query is supposed to do for you eventually. [16:02:11] etrepum, ouch. [16:02:27] so then either you need a future-proof format, a URL change, or separate files [16:02:40] OTOH, you have that problem anyway, right? [16:02:56] bear_, we're talking about the distributor of a plugin listing info for people to find/download them. [16:03:10] So, like Eclipse, an application can have an "upgrade plugins" feature. [16:03:26] It's not needed once you've found and downloaded the right plugin for your platform. [16:03:32] right - but you need to cache that locally so you know what you already have (or a helper tool can know) ? [16:03:35] pje: hummm, so I'm better using something like DBSingleStyle (http://cvs.sourceforge.net/viewcvs.py/ppa/misc/DBSingleStyle.py?rev=HEAD) hehe [16:04:12] debugger, in practice, DB portability is a myth, like I keep telling you. :) [16:04:38] an installation database of some kind would make sense, but it would only need a subset of the information present in the plugin description [16:04:42] bear_, let's say we're talking about Chandler parcels... [16:04:54] If they were distributed as .jar-like files, you could just scan the directory to know what you have. [16:05:23] jEdit and Eclipse plugins in fact work almost exactly like that. [16:05:26] yeah, I mean "installation database" very loosely, it could be some particular search locations where each "row" of the database is a .pyplugin file that has some information about itself [16:05:34] pje: humm, but using dbsinglestyle its one less thing you have to bother, I guess hehe [16:05:53] so you are saying that you will only ever have one version present at a time? or this manifest file will have version info to make it uniquely named? [16:05:59] debugger, yeah, and that's why PEAK has the type converter stuff I explained before [16:06:15] the plugin files should definitely have a version in their filename [16:06:16] bear_, the publishing manifest can list more than one version and more than one platform in it. [16:06:30] The binaries, however, are specific versions, and you can have more than one version present at once. [16:06:48] You may not be able to have more than one *imported* at once, which is a separate issue for a later phase of this effort. [16:07:00] ok, just making sure I understood the background assumptions [16:07:01] (E.g. you can't have more than one version of wxPython running no matter what, I think.) [16:07:11] pje: humm, but that only works when the results come from the driver, not the other way arround, isn't it? [16:07:15] No prob; just trying to clarify them. :) [16:07:42] I see the no-multiple-version being a small problem for a developer trying to do testing of multiple versions [16:07:57] debugger, correct at the moment, yes. However, there's another relatively new mechanism, driver-specific thunks, which Ty and I used to implement Sybase/Oracle versions of outgoing stuff. [16:08:02] having concurrently running multiple-versions of anything is a problem nomatter what [16:08:12] having concurrently installed multiple-versions is not necessarily a problem [16:08:48] by concurrently running, I mean concurrent in the same process space [16:08:54] For pure-Python modules, having multiple versions running is *possible*, but rather tricky. [16:08:54] * bear_ nods [16:09:08] pje: will it work for the other drivers? :) [16:09:11] For C extensions, it's also *possible*, but requires a lot of work from the C module. [16:09:30] but if you have a S/MIME plugin for chandler, for example, you can't run two at the same time usefully anyway [16:09:32] debugger, yes. It's just a way to add driver-specific configuration like constants or methods. [16:09:53] you could have two installed, and choose one or the other at runtime [16:09:59] but running both at the same time often makes no sense [16:10:00] * bear_ ponders plugin aliases [16:10:36] I would imagine it would be up to the application's config file to determine which version of a plugin gets loaded [16:10:36] pje: you recall the file that has that stuff? and, whats the name of the "relatively new mechanism thunk"? [16:10:37] it would be an issue if a plugin has a dependency on another plugin [16:10:42] debugger, it's the 'appConfig' attribute of SQLConnection. [16:11:11] pje: ah thx. i'll stop bugging you now :) [16:11:15] You define e.g. 'sqlite.appConfig' variables in your config, and then when you access the connection's appConfig attr you can get access to your application-defined variables. [16:11:37] It's just a simple way to have db backend-specific stuff, it's not very polished. [16:11:53] bear_: in that case, you would have to pray that each module in the plugin has a Python module name that includes the version :) [16:11:55] etrepum, I'd like the version resolution to be automatic from plugin metadata. [16:12:24] Here's the mechanism I have in mind... [16:12:24] hehehe [16:12:33] pje: it could automatically prefer the most recent version of an installed plugin, but it doesn't have to use it [16:12:41] Right. [16:12:50] Let's say plugin A depends on plugin B... it lists it in its metadata. [16:13:11] The loader that adds the plugin to sys.path checks to see what it needs. If plugin B is already on sys.path, we're done. [16:13:30] If not, search sys.path for pluginB-*.pyplugin [16:13:37] And figure out what version to add to sys.path. [16:14:19] So, there's a simple routine, akin to the wxversion select function, where you say, iwant('pluginB',ver=">=6.2") [16:14:21] or some such. [16:14:53] Heck, we might not need metadata. Just put it in the module. [16:15:02] After all, you don't need it until you need it. [16:15:17] On the other hand, having advance notice is useful for an installer to make sure it can download everything. [16:15:24] Decisions, decisions, decisions. :) [16:16:17] We should probably write up the use cases so we can prioritize them. [16:16:51] I think we'll get to all of these features eventually, but even just *having* the format will be a big step forwards. [16:17:01] I don't think .jars started out with any of this fancy stuff. ;) [16:17:32] python r0x0rz! [16:17:38] Eh? [16:17:55] sorry - got overly excited ;) [16:18:10] About us adding something Java has had for years? ;) [16:18:50] * bear_ doesn't do java [16:18:54] I was just being happy [16:18:57] If we can get to consensus about the most basic format, I can throw together a bdist command for it before PyCon. [16:19:22] humm, I see people nagging java for all sort of reasons, but in the end, i see much of their "infrastructure" ported to other systems. [16:19:24] Then, at PyCon we can focus on the hard parts like extension autoextract [16:20:13] No argument there; Java definitely has lots of good infrastructure. I steal designs from there whenever I see good ones. [16:21:30] I actually think I'll look at Eclipse's publication formats for listing download URLs, platforms, version info, etc. [16:21:53] And see if there are any ideas (if not formats) worth stealing there. [16:22:41] dev-c++ has a plugin setup that allows for downloading and such - maybe they have some insights to ofer [16:22:54] That'd be cool too. [16:23:13] This is certainly a "frequently solved problem", so we might as well learn from others. [16:25:36] You know, if I look at it in terms of what are the absolute bare minimum requirements, it seems to me all we need [16:26:09] is to have a distro format, unpack stuff into a version-and-plugin-specific directory, and have a function to add plugins to sys.path [16:26:20] At least, that's all we need if we ignore OS/X. ;) [16:26:32] http://michel.weinachter.free.fr/DevPack_howto.html [16:26:46] * pje reads the URL [16:26:48] that's the tutorial for creating a package for dev-c++ [16:27:34] It doesn't look quite like the same thing. [16:27:57] It looks more like the info for creating the package; we've been mainly talking about a file describing available versions and platforms. [16:30:27] Eclipse has a notion of "features" which are a file that lists a set of required plugins, and an "update site" that lists features and plugins for download. [16:30:58] That's more like what we're talking about as far as a vision of pushing an "update plugins" button in Chandler et al. [16:31:13] well if we require py2app to be installed in order for this to work, it can rewrite the Mach-O headers if necessary on unpack [16:31:40] How much of py2app needs to be installed? [16:31:50] Could we just bundle it inside the plugin? [16:32:11] we need a "pyplugin runtime" anyway, so that will depend on py2app [16:32:16] it doesn't need to be in the plugin [16:32:52] I was just thinking it'd be nice to be self-contained, so you can just put it on your PYTHONPATH and use a library with today's Python. [16:33:17] But I suppose if there's a lot of infrastructure needed, it might add a lot of overhead. [16:33:37] I'd rather punt on that [16:34:11] v2 todo list :) [16:34:32] Well, we only have to punt on that for C packages on Mac. :) [16:34:58] (Since unpacking should work fine on other platforms.) [16:35:02] for plugins that require C packages for the Mac - correct? [16:35:10] bear_, yes. [16:35:33] More precisely, ones that *contain* C extension modules and/or .so files [16:36:05] They won't be runnable without an install step to bind their symbols to the right OS/X libraries. [16:36:19] * pje is probably not stating that correctly, not being a Mac linking guru [16:38:14] it depends on how the C stuff was linked [16:38:32] I don't see a real benefit to *not* having a runtime [16:38:54] you're going to want one anyway to deal with listing plugins, adding them to sys.path, etc. [16:39:05] For plugins, yes, all that's true. [16:39:26] For *libraries*, it'd be nice to use 'em without headaches. [16:40:21] Just add to PYTHONPATH and off you go. [16:41:20] setting up a PYTHONPATH like that is non-trivial, much easier to have a runtime and a config file [16:41:24] py2exe has shown that you can easily use Python stubs to load an external .dll/.so file, so it doesn't add much overhead on such platforms to include autoextract stubs. [16:41:42] True. [16:42:04] Especially for click-and-run Python apps distributed as plugins. [16:42:34] But that's a different sort of "runtime" perhaps than what you're thinking of. [16:42:55] I'm thinking Windows-style double-click to run an executable .jar application. [16:43:10] Which is also not a "library" use case. [16:43:14] yeah so you would need a runtime for that too.. a different kind, of course, but a runtime nonetheless [16:43:31] So, we actually have three major kinds of distributions: library, plugin, and application. [16:43:43] Where the latter is really just the same as the other two, but with a '__main__' script. [16:44:15] I think that this plugin system is compelling enough to convince users to install a runtime (which could be included in "batteries included" distros of Python, such as ActiveState's) [16:44:20] And in theory, both py2exe and py2app could just become that click-to-run runtime wrapper, but bundled. [16:45:05] for py2app it would actually be totally trivial... [16:45:08] That is, a py2exe app is just an application distro with the launcher prepended [16:45:19] Excellent. [16:45:29] in Mac OS X, applications are opaque directories [16:45:42] so you just need to make a copy of the template, throw a file in there, and you're done [16:45:46] Ah. Interesting. [16:46:12] Okay, now *I'm* getting excited. :) [16:46:15] I can show you how the stuff works at PyCon, it's pretty cool [16:46:21] (mac applications) [16:47:38] So that template could easily include the parts of py2app itself needed to relink the resulting application, right? [16:48:19] in the context of an application bundle, it can rewrite the link commands while building it [16:48:52] But that won't help the application if it in turn has plugins. [16:49:33] when you build a plugin it could re-link in a way suitable to run from an application by default [16:50:07] Then why is a runtime needed for plugins? [16:50:18] it doesn't work for plugins that live outside of the application itself [16:50:26] plugins that are added on afterwards [16:50:29] Ah. That's what I was referring to. [16:50:40] So, bundled plugins are okay. [16:51:13] Does that mean you can build a library in a form suitable for running with the platform Python, then? [16:51:29] Oh well, never mind. [16:51:36] if you can determine a fixed place for it to live on the filesystem [16:51:48] Ugh. [16:52:12] a "fixed place" can be relative to an application bundle though, which is why building an application bundle out of a set of plugins is no problem [16:52:16] Okay, if you have a Mac, you'll have to have py2app. :) [16:53:24] py2app is 256k as an installer package (excluding examples, scripts, tools) [16:54:10] I don't think that user-supplied plugins are going to be restrictable to a single location. [16:54:18] correct [16:54:30] So the Mac "plugin support" runtime for use with an application will have to include relocation facilities. [16:54:39] but anyway, you can't load a dylib from a zip file anyway [16:54:51] so it will have to be spit out to a temporary location and rewritten to be run from that location [16:54:52] Yeah, but for other platforms, self-extraction stubs would be sufficient. [16:54:58] maybe [16:55:19] Are there other platforms that have location-specific linking like that? [16:55:32] for Linux, can you change the LD_LIBRARY_PATH at runtime (or whatever it's called)? [16:55:38] because that could be a problem [16:55:57] you can have a user-specific LD_LIBRARY_PATH if that's what you mean [16:56:13] yes but can you change it while your application is running? [16:56:15] that is necessary... [16:56:16] etrepum, I see your point. [16:56:36] hmm, good question [16:56:52] for plugins that depend on symbols from Python only, this is not a problem on any platform (if you can assume Mac OS X 10.3+ with correctly linked extensions) [16:56:59] would python "see" the changed environment variable - all that is normally loaded when the binary is launched [16:57:04] however for plugins that depend on say, libjpeg [16:57:05] I was under the impression that if the dependency .so's live adjacent to the Python extension .so's, it wouldn't be a problem. [16:57:21] it's not Python that would "see" the environment variable, it's libc [16:57:30] and libc may very well cache this value at startup [16:57:46] I know Darwin's dyld runtime caches DYLD_LIBRARY_PATH, etc. at startup [16:58:37] Hm. Here's an odd question... [16:58:48] If Python loads the dependent library, does that fix the problem? [16:59:03] I.e., if the stub loader could explicitly request the loading of the dependency? [16:59:17] Would the symbols in the extension then be resolved from what's in memory? [16:59:52] for Mac OS X, no [17:00:00] Fooey. [17:00:04] Mac OS X uses a two-level namespace [17:00:08] I had a feeling that would be the answer. :) [17:00:40] how would it know that /foo/libsdl.dylib is the same as /somewhere/else/libsdl.dylib? [17:01:00] And you can't tell it so at runtime. [17:01:28] not unless you lookup symbols on your own [17:01:37] but you also have the problem of sdl needing libjpeg [17:01:45] pygame -> sdl -> libjpeg [17:01:47] Yecch. [17:02:04] But none of this is a problem for bundled plugins. [17:02:41] if it's already in an application bundle you can specify paths that are relative to the executable so no.. because you would extract them to the right place and rewrite their headers ahead of time appropriately [17:02:46] And, there are existing reasonable distribution mechanisms for the libraries themselves, for use with Mac OS Python. [17:03:09] third party libraries are included in applications, they punted on having something like apt/rpm [17:03:28] * pje boggles [17:03:31] Okay. [17:03:37] makes life a lot easier for users, at the expense of a little disk space and bandwidth [17:03:44] I guess they plan to sell big disk drives. ;) [17:04:00] Yep, and a lot more um, interesting for developers. :) [17:04:35] well OS X comes with an extremely rich set of functionality, it's not terribly often you even need third party junk [17:04:45] Heh. [17:05:15] between Cocoa and QuickTime, plus all the unix goodies it comes with (OpenSSL, iconv, ...) [17:05:23] * pje nods [17:05:30] future versions of OS X might even come with wx [17:05:51] but I didn't say that, typing on a laptop running a future version of OS X [17:06:04] But okay, so there's no way you can use a plugin for an app on MacOS/X without "installing" it, unless it has no included non-Python libraries. [17:06:05] * bear_ chuckles [17:06:17] * pje laughs [17:06:30] Wouldn't help Chandler, they use bleeding-edge wxPython all the time. [17:06:42] (if I understand correctly) [17:06:50] yes [17:06:50] by the time said future version of OS X comes out, that may have settled [17:07:11] but anyway, *if* you rewrite headers, you can do it [17:07:23] if you don't, then you can't (deterministically) [17:07:52] Ah well, some Python platforms don't have dynamic linking at all. ;) [17:08:00] So we'll have to take what we can get. :) [17:08:29] a quick google says that LD_LIBRARY_PATH is not alterable after startup and should be avoided in favor of re-linking as required anyways [17:09:06] I'm just disappointed in that I was hoping to avoid an explicit install step. [17:09:48] But it also sounds like the vast majority of stuff can get away without it, and it's only stuff like pygame, wx, etc. that have the issue. [17:09:56] I.e. interfaces to outside libraries. [17:10:10] And for applications, most of that stuff will be included in the application. [17:10:11] do those even fit the plugin paradigm? [17:10:20] bear_, yes, if a plugin needs it. [17:10:24] they can fit into the plugin paradigm [17:10:33] Imagine if somebody wrote some sort of VOIP plugin for Chandler... [17:10:59] I guess I've always thought of an install step as being required [17:11:13] Well, I mean other than "copy the file here and go" :) [17:11:38] Eclipse until recently, you could just drop a plugin in the right directory and it would be detected on startup. [17:11:42] jEdit is the same. [17:11:57] well, even those plugins would work if the dependency was already installed as part of the system - and not in user-space ... no? [17:12:24] I think Bob's going to say, "except on Mac OS/X"... :) [17:12:30] * bear_ snickers [17:12:54] But probably if it's a Mac OS-bundled library I imagine it would work. [17:13:11] I would hope, anyway. :) [17:13:38] that still would not help chandler, since it uses it's own python [17:13:49] depending on third party stuff to be installed is bad [17:13:53] in general [17:14:06] it might be a different version, compiled in a different way, etc. [17:14:09] * bear_ agrees [17:14:21] I was just raising the point to make sure the use case was fully thought out [17:14:25] On Windows, this isn't an issue. On Linux, it may require LD_LIBRARY_PATH to be set in advance. [17:14:37] On Mac, it requires header rewrites. [17:15:14] C'est la vie. [17:15:26] On the building side of things, you definitely want to be able to read Mach-O, PE, and ELF binaries to determine the dependencies for the developer [17:15:46] py2app reads Mach-O in pure python, pyexe does it with an extension, and I don't think anyone does it for ELF yet [17:16:09] ** gbay has left IRC ("It's not like I'm leaving or anything...") [17:16:54] for platforms that don't use any of those, they can't use plugins that embed libraries (OpenBSD, ...?) [17:17:11] * pje wonders how many libraries out there have dynamic dependencies [17:17:21] not many [17:17:38] libpython, libperl, libtcl, libruby, .. :) [17:17:58] By dynamic, I mean dynamically linking to their dependencies, as opposed to static linking them in. [17:18:14] oh, lots [17:18:27] almost all link dynamically. [17:18:59] subversion links to a godawful number of different things, many of which are its own [17:19:16] subversion, godawful... check. :) [17:19:31] (build process of subversion, I mean) [17:19:42] well I love subversion, but it's split up into lots of distinct components.. not a bad thing [17:19:46] Dunno if the software's any good, but building it sure is hell. [17:20:20] if you have BerkeleyDB installed it's not any harder to build than anything else [17:20:29] etrepum: thats because you don'y known arch ;) [17:20:34] for elf you can use ldd to show dependencies [17:20:46] etrepum: aka tla heheh [17:21:00] Cool. [17:21:05] I gave tla a try, I hated it [17:21:30] relly? I feel sorry for ya! :P [17:21:41] I feel sorry for *you* ;) [17:21:57] subversion works great, never had a problem with it.. been using it for years [17:22:14] * pje is a little uneasy at the prospect of trying to bundle "system" libraries with extensions [17:22:40] it really depends on what a "system" library is considered to be [17:22:43] you are a lucky, thats what I can tell ya! heheh [17:23:11] on Mac OS X a "system" library ships with the OS, and that's that [17:23:12] thank god they now have a fsfs heheh [17:23:49] Yeah, I'm thinking about Linux RPM-installed libraries. [17:23:53] on Windows I would imagine it's largely the same [17:24:02] Linux RPM-installed libraries is going to be a mess. [17:24:30] Windows doesn't have a lot of places to put libraries. [17:24:44] I'd rather not care about Linux and rpm/apt stuff [17:24:58] I'm talking about builder-side, Bob. [17:25:13] it's not that common to include dependencies with applications for Linux [17:25:15] I mean, if I build a library to distribute on Linux [17:26:29] help [17:26:36] Like say that I was writing an expat wrapper like pyexpat, and it dynamically links. [17:26:45] How on earth can I sanely distribute that as a plugin? [17:27:14] I guess you're right about needing to extract the link info from the executable. [17:27:27] So that you can at least document the dependencies in the plugin manifest. [17:27:28] it's hard if expat was installed by apt/rpm [17:27:44] if it was installed by hand to /usr/local you can take free reign and include it in the plugin [17:28:04] that's how py2app works, if it sees it in a vendor location it says no, otherwise it includes it [17:28:06] if it's installed by apt/rpm then it will be in a very small number of locations [17:28:25] yes but there's no way to distinguish between vendor and third party if itw as installed by apt/rpm [17:28:31] glibc looks the same as wx [17:28:57] * bear_ doesn't follow - what looks the same? [17:29:09] they're installed to the same place in the same way [17:29:19] apt/rpm make everything look vendor owned [17:29:23] ahh [17:29:42] you would either need a manifest of what you expect the OS to already have, or a developer-written manifest of what you want to include [17:29:44] you mean the fact that you most of the time do "sudo apt-get install" or "sudo rpm install" [17:30:05] beyond the fact you need to sudo, it's installed to /usr/lib just like glibc is [17:30:11] * bear_ nods [17:30:24] you don't know what is "system" and what is after-market [17:30:30] I get ya now [17:30:31] precisely [17:30:45] with OS X it is trivial, Apple owns /System and /usr (not /usr/local), and the rest is after-market [17:31:01] if the developer installs crap where they shouldn't put it, then it's their fault [17:31:04] you would have to assume all third-party if on linux [17:31:36] yes but you don't want to include absolutely everything in every plugin, because then you'd have lots of redundancy and probably conflicts [17:31:38] Well, there's libc to worry about. [17:31:40] since linux uses a flat namespace for symbols [17:31:46] I think it's last-loaded wins [17:31:52] would be painful at best [17:31:59] * bear_ nods [17:32:05] I assume anything that the plugin needs that Python already loads shouldn't be included. [17:32:12] you would need a pre-install check to find out what is required and download/install any gaps [17:32:40] pje: plus GNOME, KDE, .. ? [17:32:50] Ugh. [17:33:06] Perhaps there is a way to ask RPM what package a file belongs to [17:33:23] so that you can at least make a manifest of dependent RPMs [17:33:37] and make a spec file out of that (whatever it's called) so that you can use RPM to resolve those dependencies [17:33:43] Ugh, now it's pacakger dependent. [17:33:51] there is - but it will just show the RPM package name - not whether it was present on the OS or is third-party [17:33:52] it has to be for GNU SLAHS LINUZ [17:34:19] there is no way to make it work for Linux without being packager dependent or lots of explicit work by the developer [17:34:37] Fiddly diddly. [17:34:40] right - you could create a rpm spec dynamically that is part of the plugin install [17:34:50] Or as Dennis Miller would say, "Golly." :) [17:35:04] bear_: and an apt spec, and a pkgsrc spec, and (support other OS here..) [17:35:10] * bear_ nods [17:35:15] Okay, different idea. [17:35:30] for Windows and Mac OS X this isn't a problem (presumably) [17:35:35] Suppose on Linux you have a standard location to install libraries.... [17:35:45] (plugin-packaged libraries, I mean) [17:35:58] And then you can package them as RPMs or apts or whatever. [17:36:00] on windows you just have to worry about newer XP and it's dll loading "issues" [17:36:15] Which are what, bear? [17:36:38] it doesn't like to load a dll from the app path without being told to now [17:36:57] "told to" how? [17:37:24] you have to call a win32 api to enable the .exe dir as part of the dynamic library search path [17:37:43] Doesn't sound like that would be backward compatible. [17:37:52] it's not [17:38:03] Is this some sort of "security" kludge? [17:38:16] chandler uses a special Chandler.exe launcher to "prime the pump" so the subsequent python load works [17:38:30] yea - part of XP's new dll security IIRC [17:38:54] Have you reported this to python-dev? [17:39:14] don't know - that part of chandler was already present when I started helping with build issues [17:39:18] (i.e. does it affect normal Python dependency loading?) [17:39:19] I would have to ask [17:39:22] OK [17:39:36] I don't think so - I think it's a byproduct of chandler using a tweaked python :) [17:39:51] Ah, yet another such byproduct. :) [17:40:11] possibly to make sure that chandler loads the tweaked Python instead of another one [17:40:24] That would make sense. [17:40:31] (which is why you need to rewrite headers in OS X for deterministic behavior) [17:40:46] and also to ensure that chandler's dll's are loaded from the python process space [17:40:50] s/from/within/ [17:41:13] Distribution is a bitch, isn't it. [17:41:21] oh yea [17:41:24] amen [17:41:39] that is why I wrote py2app, because it does everything right for me [17:41:43] automatically [17:41:48] the linux issue can probably be solved by using exiting bdist_rpm [17:41:50] For Linux and similar Unixes, it seems that installing plugins in a standard location would work, using the native packager. [17:42:12] err s/exiting/existing/ [17:42:14] well, not bdist_rpm exactly, sort of a bdist_plugin_rpm [17:42:15] yeah using the standard packager is a great idea for those OS [17:42:41] but it is still a per-platform problem to write the dependencies in the spec files [17:42:58] whether the developer has to do it by hand or the plugin builder knows enough about each platform to do it [17:43:12] * bear_ nods [17:43:28] Or more precisely, packager vs. distutils [17:43:29] I'm sure the 80/20 rule can be applied - the major linux os's can be covered with rpm and apt [17:43:50] Because the developer might not be the packager; the packager might be building for another platform from the developer's source. [17:43:56] and anyone that cares about SuperKludgeix can write a patch [17:44:21] pje, but that packager probably has access to the rpms [17:44:21] LOL [17:44:29] What rpms? [17:44:43] ugh [17:45:02] that's a battle for the packager then to beat the developer bloody until they are available :) [17:45:07] I was talking about building from an sdist. [17:45:25] bear_, the standard distribution format for Python libraries is source. [17:45:41] You can't expect every library developer to own a machine of every platform. [17:46:01] Heck, I only have Win 98 and Linux myself. :) [17:46:13] * bear_ must be truly odd then [17:46:53] I think having time to hack is inversely proportional to finances. ;) [17:47:04] true [17:47:15] Because if you can afford to buy those machines, you must be doing actual work. :) [17:47:29] I just have a collection of boxes from previous work and/or jobs [17:47:38] that's what VMWare is for [17:47:41] I was a sysadmin in another life [17:48:29] I'm mad at vmware - it doesn't run on x86_64 FC3 yet :( [17:48:32] wasn't everyone a sysadmin at some point? [17:48:36] true [17:48:43] either technically or practically [17:49:14] most python hackers seem to at least sysadmin their own setups it seems [17:49:23] I have XP SP2, some old Linux, an OpenBSD, and lots of Macs [17:50:01] but I am friends with lots of sysadmins so if I wanted access to Irix or Slowaris I could get it [17:50:05] right now I'm running win2k, xp2, openbsd, FC3, debian and os/x [17:50:22] why do people do OS/X? [17:50:30] are you all past OS/2 users? :) [17:50:30] Eh? [17:50:39] the trademark is "Mac OS X" [17:50:45] Ah. :) [17:50:50] I thought that's how you typed it - no wonder google keeps complaining [17:51:21] Learn something new every day. :) [17:51:27] yep :) [17:51:55] ooo - 17:50 - I'm running late [17:51:59] So, to recap: OS X needs an install step, Linux needs global plugin installation, and Windows works unless it's XP. [17:52:13] it doesn't need an install step, it can be done at runtime [17:52:19] Okay. [17:52:20] it just needs some non-stdlib-machinery [17:52:29] pure python non-stdlib-machinery [17:52:34] Okay, so OS X needs a runtime. [17:52:39] windows may work except for really odd setups like chandler [17:52:52] I think that practically we should require a runtime, anyway [17:53:14] * bear_ nods [17:53:17] I know you think that, it's just so tempting not to on other platforms. :) [17:53:19] it lets us use an import hook if we want to, for example [17:53:35] it also allows for some sanity checks [17:53:43] which can actually "solve" the versioning problem [17:54:06] Only for pure Python, alas. [17:54:06] because it can do whatever it wants with sys.modules [17:54:16] true but that is the majority of plugins, I think [17:54:34] Plugins, si, libraries, no. [17:54:51] Y'all just got through saying that most libraries were dynamically linked. :) [17:55:07] yes [17:55:14] but libraries often do a good job with backwards compat [17:55:25] * bear_ nods [17:56:15] Maybe if you have a __main__ in your plugin, the system could automatically bundle the runtime for you. [17:56:52] no need for that.. if you have a .pyapp then you already have the runtime to run it on-click [17:56:55] Nah, I guess it can go in the bootstrap executable. [17:57:00] Heh. GMTA. [17:57:02] otherwise py2exe or py2app will put it in the bootstrap [17:57:09] Right, right. [17:57:31] And future stdlib would include it anyway. [17:57:34] precisely [17:58:12] So, we have to build a bdist_plugin, a plugin_runtime, and then there's the whole question of download/update site stuff. [17:58:12] I need to leave - is this channel logged? [17:58:17] * bear_ sees peak-logbot [17:58:23] yeah it's logged [17:58:23] Yes, see http://peak.telecommunity.com/irc-logs/ [17:58:36] Thanks for your input, bear! [17:58:42] Have a good one. [17:58:50] cool - nice meeting everyone and I'll have to set my home irc up to auto-log this channel [17:58:52] later [17:58:53] * bear_ waves [17:58:57] adios! [17:59:06] ** bear_ has left IRC ("Leaving") [17:59:52] Hm, Ted's been awful quiet. What happened to the whines he promised us? ;) [18:00:04] perhaps that's coming later? :) [18:00:20] Or maybe we addressed all his use cases. :) [18:00:31] one can only hope [18:00:33] Seriously, I believe he's actually been in meetings most all of this time. [18:00:43] I wouldn't be surprised [18:01:03] AFAIK, Bear's the one who works most directly on Chandler build stuff anyhow. [18:01:25] I'd love to start baking this plugin stuff into py2app, if for nothing else than internal use [18:01:46] so that py2app will use this plugin format to encapsulate its application (with libraries already extracted to the right place, of course) [18:02:05] Yeah, so maybe we can get started on the build part soon. [18:02:39] I'd still like to keep the distro format such that if it's pure Python you can just put it on sys.path; it'll help sell the inclusion of the format in distutils if its usable for lots of things. [18:02:54] IOW, the zip is in the layout used for import. [18:03:13] well we'll want a runtime API to load data files anyway [18:03:29] in such a way that it'll load them in or out of plugin [18:03:50] Oh, you mean ala __loader__.get_data? [18:03:53] yes [18:04:00] b/c the normal import mechanism has no __loader__ [18:04:04] That'd sure be nice. [18:04:33] PEAK of course falls back to __file__ in that case, but I can see that it would be nice not to have everybody write their own version of that. :) [18:04:47] I would immediately put that part of the runtime into pygame, providing I could find a way to convince py2exe to put the files in-zip [18:05:15] That actually sounds like something that should be in Python's "pkgutil" module. [18:05:19] The data loading, I mean. [18:05:26] it should be, but isn't, so we need to implement it [18:05:47] I meant, "should go there in future" [18:05:49] :) [18:05:52] yeah [18:06:59] Not a big deal, though. [18:07:12] Just define and document an API for it. [18:07:12] nah not at all [18:07:29] I would suggest having an API that requires you to pass __name__ [18:07:34] This plugin runtime also needs a way to request a plugin. [18:07:44] pkgloader.get_loader(__name__) [18:07:45] or something [18:08:05] and that object will have getdata(path) and getfilelike(path) [18:08:10] Sort of like config.packageFile(modulename,filename) [18:08:25] yeah [18:09:00] That function returns a stream factory, but this could be a stream or string I guess. [18:09:45] Other APIs, we need something like 'require(pluginName,version)' [18:09:47] well since nearly all packaged formats are going to give you a string natively it makes sense to use a string "natively" [18:10:03] Or perhaps 'available(pluginName,version)' to query. [18:10:14] *or* with an import hook you just say "import pluginName", with the required version as metadata [18:10:21] etrepum, not necessarily. Configuration files you might want as a stream. [18:10:45] Although I guess the point is that you get a string anyway with a zipfile. [18:10:59] yes that was my point [18:11:07] it useful to have both because some APIs want a stream [18:11:48] So what else is needed in the runtime? [18:12:04] I suppose that we need config for where to extract things to, if you want to override the defaults. [18:12:08] but it's silly to wrap it because StringIO's read() may return something other than getvalue() [18:12:43] why would you want to override? [18:12:52] tempfile should give you a temp location that you can write to [18:13:22] What if an app wants its own extract area to leave stuff extracted in? [18:13:34] (i.e. for faster startup) [18:13:50] but should that be a config file or something the application does before runtime? [18:14:00] how would the runtime know where to look for this config file if the app doesn't tell it? [18:14:02] No, not a config file. [18:14:09] oh ok [18:14:20] I'm saying that the runtime module would have an API to say, "setExtractLocation(options)" [18:14:31] So the app would set that before loading plugins. [18:14:33] with keep_temp=... [18:14:39] Yeah, something like that. [18:15:19] I'm thinking that'd also be useful for py2exe. [18:15:34] Not that I'm an expert on that either. :) [18:15:36] py2exe doesn't extract crap to a temp location, does it? [18:15:46] No. [18:16:05] In fact, I think Thomas wants to make it load extensions from the .zip; he has a prototype but it depends on GPL stuff [18:16:26] ah right, a re-implementation of the PE linker [18:16:30] However, before he discovered that possibility he was intrigued by the self-extract notioin. [18:16:36] er, notion. [18:16:55] the NSIS stuff will do the auto-extract.. for my win32 app that's what I do, extract the py2exe stuff to a temp dir and wipe it out after it's done [18:17:17] startup time sucks but it's an installer and isn't run often... [18:17:39] Right. Anyway, my point was that on any platform where you need to extract and want to cache, the option to request that would be handy. [18:17:48] yeah [18:18:08] I'm running out of other API needs for the runtime, though. [18:18:24] I'm sure there are probably OS X-specific additional APIs needed, though. :) [18:18:46] so we have [a] loading the plugin [b] loading data files [c] configuring how temporary stuff is extracted [18:18:47] Or more to the point, platform-specific startup modules. [18:18:58] Yeah. [18:19:07] [b] may actually depend on [c], if you ask for a *real file*, for a C API for example [18:19:23] Hm. Interesting point. [18:19:44] for example, pygame depends on a .ttf file loaded by a C API [18:19:50] Although, that means that extensions are actually a special case of b+c in that sense. [18:20:01] that is true [18:20:08] Ah yes. And icons or images might have that need as well. [18:20:38] Worse, some things might need a *filename*, not just a file. [18:21:10] So the file API should allow string, stream, or filename. [18:21:16] yeah [18:21:22] And in the filename case, it has to extract. [18:21:52] The stub loaders for extensions would basically call the file API to get their .pyd or .so file, I guess. [18:21:58] we also need a cleanup function [18:22:10] I'd rather not automatically assume atexit is safe to use [18:22:28] How do you mean? [18:22:41] if you don't care to cache [18:22:49] you write the files to a temp dir [18:23:01] how do you trash them afterwards without a cleanup func? [18:23:06] Doesn't tempfile API include an option for autodelete? [18:23:19] autodelete when the file-like object goes away, sure [18:23:26] but we're talking about C APIs here [18:24:10] No I was thinking about unlinking files... never mind. [18:24:19] Doesn't work on all platforms and definitely not with directories. [18:24:27] And the filename trick would be useless that way too. [18:24:34] and we'll need directories to prevent namespace conflicts [18:24:51] Why don't you want to use atexit? [18:25:20] it's not necessarily reliable [18:25:25] How so? [18:26:36] I'm pretty certain it's not compatible with some things [18:27:14] So, you think there should be a cleanup explicitly called by the app? Or the app can register that function w/atexit? [18:27:21] yes [18:27:30] the app should explicitly do one or the other [18:27:59] Hm. I don't see why the runtime shouldn't just register it, and allow it to be called twice. [18:29:11] maybe that would work, but then nobody would do it and get confused when it doesn't work [18:29:41] let's say that 5% of the time it doesn't work, 95% of the time people won't be aware it even exists... and then for that 5%, they will have no idea how to fix it [18:29:51] but if they're aware that this needs to happen, they will know where to look [18:30:12] Sigh. [18:30:26] You're probably right, I just wish you weren't. :) [18:30:54] I personally don't know that atexit is so unreliable, short of a core dump. [18:31:04] well we're already requiring say, 5 lines of code to use plugins from an application, what's two more? [18:31:15] Yeah, yeah, yeah. :) [18:31:33] I'm still thinking about trying to use it for non-plugin libraries in regular Python. [18:31:57] in that case you can assume you want a persistent cache somewhere [18:32:06] and you don't need to call a cleanup function [18:32:36] Hrm. Does that mean we should default to a persistent cache in site-packages, unless it's not writable? [18:33:18] wouldn't it make more sense to have a ~/.pyplugin or something? [18:33:28] most of the time site-packages will not be writable [18:33:33] Good point. [18:33:49] hopefully it isn't most of the time, anyway :) [18:33:53] for a system install [18:33:58] Still needs to fall back to temporary if ~/.pyplugin isn't available. [18:33:59] except on Windows, of course [18:34:08] Yeah. :) [18:34:26] We'll need to write this up and float it for others to pick holes in, too. [18:34:42] we should force them to have a config file or env var if ~/.pyplugin isn't writable [18:35:10] So, we only go to temp files if explicitly enabled? [18:35:18] yes [18:35:35] Thus avoiding the cleanup issue. [18:35:43] for a normal developer-installation you will want persistent files.. otherwise you will know what you are doing [18:36:18] well if you ask for temp files you will need to clean them up, yeah [18:36:38] Speaking of which, how will we will tell whether a "cached" extract is up-to-date? [18:36:54] mtime of the dir >= mtime of the plugin [18:37:05] Of the directory? [18:37:14] yeah each plugin gets its own directory [18:37:26] (or more if it has subpackages [18:37:28] Hm. Okay. [18:37:47] Can you set the directory mtime? I'm thinking it should be exact match, not >=. [18:38:05] Otherwise clock skew can be weird, like on Windows where mtime is localtime rather than gmtime internally. [18:38:23] OTOH, not sure you can set directory mtime on Windows. [18:38:33] well what about naming the directories with the version? [18:38:52] Doesn't help during development; what if I build 6 versions of PEAK-0.5a3? :) [18:39:17] the package build command could clean your cache? [18:39:30] I guess that could work. [18:39:46] I was originally thinking checking individual file date+size [18:39:49] you could set an env var that doesn't use the cache at all? [18:40:11] But was thinking it was a problem for OS X since you'd be rewriting the .so's... or do they stay same length? [18:40:22] they stay the same length [18:40:51] In that case, set their mtime based on the info from the zip, and compare size+date when reusing a cached extract. [18:41:07] Then we don't have to special case anything. [18:41:10] could be done that way [18:41:26] You have to stat to make sure it's been extracted anyway. [18:41:35] maybe we could also let plugins have "cookies" [18:41:50] Hm? [18:41:53] a little place to store persistent info from the user [18:42:01] Oh. [18:42:23] That's not for version sync, though, right? You mean for configuration of the plugin. [18:42:32] yeah [18:42:46] but also included in the runtime API [18:42:49] Dunno, that seems like something the app should do. [18:43:22] you think? the app would set its ".pyplugin" dir, that would make cookies app-specific [18:43:24] Because the app would presumably want to keep that data with *its* configuraiton. [18:44:05] but plugins still need a way to ask the application to store data for them [18:44:19] I'm just having a hard time imagining a use case that isn't application-specific to begin with. [18:44:55] Even OSGi doesn't supply something like that. [18:44:58] configuration for a python repl? [18:45:03] repl? [18:45:19] read-eval-print-loop [18:45:33] For it to do what? [18:45:37] like if you have an embedded interpreter and the user has the IPython plugin [18:45:44] and wants to setup their IPython in some way [18:45:52] doesn't it have dot files for that? [18:46:29] yeah but if I have a MATLAB-alike application and a Python IDE I may want different settings [18:46:36] I suppose we could allow it space in the plugin directory for that kind of thing, but what if you're using a temp dir? [18:47:08] then it could raise or return None if the plugin asks for a persistent location [18:47:12] I guess we could have an API for the app to publish a base directory for plugin configuration. [18:47:48] I know jEdit plugins use that approach for their config. Eclipse's is a bit more sophisticated because of more complex inter-plugin communication requirements. [18:48:29] Gr. [18:48:44] There's a problem, though... plugins for systems like Zope need to be instantiable. [18:49:00] You really *don't* want them to have an app-wide configuration. [18:49:25] well those plugins won't use that mechanism [18:49:34] Yeah, you wish. :) [18:49:51] they will store their configuration *in* the instances which will be persisted by Zope [18:50:09] If Zope had had something like this, there would have been lots of badly-written plugins because the author found it easier to use plugin configuration. :) [18:51:19] Still, I guess we could put lots of warnings in. :) [18:52:03] it's really Zope's problem to say "use our persistence" [18:52:15] Yeah. [18:52:50] I think I see some problems with this rather utopian API, though. [18:52:56] ** bear has joined us [18:53:20] Like if we support multiple versions being loaded at once, how do we know which one is "current" when we want to hand out a configuration or a data file? [18:53:40] It seems we might need to have the actual *module*, not just its __name__ [18:53:49] Hey bear. [18:53:54] the __name__ would be exactly what is in sys.modules [18:54:05] the loader sets __name__, not the bytecode [18:54:08] Oh, right. [18:54:28] You just can't ask for *another* module's stuff. [18:54:40] Unless you import it and then use its __name__. [18:54:44] yeah [18:54:48] which is reasonable [18:55:11] Hm. Sorta. PEAK's pkgfile: URLs are typically being used somewhere generic. [18:55:15] the search algorithm could walk up the package tree when you ask for a file [18:55:21] I mean, they can't be plugin specific. [18:55:44] well I have to run, we can continue this tomorrow if you're around [18:55:54] Sure. Good talking to you! [18:55:58] * pje waves [18:56:23] good talking to you too, I think we've specified a lot today :) [18:56:25] We really made pretty good progress today. [18:56:33] Heh. GMTA. [18:56:39] Have a good one. [18:57:31] FYI Bear, we've sketched the plugin runtime API a bit, still lots of details to fill in, but lots of policy solidified. [18:57:57] (Default policies overrideable by applications, that is.) [18:58:05] cool [18:58:38] By default, libs and data files extract and cache to user's ~/.pyplugin directory, w/fail if not writable. [18:58:47] Apps can change the location [18:59:04] * bear nods [18:59:06] And also specify that a temporary directory is used, in which case the app has to call a cleanup function on shutdown. [18:59:47] We've also specified how to deal with files like icons, images, fonts, etc. that need to be read by C code. [19:00:04] (And therefore also have to be extracted to the cache or tempdir) [19:00:19] right - resource files [19:00:52] So far, the only really complicated part is OS X header munging, which Bob has already written lots of stuff for. :) [19:01:10] yea - that seems like a huge issue to us but not to bob :) [19:01:46] It might be really cool if we could also have the plugin runtime API know how to locate/download/install dependencies. [19:02:05] Because then I can imagine a Chandler application startup that just says something like: [19:02:10] import pyplugin [19:02:25] pyplugin.require("wxPython-X.XX") [19:02:28] etc. [19:02:32] * bear nods [19:02:32] Heh. [19:02:51] for chandler I would imagine it would be an osaf parcel - that way it can know and use all the required ACL items [19:03:34] Well, presumably the bootstrap code I'm talking about also needs to install the plugins needed to run the repository, like PyLucene et al. [19:03:48] So it can't depend on the repository. :) [19:04:06] ah - your already planning for when chandler doesn't have a tweaked python [19:04:44] ohh - I mixed python plugin with chandler plugin - sorry [19:04:48] yea - that makes sense [19:05:25] A parcel is a subset of "plugin" here. [19:05:54] A parcel might actually depend on non-parcel plugins, too. [19:06:16] true - but a parcel lives in the context of the repository - so that will always be available to it [19:06:16] Plugin isn't really a great word for this. [19:06:27] Right. [19:07:02] * pje yawns and stretches [19:07:09] I think I've been at this too long for today. [19:07:16] :) [19:07:28] But we made some really good progress, I think. [19:07:51] * bear does also [19:08:08] Did you get a chance to look at that sample distutils command? [19:08:29] the unit test one - yes [19:09:15] very straight forward - I feel I need to do a mental shift to fully get into the distutils mindset [19:09:25] I've been living in the rpm/apt world for too long [19:09:37] * pje nods [19:10:30] I think we'll probably end up just wrapping some of hardhatlib stuff as distutils commands [19:10:51] well.... [19:10:59] And dropping the parts that are duplicated in distutils, as distutils includes tons of file/directory utilities that support verbose/quiet, dry-run mode, etc. [19:11:02] most of what hardhat lib does is already wrapping things [19:11:46] the reason hardhat has been used, IMO, is that for all three platforms the current install method is "unpack and run" [19:12:03] now that I've done installers for linux and win32, that needs to be changed [19:12:34] Hm, we should probably take this part back to #chandler [19:12:52] :) [19:13:07] Now that we're done buttonholing Bob for today. :) [19:13:13] LOL [19:13:13] yea [19:14:16] You should probably change your nick over there to not be bear_afk any more. :) [19:40:16] ** debugger has left IRC () [20:05:06] ** vlado__ has joined us [20:12:37] ** vlado_ has left IRC (Read error: 60 (Operation timed out)) [20:17:37] * pje heads off to get food [20:17:40] ** pje has left IRC ("Client exiting") [21:50:18] ** sprout has left IRC ("Snak 4.13 IRC For Mac - http://www.snak.com") [22:18:27] I'm back