[00:44:59] ** hazmat has left IRC ("Leaving") [02:28:37] ** vlado|away has left IRC (Read error: 54 (Connection reset by peer)) [02:29:05] ** vlado|away has joined us [02:45:44] vlado|away is now known as vlado [04:02:30] [connected at Fri Apr 1 04:02:30 2005] [04:02:30] <> *** Looking up your hostname... [04:02:30] <> *** Checking ident [04:02:30] <> *** Found your hostname [04:03:01] <> *** No identd (auth) response [04:03:01] <> *** Your host is orwell.freenode.net[orwell.freenode.net/6667], running version dancer-ircd-1.0.35 [04:03:01] [I have joined #peak] [04:03:01] ** orwell.freenode.net set the topic to http://dirtsimple.org/2004/11/generic-functions-have-landed.html [04:34:38] ** apoirier has joined us [06:00:16] ** vlado_ has joined us [06:03:27] ** debugger has joined us [06:15:38] ** vlado has left IRC (Read error: 110 (Connection timed out)) [07:57:51] ** [apoirier] has joined us [07:59:50] ** apoirier has left IRC (Read error: 110 (Connection timed out)) [08:21:36] ** debugger has left IRC () [10:48:35] ** vlado_ has left IRC ("Leaving") [12:02:27] ** [apoirier] has left IRC ("Using KVIrc 3.0.1") [12:11:20] bear_afk is now known as bear [15:35:48] ** bear has left IRC (Read error: 110 (Connection timed out)) [15:46:57] ** jemfinch has joined us [16:02:04] does peak have any command-line parsing tools? [16:11:45] ** bear has joined us [18:30:37] bear: are you code-bear? [18:57:10] ** pje has joined us [18:57:21] Hey etrepum [18:57:28] hey [18:57:43] I'm working on eggs, and had a question for you [18:57:48] ok [18:57:48] Or at least an idea to bounce off of you [18:58:14] It's starting to look like paths are the right way to deal with import/load/distro etc. [18:58:35] To the point that "Distribution" is just a thing with a name, version, and path, almost. [18:58:55] That just delegates to an importer for that path [18:59:05] Or an adapter for the importer, I should say. [18:59:07] a "path hook" if you will :) [18:59:18] Pretty much, yeah. [18:59:37] I found a problem with baskets, pth files won't load them [18:59:42] http://python.org/sf/1174614 [19:00:02] The only downside I'm seeing is that I'm not sure if it locks you into using only paths that can be processed w/os.path [19:00:49] basically paths where normalizing / and \\ to one or the other is a lossless operation? [19:01:17] Nah, I mainly mean os.path.join to add pieces at the end [19:01:52] I'm not sure there are any situations where we have to walk *up* the path [19:02:01] hey pje, bob [19:02:22] Although the current code does, in order to find a .egg path. If I refactor, that might not be needed because we could always walk down instead. [19:02:26] pje: should I avoid implementing new stuff with p.storage and p.model? or is that refactoring still far enough off? [19:02:43] arg, it's still pretty far. [19:02:56] well if we use normpath, you can always walk back up [19:03:08] ** jemfinch has left IRC (Remote closed the connection) [19:03:14] ok good.. i told bob converting to schema would probably be fun anyways, so i'll plow ahead with stoage/model [19:03:28] arg, I might start on peak.schema soon, but it'll take a while before there's anything usably "real" [19:03:29] I think the only sane things to do are standardize on / as a separator, or use normpath form (while also taking /) [19:03:47] normpath form is probably best, because __file__ is already in normpath form [19:03:54] Hm. [19:04:04] i need to talk to you guys at some point about a common deployment model for python web apps [19:04:08] like eggs + wsgi [19:04:15] for on-demand python appservers [19:04:35] Rather than deal with low-level aspects of the path syntax, I'd like to look at how the high-level model works if we use paths [19:04:36] i wonder if there's a shot with egg yolk and whiskey or something ridiculous like that [19:04:56] ewww [19:05:13] haha, ever had a cement mixer? [19:05:14] egg nog I guess [19:05:21] no I haven't [19:05:22] traditional 21st bday gag drink [19:05:30] lemon juice and baileys... [19:05:37] curdle-mania [19:06:31] wsgi + eggs is so much more lightweight than JSP plus .war files [19:06:39] eventually im going to pitch this to work [19:07:00] eggnog isn't that great of a name, unfortunately [19:07:23] Y'know, I think the path joining part is irrelevant, now that I think about it. [19:07:43] The real issue with the approach I have in mind is that it assumes a symmetric importer/loader [19:08:11] well all importer/loaders that I know of are symmetric [19:08:28] Yeah. [19:08:31] but I only know of zipimport, PEP examples, and my experiments [19:08:41] It's just not strictly PEP 302 compatible to assume it'll be the case. [19:09:00] Of course, we're going to fail for any "unknown" importer types anyhow [19:09:27] The only really generic way I can think of to make this work (for namespace packages) [19:09:45] is if you ask each loader to load_module, and save the __path__ in between [19:10:03] since modules can't really be unloaded cleanly, there is no reason to separate importer/loader [19:10:04] Then combine them at the end [19:10:15] because you have to preserve the full path in __file__ anyway [19:10:46] Yeah, but then __file__ has to have os.path.dirname work on it if you need to go "up" [19:11:21] but you don't need to go up.. the __loader__ will know its "prefix" and can [len(prefix):] [19:13:14] I'm just thinking it could be pretty darn 302-compatible if we actually load the modules... [19:13:25] It also merges their contents, FWIW. [19:13:42] well it can be perfectly 302 compatible, because we're implementing the loader/importer pair :) [19:14:13] except for zipimport, which we can assume stuff about [19:14:14] I mean namespace_package() would be compatible with *any* PEP 302 importer/loader that can work from paths [19:14:21] oh I see [19:14:24] The paths don't have to follow a particular syntax [19:14:45] is there a reason to "go up"? [19:14:46] As long as it makes __path__ entries that are meaningful to its own path hook [19:14:55] That's what I'm trying to figure out. [19:15:08] In order to iter_distributions, you only need to go down [19:15:28] I can't think of any import mechanism that ever looks up either [19:15:49] But, if you do a resource_filename() on a zip-loaded thing, it needs to find its metadata. [19:16:40] It would kind of suck to have to re-walk sys.path (which might have changed) in order to find the egg and metadata [19:17:03] OTOH, that's adapter-specific code, so I suppose it's okay to walk up in that scenario. [19:17:17] Non-egg loaders are on their own there. [19:18:03] Okay, I think that takes care of it; namespace packages can be implemented using only PEP 302 features, as long as it's done in a slightly kludgy way. :) [19:18:13] won't the loader be looking at EGG-INFO when it gets started up? [19:19:00] well on iter_distributions anyway [19:21:16] Actually, that brings me to the second point [19:21:51] Which is that I think the EGG-INFO idea for unpacked distros and development may have some issues. [19:22:13] For example, you can't have something in a directory that provides more than one egg [19:22:29] And where do you create EGG-INFO exactly? [19:22:41] vs. EGG-INFO.in, for example [19:22:55] E.g. in a source distro with a 'src' subdirectory [19:23:10] so then what? [19:23:17] I've been thinking about maybe just having a "provides.txt" in the directory that's on the path... [19:23:26] That lists distro names and versions provided [19:23:38] Which means you could stick one in site-packages and one in Lib, for example. [19:24:00] why not have EGG-INFO/eggname/stuff ? [19:24:13] Ease of use, mainly. [19:24:29] But also our current convention is eggname/EGG-INFO/stuff anyway [19:24:51] yes but you're talking about slopping these things all over the place and making it more ambiguous [19:25:11] I could see calling it maybe "eggs-provided.txt" or "eggs-installed-here" or something [19:25:25] but then you have the mess of updating a text file in-place [19:25:29] This is strictly a development thing, for paths that *don't* end in .egg [19:25:41] This isn't for unpacked eggs, IOW [19:26:11] hmm [19:26:43] the filename should probably denote the fact that you should only use it for development [19:26:53] faked-egg-info.txt or something [19:27:23] Yeah, that'd be an okay name even for marking the stdlib as installed. [19:28:03] OTOH, I suppose an alternate way to do it might be just having some .fake-egg files [19:28:40] That is, for each .fake-egg file you have, treat it as a distribution with no metadata and its home in that directory. [19:30:10] So, in my PEAK source distribution, i'd put a bunch of files like 'PEAK_Events-0.5a4.fake-egg' in the 'src' dir [19:30:11] I dunno about dot files, seems like it would be more annoying than helpful to hide them most of the time [19:30:21] ah that's an extension [19:30:23] makes sense [19:30:31] So that if 'src' is put on sys.path, then it's as if the eggs are installed [19:30:45] Hm, maybe .installed-egg would be better, though. [19:31:05] .filesystem-egg [19:31:12] Ah, better. [19:31:14] since it would be fundamentally incompatible with anything PEP 302 [19:31:19] Right. [19:31:36] I suppose we could let it just be a directory, optionally with metadata. [19:31:49] .egg-shell? ;) [19:31:56] haha [19:32:13] CVS users might hate versioned dirs in their source dist [19:32:48] You'd rather we make the file a shell archive? ;) [19:33:02] maybe take the version out [19:33:21] Ah, right. [19:33:29] I thought you meant to get rid of the directory [19:33:32] nah [19:33:38] CVS just hates directories [19:33:44] I don't really use CVS anymore for that reason :) [19:33:56] So I'd make dirs like PEAK_Events.egg-info [19:33:57] that and its crappy performance [19:34:00] yeah [19:34:23] And for those, you'd get the version info from PKG-INFO inside of there. [19:34:31] or a versionless .egg-info directory always sorts newest [19:34:59] I don't see why we would need to support multiple half-baked eggs of the same thing [19:35:14] Half-baked egg? [19:35:15] on the sys.path, at the same time [19:35:36] Dunno, I'm kind of torn on that point myself. [19:35:39] since they are subject to sys.path anyway [19:35:56] I mean, not the multiples on sys.path, but rather on whether you should be required to have version info [19:36:03] ah [19:36:40] Part of it is I'm not sure what all metadata you want in there. [19:36:51] Like, do you want requires/provides stuff? [19:37:13] I also wonder if bdist_egg should read the info from this? [19:37:29] That is, should this replace EGG-INFO.in on the build side? [19:37:37] possibly [19:37:46] It'd certainly be handy for building multiple eggs from the same setup.py [19:37:58] As would be desired for PEAK and other large libs [19:38:31] that might work [19:38:47] So bdist_egg could look for 'distroname.egg-info' in the '' package directory, as the default egg-info location [19:39:36] Cool; that should then work for all manually-created metadata [19:39:46] seems pretty sane [19:40:27] Probably bdist_egg should write its autogen files to the .egg-info directory, so that the distribution can then be run and tested in-place [19:41:16] Interestingly, if you do your bdist_egg without any metadata, it would end up creating the metadata directory for you [19:41:53] ISTM it should also add the egg-info dir to your MANIFEST.in, but I'm not sure how safely that can be done. [19:42:40] nothing else in distutils tries to muck around with MANIFEST.in [19:42:54] it could hook into the MANIFEST generation for sdist maybe [19:43:24] I wish they *would*; it's annoying as hell to have to do it yourself when the tools already *know* all the files involved. [19:43:59] 80-90% of my distribution build errors are due to stuff being left out of MANIFEST.in that the system should just bloody include on its own. [19:44:34] yeah MANIFEST.in is dumb [19:45:41] On the whole, it probably makes more sense to just tarball up your CVS/svn tree and ship it as the source distribution. [19:46:01] PyObjC just grafts all of its dirs, then trashes temp files [19:46:23] build dist *~.nib .svn [19:47:38] Yeah, I should probably just do cvs export to a named dir and package it up; the only thing distutils adds is a PKG-INFO, which isn't very useful anyway [19:47:55] since it doesn't get f*ing installed anywhere [19:48:16] Is that a wildcard? :) [19:48:30] just too lazy to type it out :) [19:49:25] http://svn.red-bean.com/pyobjc/trunk/pyobjc/setup-lib/di_sdist.py [19:49:33] Okay, so I need to fix bdist_egg to look for and create a .egg-info dir [20:13:38] Hm, Distutils doesn't do anything about '-' in names [20:13:49] Or versions, for that matter [20:13:53] what do you mean? [20:14:43] Meaning you can name a package 'setup-tools' and the - stays in [20:15:02] So I'm changing bdist_egg to convert to '_' for purposes of naming the archive file [20:15:32] I also need to make pkg_resources convert '-' to '_' when parsing requirement names and versions. :( [20:16:06] I'm not sure how this'll affect PyPI, though. [20:17:26] uh [20:17:34] tell people not to do that? [20:17:40] Yeah. :) [20:17:56] I'm not sure why it matters if there is a - in the name [20:18:03] it's not a package with an __init__ [20:18:07] Actually, if PyPI just stores it internally with '-' changed to '_', it'll be fine [20:18:16] No, the archive name needs to be parseable [20:18:29] Unless we're always going to read PKG-INFO, which I guess we could. [20:18:30] ah yes [20:19:36] I should probably just replace all non-alphanumerics in the name and version with '_' [20:19:54] Some fool might be putting spaces in there. ;) [20:20:14] well spaces don't get in the way [20:20:21] just - [20:20:21] True. [20:20:39] But it's easier on the parser to match \w+ :) [20:20:56] also true [20:21:18] Alas, the version info has to be (\w|\.)+ [20:21:28] But I guess I could standardize on the latter. [20:21:52] And make bdist_egg barf if you have an unworkable version name [20:21:59] yeah [20:22:30] Hmm... I could just use pkg_resources.parse_requirement("%s==%s" % name,version) [20:22:58] That way, I'm not double-coding the check. [20:23:09] I don't know what that means [20:23:20] name == version? [20:23:26] Yes [20:23:43] I'm saying bdist_egg should try to parse the spec for the thing it's creating [20:23:47] oh ok [20:23:50] that makes sense [20:23:53] To make sure you have a sane distro name/version [20:24:33] yeah [20:29:10] You can still fool it, though. I put a version of '0.0,>1' and it tried to build it anyway. :) [20:29:46] But I guess if you use version numbers like that, you deserve whatever happens. [20:29:50] that is a "valid" LooseVersion [20:29:52] :) [20:30:04] (but what isn't) [20:30:24] True, but currently the requirement syntax only allows \w|\. in names and versions [20:32:23] Okay, so bdist_egg now supports making and using a distName.egg-info directory [20:32:35] excellent [20:32:43] And it even removes native_libs.txt if it exists but there are no extensions. [20:33:39] Now I've got to figure out how to get the runtime to work the same way. ;) [20:34:22] I'm thinking that we'll have Distribution(name,version,path,metadir) objects [20:34:37] Where metadir is a thing that lets the distribution retrieve its metadata. [20:35:08] So, for a path with .egg-info subdirs, you'll create Distribution(name,parentdir,egginfodir) [20:35:16] Or something like that [20:35:47] makes sense [20:36:06] I suppose we could actually make name and version optional and get them from PKG-INFO [20:36:37] So that the only things you really need are a path entry and a metadata source [20:37:53] But then that means we have to read PKG-INFO to do iter_distribution, so I'm not sure how good an idea it is. [20:39:47] OTOH, there's no way to escape it where .egg-info versions are concerned [20:40:48] yeah, eggs in general are going to require a lot of stats and listdirs anyway [20:41:12] Baskets don't, but they hardly count. :) [20:43:22] yeah [20:43:24] Hm. Well, the only problematic case is for .egg-info dirs [20:43:33] Which are mainly for development and such. [20:44:00] And if the version attribute is lazy, and requirements objects check name before version, we can avoid loading it unless we need it [20:46:18] Hm. require() could probably be simplified by having it just create a dictionary out of iter_distribution() [20:46:26] mapping name->distros [20:47:00] yeah it could do that internally [20:47:07] That's what I mean [20:47:17] It should also ask the distros what *they* require [20:47:40] And loop until there are no unresolved requirements. [20:49:10] I'm also thinking to move iter_distributions and require out of the ResourceManager class, since they don't have anything to do with resources. [20:49:20] that makes sense [20:50:50] Hm, actually we need to track which distributions are *already* on sys.path when you do a require() [20:51:38] And the name->distros map would in such cases reduce to just that one item, before trying to apply requirements. [20:52:17] Or more simply, if you are about to reject an installed distro for not meeting requirements, then that's a fatal error [20:53:15] Hm. Somehow this all seemed so much easier when we first talked about it. :) [20:55:25] well, we can punt on a lot of the conflict resolution crap [20:55:43] cause it improves things like 20x before that problem is solved [20:55:50] Yeah, it's just that require() was a nice simple routine previously. :) [20:56:10] well it can be nice and simple, it doesn't have to work completely yet :) [21:04:44] * pje sighs [21:16:26] Well, I've got a draft now, but it's filled with XXX's. [21:17:48] Did we decide at one point to just issue a warning if you get a version conflict? [21:18:18] I don't think so [21:18:25] Okay. [21:18:35] I guess a warning makes sense [21:18:42] make it a subclass of ImportError [21:18:57] I think you have to derive from Warning, though [21:19:05] I don't think so [21:19:21] Hm. I dunno. [21:19:26] I think I'll stick with an error [21:19:44] Because normally, people won't be putting version requirements unless they actually *need* them. [21:19:48] ugh it looks like it does care [21:20:13] you can always multiply inherit from warning and importerror :) [21:21:39] Okay, so the algorithm is like this... [21:21:59] Go through a set of requirements, adding them to a table as we go, along with the source (e.g. 'foo' required 'bar>1.2') [21:22:26] For all distributions of the correct name, check to see if they match, and if not, eliminate them from the overall map [21:22:43] However, if there's an installed distribution for that name, first check to see if the version check fails [21:22:54] If so, then raise a VersionConflict error [21:23:20] Anyway, if no distribution matches, raise a DistributionNotFound error [21:23:36] Otherwise continue looping over requirements [21:23:59] ok [21:24:16] Then, when all requirements are exhausted, go back to the requirements table and loop over it... [21:24:31] And look for the "best" distribution in the name->distros map [21:24:34] and the VersionConflict error can tell you about what you have and why it doesn't match? [21:24:46] Yeah. [21:25:05] another thing to consider is modules already on sys.path [21:25:15] Anyway, you then install the "best" distribution for each requirement, and add it to a map of nested requirements [21:25:35] distro.installed() will check if it's on sys.path already [21:26:01] say you have Foo 1.0 and Foo 2.0. the FooBar package inside Foo 1.0 is already in sys.modules [21:26:29] but pkg_resources thinks Foo 2.0 is a better distribution, so it puts it early on sys.path, and *boom* [21:26:35] Then Foo 1.0 must already be on sys.path [21:27:11] Once you've installed all the "best" distros, you then recurse for each one's requirements [21:27:30] so having it be on sys.path is important enough to have it blow up if it's not the best, even if nothing from it is loaded? [21:27:48] That's the way I'm treating it at the moment, yes. [21:28:04] If it's on sys.path, you explicitly put it there already. [21:28:10] that might be an issue with the .egg-info stuff maybe [21:28:32] Hm. [21:28:50] I don't think we can walk sys.modules for this [21:28:57] It's already hellishly complicated as it is. [21:29:01] true [21:30:01] Ugh; even worse, what if *multiples* of one egg are on the path already? Eeyagh. [21:30:30] I guess we have to treat the first one as the best one. [21:30:44] Yuck, yuck, yuck. [21:30:45] yeah [21:30:56] maybe it should warn? [21:31:18] No, because if you want to replace a stdlib module that's arguably the right way to do it. [21:31:28] there is no right way to replace a stdlib module [21:31:37] Heh. [21:31:45] I mean, with an upgraded version [21:31:51] even still, there is no right way [21:35:05] Hm, the nested requirements check needs to be depth-first, so you never add anything to the path without *all* its dependencies being resolved first. [21:35:53] Unfortunately, this algorithm as a whole is actually non-deterministic. [21:36:12] So in principle, if the hash order changes you can end up with a conflict where you had none before, or vice versa. [21:36:55] In practice, I'm hoping things are simpler than that. :) [21:38:00] maybe it should sort to make it deterministic? [21:38:16] Perhaps. [21:38:24] seems reasonable, sorting is fast anyway [21:38:26] Then at least you'll always fail or always succed. [21:38:41] succeed, even [21:38:46] site.py sorts when it's doing .pth files [21:38:54] for the same reason [21:43:26] * pje checks in a current draft [21:43:45] Btw, please don't PEP 8-ify everything again just yet [21:44:11] I don't mind if you break my 41-line paging, but I'd prefer you didn't remove it until we're a lot closer to having this stdlib-ready [21:44:59] the 41 line stuff makes it a lot harder for me to read [21:45:02] but ok [21:45:20] Then size your window to 41 lines. :) [21:45:40] well I don't scroll in page sized increments most of the time [21:45:44] Ah. [21:46:09] You don't have a next-paragraph scroll? [21:46:28] I *could*, but that's just not what I normally do [21:46:32] Ah. [21:46:34] Okay. [21:46:43] We all have our quirks, I guess. [21:46:48] mine are standard :) [21:47:10] I'm just asking you not to go and hunt 'em down in places you aren't actively touching. [21:47:14] yeah [21:47:16] that's fine [21:48:26] * pje doesn't care for PEP 8, obviously [21:48:34] everyone else does [21:48:46] for the most part [21:48:52] Heh. You wish. [21:48:53] greg stein writes horrible 2-spaced code :) [21:49:03] well all the code I work with, except yours, is PEP 8 [21:49:16] or close enough to it not to notice [21:49:33] I still wonder why spaces have to be around every piece of punctuation. [21:49:40] It's punctuation, for goodness' sake. [21:50:18] well, in English there's space around at least one side of most punctuation :) [21:50:26] My other pet peeve is the indenting of parentheticals... where if you have something like: [21:50:36] veryverylongnamethinghere(somethingelselong, [21:50:44] ...etc. [21:51:02] Then indenting all the way to the previous parenthesis just means you get rapidly diminishing space in which to say anything. [21:51:12] that's not PEP 8 though [21:51:38] that does bother me, when people try to line stuff up that doesn't follow sane indentation rules [21:53:01] Hm, PEP 8 says "Make [21:53:01] sure to indent the continued line appropriately. Emacs Python-mode does this right. Some examples:" [21:53:21] It then follows with several examples of indent-to-the-paren [21:53:57] that is bizarre [21:54:27] And when Fred Drake reformatted other setuptools stuff, he mostly used that hideous form. [21:54:34] I could've sworn that wasn't in there.. I don't see much code that looks like that [21:54:47] yeah that form is terrible [21:55:27] Too bad jEdit and PFE don't support treating ^L as a page break, since then I could replace my multi-blank-lines with them. [21:55:40] yeah [21:55:45] you could patch jEdit :) [21:58:45] Tell you what, if the extra whitespace bothers you, replace it with ^L's and I'll write a macro to change them back. ;) [21:59:04] haha [21:59:13] ** jemfinch has joined us [21:59:19] I'll live [21:59:28] OTOH, maybe I should write a macro to replace the whitespace with ^L's, and then I can be more-or-less PEP 8 compliant. [21:59:37] yay, people are here! [21:59:49] it's only really a problem when I'm trying to digest large chunks of PEAK code to see how it works [21:59:53] Hm, I might be able to create some kind of an editor hook that replaces the ^L's on load, and puts them back on save. :) [22:00:04] I had a few questions I wanted to ask about PEAK in general. [22:00:16] like when I was writing that dispatch stuff I had to read a lot of code to figure out what I needed to do [22:00:18] etrepum, if you use Eclipse or another Python-smart editor you can use the code outline to browse. [22:00:25] I don't [22:00:36] well there is a code folding thing for Vim, but I don't like it [22:00:36] What editor do you use, btw [22:00:42] jemfinch, go ahead [22:00:46] etrepum: I very much dislike code folding. [22:01:19] the Java editors I can't stand, and I haven't retrained my fingers for emacs keys yet, so I'm more or less stuck with Vim [22:01:44] once I switch to emacs, I'll have a choice, cause Mac OS X in general uses Emacs-ish key bindings for all text editors [22:01:55] because it's part of the text handling system of the OS [22:01:56] pje: Mostly what I'm curious about is what kinds of projects PEAK would be useful for, and what kinds of projects it wouldn't be so useful for. It seems like a big piece of software, and a significant investment in time to learn, and I'd like to know what sorts of projects would see a significant ROI. [22:02:10] etrepum: I use emacs in viper-mode, fwiw. [22:02:29] emacs in viper mode doesn't seem that useful to me [22:03:02] etrepum: it doesn't have all the vim-specific stuff, but I never learned vim-specific stuff, so I don't really feel like I'm missing much. [22:03:09] it's definitely easier to edit files than pure emacs, IMO. [22:03:33] (but then again, you can use both emacs keybindings and vi keybindings, so it's like the best...or at least some "pretty good stuff" of both worlds) [22:04:05] I haven't seen a really compelling reason to switch from vim to emacs if I'm going to stick with almost-vim keys [22:04:34] the python-mode alone is enough for me. [22:04:52] the biggest reason I want to switch to emacs is simply to train my fingers, because the rest of Mac OS X has emacs-like keybindings [22:05:09] etrepum: except when websites override them with stupid accesskeys. [22:05:16] alternatively I could write an InputServer that makes Mac OS X like Vim.. but I'm probably better off not doing that [22:05:17] * jemfinch *really* wants that option in safari. [22:05:27] I don't use websites like that [22:05:45] etrepum: I don't have a choice, I work with Plone, and it does that. [22:05:51] ugh [22:05:52] apparently it's more "accessible" that way. [22:06:18] well safari's text fields suck anyway in panther [22:06:26] they have emacs keybindings :) [22:06:33] but no undo! [22:06:37] are they going to get better in Tiger? [22:06:44] no comment? [22:06:51] hehe [22:07:00] this channel is logged :) [22:07:06] * jemfinch unfortunately won't be able to afford tiger for some time. [22:07:21] pje: any answer as to my question? :) [22:07:23] that's a shame, it's not that expensive [22:07:31] jemfinch, I stepped afk; just got back [22:07:47] * jemfinch <- college student and trying to save up for an engagement ring [22:08:02] Anyway, it's for enterprise applications. [22:08:22] == applications with $$ constraints on reliability, uptime, configurability, etc. [22:08:25] pje: I work for a small consulting firm, I'm not really sure what exactly is meant by "enterprise applications" [22:08:49] but some features that are in PEAK and PyProtocols are useful in general, for a much broader problem domain [22:09:12] etrepum, true. But he asked where the ROI was [22:09:26] And in monetary terms, that means apps where not using it costs $$ :) [22:09:49] PEAK isn't yet what I'd consider best-of-breed in any other category yet. [22:09:51] if nothing else, the ROI is that you'll be smarter after having learned it. it's designed well, and it'll teach you a bunch of good practices for how to design your own code. [22:10:12] etrepum: that's certainly worth it, but I'm also working on a bit of a (self-imposed) time constraint. [22:10:24] I'm writing YAVCS and time-to-market is probably pretty important. [22:10:25] jemfinch, etrepum can say that, but I can't/won't. Because you might believe him, but not me. :) [22:10:36] if it needs to be done tomorrow, don't learn peak :) [22:10:38] pje: I understand :) [22:10:41] (He hasn't written any part of PEAK) [22:11:45] it certainly seems interesting, but I'm leary of being drawn into the "feeling like I'm doing work without actually doing any" that so many "java monkeys" are satisfied with. [22:11:54] Hm. Do Macs have arrow keys? [22:12:02] yes macs do have arrow keys [22:12:08] (of course, it could be the case that I'm doing that now, and PEAK would fix that, but I've seen results with my current method :)) [22:12:08] Ah, good. :) [22:13:03] most of PEAK is about writing less code, not more [22:13:23] Yep. Take the command-line apps framework, for example. [22:13:36] definitely tell me more about that :) [22:13:41] and bindings are shorter than their regular python equivalents [22:13:50] Once we have eggs, I'm going to start versioning different parts of PEAK separately, so it'll be more obvious what parts are stable/useful etc. [22:14:00] eggs? [22:14:09] http://peak.telecommunity.com/DevCenter/PythonEggs [22:14:22] (not really PEAK related, just happen to be on this wiki) [22:14:26] And see also IntroToPeak for an intro the command line framework [22:15:06] And http://peak.telecommunity.com/DevCenter/OptionsHowTo for how to specify option parsing [22:15:17] IntroToPeak is http://peak.telecommunity.com/DevCenter/IntroToPeak [22:16:01] It's actually tempting to "disbrand" PEAK altogether; i.e. not even bother to make the packages be peak.* anymore. [22:16:27] OTOH, it's also really convenient to do from peak.api import *, so there you go. [22:17:03] the "import *" stuff isn't very popular in some circles [22:17:07] (did everyone understand what I meant when I said YAVCS, btw?) [22:17:30] etrepum, I know, but those people can do 'from peak.api import foo,bar' [22:17:59] I personally think you can get away with *one* 'import *' per module. [22:18:07] yeah [22:18:10] that is usually sane [22:18:51] I didn't know what you meant by YAVCS.. I assumed Yet Another (something complicated enough to ask about PEAK) [22:19:05] But it's certainly interesting to think about what happens to package branding in a post-eggs world [22:19:36] well in the post-CPAN perl world they still use packages even though they don't have to [22:19:54] (phone call, btw, so I may be slow to respond) [22:19:55] not that perl is a good example for most things [22:20:02] Yeah, but Perl packages are rarely grouped for branding like PEAK/Zope/Twisted [22:20:14] I think when packages have specific modules from which "import *" is permitted, it's fine. [22:20:21] * pje assumes YAVCS is Yet Another Version Control System [22:20:36] yup, that's it (sorry, I would've explained it earlier, but the phone call) [22:20:54] that was a guess, but it seemed like sort of a weird thing to use PEAK for [22:21:03] For command-line VCS tools, the command-line framework and config stuff are probably quite useful [22:21:08] that's true [22:21:11] that's one of the reasons time-to-market is important, because there are some other VCSes (written in Python) that are also being written. [22:21:23] that have already been released, even [22:21:24] Possibly some of the file locking support, too. [22:21:36] i think perforce got it right [22:21:38] dispatch is probably useful in a VCS [22:21:48] If there's any sort of database back-end involved, the DBAPI support is also likely to be handy [22:21:57] I think multiple dispatch would be useful (I can explain why once I'm off the phone) [22:22:11] dispatch is predicate dispatch, not multiple dispatch [22:22:27] predicate dispatch is a simple superset of multiple dispatch, though. [22:22:30] but multiple dispatch can be done with predicate dispatch, of course.. [22:22:31] yeah [22:22:34] yeah [22:22:34] So you get both. [22:22:47] ok, done with the phone. [22:22:52] predicate dispatch doesn't require so many damned subclasses [22:23:04] PEAK's naming and addressing frameworks are likely useful for URL processing and resolving symbolic repository names to actual addresses. [22:23:09] file locking support isn't too important, I don't think; I've got a pretty simple scheme to take care of that. [22:23:40] command line stuff is pretty important, though: one of the major advantages of this system (sdf) is that it's going to be *easily* extensible. [22:23:46] dispatch would be a good fit for PyObjC [22:23:59] post-1.4 I may integrate it [22:24:01] PEAK also has configuration frameworks for dealing with .ini, .xml, and ZConfig files [22:24:19] Which covers a pretty broad spectrum of configuration use cases. [22:24:58] yeah, .ini is likely what I'd use (sdf is geared very strongly toward line-oriented files, so that would be much better than XML or ZConfig) [22:25:00] (ZConfig is an Apache-style configuration format, with "key value" lines and ... sections) [22:25:16] pje: I work with Zope/Plone :) [22:25:22] Ok. [22:25:24] so I've dealt with ZConfig my fair share :) [22:25:47] PEAK integrates ZConfig with its own URL systems, which allows it to do things like reference resources in zipped packages. [22:26:03] what's the advantage of ZConfig over .ini files? [22:26:21] ZConfig is designed for less-technical people to edit. [22:26:33] Although .ini can compete to an extent I suppose. [22:26:41] odd. [22:26:57] * jemfinch wouldn't have imagined that'd be the case. [22:27:00] Well, PEAK-style .ini files often require Python code or Python quoting. [22:27:14] hmm. [22:27:32] pje: which parts of PEAK *wouldn't* be useful for YAVCS, do you think? [22:27:50] But, PEAK considers "configuration" to mean "application assembly/deployment" more than "runtime configuration" [22:28:01] Wouldn't be? Well, peak.metamodels. [22:28:24] peak.events isn't likely unless you're doing some kind of async I/O or daemon process. [22:29:11] peak.web, unless you have a web interface [22:29:24] peak.security, unless you need access control rules [22:29:34] can access control rules be configurable? [22:29:52] See http://peak.telecommunity.com/DevCenter/SecurityRules [22:30:28] You can certainly generate them by reading from a file and then using security.hasPermission[rule] = method [22:31:23] Btw, I don't suppose this YAVCS is going to be open source, is it? [22:31:28] oh, of course. [22:31:45] I've actually written a few documents and whatnot about it. [22:31:55] what will YAVCS do better than anything else? [22:31:57] Great. If you end up using lots of PEAK stuff I'll have something I can point people to when they ask, "Who's using PEAK?" [22:32:18] Because 90% or so of all PEAK apps development is closed source. [22:32:26] yep :( [22:32:29] ;) [22:32:29] yeah, that makes it hard for it to catch on. [22:32:38] but it's probably also harder to do OSS PEAK development. [22:33:03] Well, I don't really care about it catching on as yet. After eggs, the pieces that are mature will "leave the nest" I think. [22:33:11] because in a proprietary environment, you have complete control over the machines involved; in an OSS environment, dependencies need to be kept to a minimum to make sure as many users as possible jump on board. [22:33:29] catching on is a double edged sword too, because then you have to support old code [22:33:35] jemfinch, even so, PEAK builds a massive amount of stuff on its own. [22:33:40] etrepum, yeah, definitely. [22:34:11] twisted has a lot of cruft for old code, for example [22:34:19] pje: what do you mean, "builds a massive amount of stuff on its own"? [22:34:20] maybe that should be: A LOT [22:34:22] But if I have separate packages I can version them 1.0, 2.0, etc. and tell people not to spec higher major versions without getting breakage. :) [22:34:44] jemfinch, it includes a lot of packages. [22:34:56] ah, ok. [22:35:05] Like for FastCGI support. [22:35:17] how easy would it be, for instance, to include the parts of PEAK that I use with my application? [22:35:36] (Here's something I wrote about SDF, btw: http://supybot.com/Members/jemfinch/vcs/sdf/ ) [22:35:37] jemfinch, PEAK currently builds as a single distutils distribution [22:35:42] you really wanna include the whole thing [22:35:47] And I know some folks using Py2Exe to wrap it [22:36:08] And when we have .eggs available you'll be able to do single-file deployment for PEAK. [22:36:41] Heck, PEAK already supports running from a zipfile (for everything but extensions) using Python 2.3.5 or Python 2.4.1. [22:37:11] how big is PEAK, if it's distributed with an application? [22:37:27] (or is it even possible to distribute the two together?) [22:37:42] A Windows .egg file is 2.7 megs for all of PEAK [22:38:32] is that with compression? [22:38:54] Yeah. [22:39:03] and extensions? [22:39:08] But source and .pyc both, with extensions [22:39:23] ok, I was going to say :) [22:39:28] I'm rebuilding it right now w/out metamodels and tests, to see how much that shrinks it [22:39:37] what is metamodels useful for? [22:39:41] Ah, 1.3 megs without those [22:39:52] jemfinch, only experimentation at present. [22:40:05] If you need to process UML or CWM files written in XMI, for example. [22:40:20] ok. [22:41:15] It includes PyProtocols in that bundle too, with all the predicate dispatch stuff. [22:42:07] It also includes wsgiref... fcgiapp... ZConfig... [22:42:26] Gosh it's going to be nice to be able to make those optional, separate downloads. :) [22:42:38] Oh crap. [22:42:40] yeah, I think that's important for uptake in the OSS world :) [22:42:49] if python standardized on eggs and wsgi like java does with .wars and whatever else, CDNs could offer a python-appserver-on-demand solution [22:43:00] I just realized I forgot to implement optional items in pkg_resources requirement specs. [22:43:25] CDNs? Does that mean Canadians? :) [22:43:31] content distribution network [22:43:33] Content Delivery Networks [22:43:35] or that [22:43:44] like Akamai or Digital Island [22:43:51] Interesting. [22:43:55] Speedera (now Akamai) [22:44:03] htey offer java appservers on demand [22:44:04] But wouldn't your databases need to be distributed also in that case? [22:44:12] they handle that, kinda [22:44:14] not the databases [22:44:20] but a decent size of session data [22:44:21] PEAK does have a JNDI equivalent, so I guess a similar approach could be used. [22:44:30] that you can use to batch updates back to an origin database [22:44:42] Hm. [22:44:53] you're guaranteed to have synchronized session data across any server that happens to be running the app [22:45:04] * pje dislikes session databases -- they break transactional guarantees wrt enterprise data [22:45:28] its a hard problem to have full RDBMS stuff going on in a grid computing/on demand environment [22:45:30] transactional guarantees are the antithesis of distributed computing [22:45:47] its not a panacea [22:45:51] but it works for a lot of apps [22:46:04] Akamai does the plane arrival times for the DOT [22:46:08] at all the airports [22:46:23] * pje has got to get some dinner [22:46:27] Later, all [22:46:30] later, phil [22:46:30] ciao [22:46:34] Bob, thanks for the help w/the egg design [22:46:43] * pje waves [22:46:47] pje: I wish I had some time to hack on it.. I hope to soon [22:46:48] ** pje has left us [22:46:59] thats not a bad idea for a startup [22:47:11] what's not a bad idea? [22:47:12] buy 500 servers in a few good locations [22:47:21] make a nice portal, you upload an egg [22:47:25] dns magic [22:47:41] your egg gets instantiated on servers close to where its requested [22:47:58] distributed consensus protocols provide synchronized session storage [22:48:00] well, that assumes you have the sandboxing technology to pull it off, the capital to start it, and that Python is popular enough to sustain the business [22:48:11] Python+WSGI that is [22:48:13] and my non-compete is voided [22:48:25] that doesn't stop me from stealing your idea ;) [22:48:28] haha [22:48:39] how much sandboxing do you really need? [22:48:55] how much do you trust your customers? [22:49:31] you can get started without heavy sandboxing [22:49:35] true [22:49:53] technically I guess you could make them put everything in an egg and get away with chroot jails on freebsd or whatever [22:49:56] i think if you had 50 servers [22:49:59] and a slick demo [22:50:21] *shrug* [22:50:26] but the infrastructure to support developers making eggs+WSGI is so nascent right now [22:50:53] well right, you'd need some capital to hire 5-10 people to hack heavily for a few months [22:50:58] the company would be far ahead of its time if started in the next year or two [22:51:10] so you'd need to hack heavily on pushing egg+WSGI and on the CDN [22:51:10] we should discuss this further [22:51:47] its a fundamentally better solution [22:51:54] for the type of apps that get run at the edge [22:52:03] maybe with the new startup I'm doing, I could put in some time to build a prototype for our edge servers [22:52:22] it won't be hosting untrusted apps, of course, but it could do the rest [22:52:35] no, i mean, we only host trusted apps [22:52:49] we assume the customer has a vested interest in the thing working [22:53:03] theres a lot of tough issues, debugging, etc... [22:53:10] yes but the customer might also have a vested interest in poking around the system for other people's apps [22:53:20] which are easier to solve in python than java, but still not easy [22:53:25] true [22:53:39] backdoor the app to give you a shell prompt to poke around [22:53:47] im sure theres ad-hoc solutions that can be used until linux gets real sandboxing [22:53:52] maybe you could use UM< [22:53:53] UML [22:54:17] it could open a tcp connection to your servers, then you get N connections opening up to you and giving you shells on all those boxes to poke around for other people's apps [22:54:41] UML might be a little too heavy [22:55:15] I think the chroot/jail solution is ideal if it can be made to work safely [22:55:25] i dont think any of those vulnerabilities would be a showstopper, given some smart engineers, a little time, and auditing of the code that gets pushed out [22:55:46] yeah auditing code is an expensive workaround [22:56:05] i mean, you can p;ut arbitrary constraints on how you get resources [22:56:17] modified VM and file/socket modules, im sure [22:56:32] well you're going to have to let the customer more or less run arbitrary C code [22:56:34] checks on things like INPLACE_POW [22:56:38] for performance [22:56:51] I'm not talking about putting multiple people's apps in the same Python VM.. that simply won't work [22:56:57] right, of course [22:57:32] but the overhead of a VM isn't big at all if you're paging in the same python library to all the instances [22:57:44] the ideal is sandboxed spaces, or however close you can get to that with cleverness and brute force [22:58:01] well [22:58:05] we do it with java, and it works [22:58:10] I think it can be more or less done, on FreeBSD anyway [22:58:23] Java is designed to let you do that [22:58:24] so, it would seem to follow that its possible with python, and likely easier [22:58:47] no, it's intractable in Python without a rewrite of large interpreter chunks [22:59:17] yeah, i guess you're right, actually [22:59:23] consider the sad history of rexec [22:59:41] a metaobject protocol like pythons is a security nightmare [23:00:35] but it doesnt matter if you sandbox on the kernel level [23:00:39] but having an interp-per-customer is fine.. the way Python's memory management works is a lot more conservative than Java's [23:01:26] yeah, maybe its too soon [23:01:35] maybe pypy will make things like that eaiser [23:01:39] yes it will [23:01:47] SandboxedObjectSpace [23:01:49] or some shit [23:02:20] even still, if C extensions are supported, you have the same problem [23:02:42] you could offer the twisted/peak platform [23:02:46] or whatever [23:02:58] yeah [23:03:13] it could all run on the CLR with IronPython too :P [23:03:19] haha [23:03:46] i dont think cpython is going anywhere for awhile [23:03:52] likewise [23:04:23] operating systems have a more robust security model anyway, better not to do it in userspace any time soon [23:05:17] yeah, it should be orthogonal to the language [23:06:06] i dont think python on the CLR is ever going to feel pythonic [23:06:27] well, it can feel more pythonic than PyObjC [23:06:38] the devil is in the details with that shit [23:06:52] because there's less impedance mismatch between C# and Python syntax than there is with Objective-C [23:07:17] yeah, true [23:08:00] and the threading model of .NET GUI stuff is a lot more "pythonic" in that it has its own thread behind your back so you can write cute clippy demos in the interpreter [23:08:07] blocking style [23:08:22] where in other models you need a runloop to be pumping [23:09:18] or maybe that's just part of the interpreter executable [23:09:36] I haven't looked at IronPython's source [23:09:41] yeah, nor have I [23:09:50] but the interactive demo like that is not something we can do with PyObjC [23:09:55] the rest we can, though [23:10:07] but they can't load CPython stuff :) [23:10:24] haha [23:10:29] hey, i have an idea [23:10:34] lets blog about this for the next 6 months [23:10:53] blog about what? [23:11:29] dynamic languages on the CLR [23:11:29] dynamic languages on the CLR [23:11:34] lol [23:12:10] I think it's funny that he isn't even on the IronPython mailing list or anything [23:12:21] its like he reads slashdot and just spews random shit every day [23:12:29] I don't think he's *really* paying attention to anything [23:12:35] haha [23:14:17] should name your blog Making Shit Up [23:22:05] hahaha [23:22:07] yeah [23:22:09] seee [23:22:24] i have a problem establishing my real voice in a blog about programming [23:22:43] i've had visceral political blogs that last a month or so, because I run out of bile [23:22:48] i need to find the middle way [23:22:55] my blog is like my mailing list voice [23:23:05] "this is how you do shit" [23:23:06] i dont like holding myself back tho [23:23:11] yeah, exactly [23:23:24] i'd like to be able to post the quote from the topic in my blog [23:23:31] but then i'd be viewed as a wacko [23:23:48] topic in macpython [23:23:52] yeah [23:24:03] thats how i really feel [23:24:18] that's almost more of a livejournal kinda rant [23:24:20] but i mean shit, what would ryan tomayko and patrick logan think [23:24:21] yeah [23:24:31] mark that shit friends only, so the interweb doesn't archive it all [23:25:43] hahaha [23:25:44] ok [23:25:47] time to stop testing [23:25:53] just got a call f rom neteng-oncall [23:26:08] maxing out the cambridge->santa clara link [23:28:48] haha [23:31:31] magic ping of death [23:31:36] yeah [23:31:55] rather, magic udp flood of 6000k pps/sec bidirectional [23:32:04] replicated x3 on the overlay [23:33:20] my co worker came crashing into my apt, kramer style, ranting about something about work [23:33:23] like 20 mins ago [23:33:28] he left a six pack here tho [23:34:13] negro modelo [23:34:14] not bad [23:34:30] I should grab some beer [23:34:43] do you live in NYC proper? [23:34:56] yes [23:35:10] nice [23:35:31] my friend lives in manhattan, east 95th st... i visit him a lot [23:35:38] I'm down in the lower east side [23:36:09] let me know next time you come down, if I'm still here :) [23:36:16] do you pay $12 for a six pack? [23:36:23] or is that just in times square [23:36:30] just in times square [23:36:35] haha, yeah, i know [23:36:40] its ridiculous tho [23:36:47] it's not cheap, but not that bad [23:36:51] are you planning on emigrating? [23:37:00] I'm moving to SF [23:37:05] ahh, lucky fucker [23:37:10] sometime in the next few months [23:37:11] i almost did that in '00 [23:37:16] I almost did it last year [23:37:20] going to do it for real this year [23:37:26] but the bubble spewed feces all over my whole game [23:37:43] yeah [23:37:44] i could do it now, but im married and shit [23:38:02] linda would be down, i guess.. maybe if i could get a gig at google [23:38:07] yeah I need to do it before I accumulate even more stuff [23:38:08] or at our san mateo office [23:38:20] yeah actually, im supposed to visit nik in the next couple weekends [23:38:27] are you etrepum on AIM too? [23:38:29] yeah [23:38:32] im argv0 [23:38:36] nice [23:38:53] i cant wait til tiger [23:39:02] 4 way ichat + subethaedit [23:39:07] haha [23:39:11] you can do group chat now [23:39:14] just not with video [23:39:17] yeah [23:39:50] yeah, i like nyc, i dont visit enough... [23:40:00] last time i went was to see amon tobin at the knitting factory [23:40:36] I haven't seen a show in a while [23:40:38] i think i'd go crazy living there [23:40:40] tho [23:40:51] how so? [23:40:52] it seems more stressful than boston [23:40:57] yeah I guess so [23:41:00] I don't know boston [23:41:05] maybe because I was on mushrooms, lost in times square tho [23:41:10] that would do it [23:41:12] haha [23:41:13] yeah [23:41:24] times square is whack [23:41:25] i figured it might have played a part [23:41:28] yeah its totally whack [23:41:34] it was an efnet irc meetup [23:41:40] which turned totally fucked up [23:41:41] haha even worse [23:41:50] which channel? [23:41:54] haha [23:41:55] #high [23:41:59] ah [23:42:12] so, you can imagine [23:42:21] I know a bunch of #2600 people around here, but not because of the channel [23:42:27] yeah [23:42:42] i used to go to 2600 meetings at the local shopping mall gorwing up in CT [23:42:49] organized thru bbses and shit [23:42:59] i went to one at pentagon city when I was a kid [23:43:11] i live upstairs from a bunch of LoPHT and CdC people [23:43:14] Dr. WHo [23:43:27] they're old timers now tho [23:43:31] yeah [23:47:14] yeah man, more power to you if you go to SF [23:47:27] i think its the most beautiful city in the world [23:47:38] and the most sane in the US, these days [23:47:56] I'm not too thrilled about the whole transport thing there [23:48:01] NYC is so easy to get around [23:48:11] yteah [23:48:28] I sold my car years ago and didn't bother to renew my license [23:48:32] i assume you've been out there before tho [23:48:33] recently [23:48:37] yeah I have [23:48:39] and have seen the beauty, no? [23:48:40] yeah [23:48:41] I was there last year for WWDC [23:48:44] nice tittie bars to [23:48:45] o [23:48:47] I'm going there next week [23:48:55] never been to one of those out there [23:49:11] we went there on our honeymoon on the way to hawaii to break up the trip [23:49:20] cool [23:49:21] it was a blast [23:49:27] coolest city, i swear [23:49:40] whatever it is is intangible [23:49:40] I had fun geeking out with the apple engineers [23:49:42] but i love it [23:50:12] we went to this mexican place and drank soooooo much tequila, it was retarded [23:50:21] thats the other thing [23:50:31] ALL the restaurants serve good food [23:50:43] there's lots of good food here too, different stuff [23:51:10] man, wwdc costs $1500 [23:51:26] thats insnane [23:51:34] not if you're speaking [23:51:40] well, yeah [23:51:42] lucky [23:51:57] it would cost twice that to keep my wife occupied in SF for 4 days [23:52:00] it is a week long, and there's lunches, and a bash on apple campus, and free jamba juice.. and you get stuff, like a laptop bag or an iSight [23:52:19] im trying to decide on a summer vaca [23:52:37] i think we're just gonna get a place on cape cod for a couple weeks thi [23:52:39] tho [23:52:57] that doesn't sound bad [23:53:11] my cousin is getting married in cape cod this summer [23:53:20] nice, cape cod is beautiful [23:53:47] http://shakti.ennui.net/~andy/images/sunrise.jpg [23:53:55] nice