[00:34:16] {global notice} Hi all, We have some more sponsors that need to reboot boxes for upgrades. I need to do some rehubbing, then some rotation servers need to be restarted. I will notice each of the effected rotation servers before they are rebooted. I will give more information in wallops '/mode your_nick +w' to see them. Thank you for your continued patience, and thank you for using freenode! [00:43:05] {server notice} Ok guys, it's you. I'll reboot it in a few mins. [00:45:14] [disconnected at Tue Dec 2 00:45:14 2003] [00:45:21] [connected at Tue Dec 2 00:45:21 2003] [00:45:21] <> *** Looking up your hostname... [00:45:21] <> *** Checking ident [00:45:22] <> *** Found your hostname [00:45:52] <> *** No identd (auth) response [00:45:52] <> *** Your host is brunner.freenode.net[brunner.freenode.net/6667], running version dancer-ircd-1.0.32 [00:45:52] [I have joined #peak] [00:45:52] ** brunner.freenode.net set the topic to PEAK http://peak.telecommunity.com || WikiPages at http://peak.telecommunity.com/DevCenter || IRC-Logs: http://peak.telecommunity.com/irc-logs/ [01:30:16] ** _Maniac_ has joined us [01:30:31] ** _jpl_ has joined us [01:35:23] ** hazmat has joined us [03:15:55] ** vlado_ has joined us [03:16:08] morning [03:43:16] good morning [04:55:55] ** jack-e has joined us [04:56:11] morning [05:03:56] hi jack-e [05:05:03] hey kapil .. welcome to #peak :))) [05:05:10] how're things going ?? [05:06:06] busy.. [05:06:32] starting a biz, moving across the country, doing some contracts [05:07:19] hey .. did you cancel your employee contract ? [05:13:27] yup.. 1st of nov was last day [05:13:47] much happier now [05:13:56] congrats :) .. you didn't seem enjoying your job .. [05:14:11] it was a nice place and fun people, but the work was lame [05:14:41] i'll keep my fingers crossed for your new project .. [05:15:09] what is it about ? [05:15:16] i think it will go well, ben and i are working together on it. did you get all the govt requirements for the station covered? [05:15:35] bcsaller ?= [05:15:52] yup, we're focusing on r&d infrastructure for plone at the moment, although a little bit of consulting as well initially.. [05:16:12] i'd like to broaden to take on other things like developing with chandler and peak.. but thats for the future [05:16:17] hey .. if you succeed and i get my gas station sold .. i'll join you :))) [05:16:36] sounds good! that would be fun [05:17:18] you'd have to throw a party ;-) [05:18:05] i'm trying to make my way to nyc to live... its a fun city.. but moving sucks.. [05:18:16] anyways for now its time to sleep. cheers [05:19:04] cu soon [05:19:59] * jack-e will visit hazmat when he found a flat in NYC to get out for a nice clubbing tour ];-) [05:24:11] http://newyork.craigslist.org/ [08:31:55] ** vlado__ has joined us [14:38:49] _Maniac_ is now known as Maniac [16:03:01] <_jpl_> Greetings all [16:03:05] hey! [16:03:14] * Maniac has lightning fast irc reflexes [16:03:39] <_jpl_> Wow, yes, that's the quickest response ever on #peak. :) [16:06:22] hey john, maniac :) [16:06:38] * jack-e just came back from dinner [16:07:13] _jpl_: i didn't have time to try out your stuff .. but i read through the code and it looks very promising .. [16:07:50] * jack-e is fighting with unicode and zope2 and reportlab and EURO-Sings %-( [16:12:19] <_jpl_> I've been working on separating Junction into its own independent package with extensive unit tests, etc., and am just about finished. [16:12:28] ql :) [16:13:19] <_jpl_> Once that's done I'll either put it on the PEAK wiki or some other accessible place for general use/testing/improvement. [16:13:55] i think the idea of a peak component/package repository ala collective vor CMF/Zope is not a bad idea [16:14:03] s/vor/for [16:14:21] <_jpl_> I haven't had time to do anymore SLP work since we talked last, but I really want to get what you did integrated with Junction. Dynamic discovery of Junction servers is really important to me. [16:15:04] i'ld contribute that part to junction if there is a chance for us to use it as well .. [16:15:21] <_jpl_> Yes, I think so too. We have a few other PEAKish things we could share, like TableDM (which is also getting closer to being sharable, and which now has pretty strong unit tests), some custom data types for peak.model classes, etc [16:15:49] i thought about improving the whole thing to get a "background discovery feature" that runs in the twisted.reactor loop using slp's callback features [16:16:11] if you enable this in your loop you get notified when services (dis)appear [16:16:47] (but there is no time for playing around with this currently) [16:19:19] <_jpl_> I've not seen the SLP callback features. Sounds interesting. [16:19:46] the slp api-functions have a callback keyword-arg [16:19:53] <_jpl_> We're now hitting a bit of a wall with peak.storage when it comes to inheritance, and have been considering going with ZODB instead. [16:20:14] inheritance of model.Element Classes ? [16:20:42] so that you can mix and extend existing types ? [16:20:52] <_jpl_> Right, trying to get DMs to do something intelligent on the RDBMS side is not easy. [16:21:42] how would you expect it to work .. have one type, create a subclass and add attributes .. where should the extra attrs be stored ?? [16:21:48] <_jpl_> We have Hosts, which extend Devices, which extend Assets, which is all find on the peak.model side, but translating to tables on the RDBMS side is a lot of work, and we don't have the time. [16:22:36] can't you do this through configuration of the ElementClass/DM ? how many types do you have to make persistent, which db do you use ?? [16:22:46] <_jpl_> There are three approaches we could take on the RDBMS side: 1) have one Assets table which contains fields for all Asset subclasses (this is the worst option) [16:23:02] i wrote a SQL-Schema -> Model.Element Python-Code generator for sapdb .. that one coud be used for postgres etc as well [16:23:11] (with some additional drivers) [16:23:53] <_jpl_> 2) have a separate table for each class, where each table has fields for all its parent classes (Martin Fowler calls this concrete table inheritance) [16:24:35] <_jpl_> 3) have a separate table for each class, where each table contains only the fields that are unique to the class (Fowler calls this class-table inheritance) [16:24:43] right [16:25:00] 2. is easiest to use with code-generation i think [16:25:04] 3. is nicest [16:25:51] <_jpl_> Agreed, but both are quite complicated to implement using EntityDMs. [16:25:54] but havin data in ZODB would not be an option for me .. you might end up in having more work enforcing relations than writing DM's [16:26:18] have you looked at my SQLDatamanager (inspired by PyDO) == [16:26:20] ? [16:26:36] i can read the schema of a db, and create the whole mapping for the SQLSchema [16:26:52] you end up in a mostly functional persitence layer using DM's [16:27:02] <_jpl_> Well, maybe that's not terribly fair to EntityDM -- there's just a lot of work involved in writing all the various SQL for each DM, which we were trying to get away from with our TableDM class, but TableDM turns out to be pretty useless for this sort of thing. [16:27:17] (the module is not uptodate for the latest name-changes .. but that is doable) [16:27:35] <_jpl_> I haven't seen it -- where does it live again? [16:27:44] my SQLDatamanger creates the SQL in a metaclass or latebinding from the field-spec [16:27:48] mompl. [16:28:40] (i currently work with a frozen version of peak-cvs pre naming changes) [16:28:49] this it the unit-test for the SQL-Datamanger: [16:28:52] http://cvs.net-labs.de/cgi-bin/viewcvs.cgi/libs/nll/src/nll/database/sqldm/tests/test_datamanager.py?rev=1.5&content-type=text/vnd.viewcvs-markup [16:29:17] there is an ElementClass defined (HostBackup) [16:29:27] and a TestDM [16:29:56] the TestDM is a SQLEntityDM that is configured with tableName, fieldSpec, indexSpec [sequenceSpec] [16:30:18] and a simple query-dm baseclass for trivial id-queries [16:30:45] what database do you use ?? [16:32:33] <_jpl_> We use Postgres, but were trying to be db neutral. Until yesterday when we decided to use Postgres's table inheritance feature. [16:33:17] we tend to use postgres as well .. so i might be interested in creating the schema-code generator driver of postgres .. but probably not within the next month [16:33:30] i used sapdb but its oversized for what we need [16:33:39] <_jpl_> (one specific problem we have with standard tables is that you can't easily query Assets to get all assets, devices, hosts, etc. *and* get the right object type back from the query) [16:34:13] right .. that is a feature, that is on the TODO list of PEAK afaik .. but scheduled after 0.6 i think [16:34:42] in Transwarp you could do this type of inheritance with support of the DM (or whatever name they had) [16:34:53] <_jpl_> Each DM would have to be aware of the child classes of the class that it supports... it's pretty unpleasant. [16:35:01] right [16:35:25] <_jpl_> Hence the desire to go with an OODB... [16:35:50] how much do you rely on the relationational part of your RDBMS ? [16:36:23] i tried to replace rdbms with zodb a few times .. but it's hard to do it right imho [16:37:05] so what does the table-inheritance feature of postgres do ?? [16:37:07] <_jpl_> Have you worked with IndexedCatalog? [16:37:31] does it do the work required by option 3 ? [16:37:46] it was not available at that time .. but it still doesn't enforce any relations .. right ? [16:38:49] <_jpl_> Yes, it essentially does all the hard work of option 3 (class table inheritance). We'd still have to deal with inheritance somehow on the DM side, though. [16:38:51] if so .. does the assets table then hold all entries (also from hosts, devices) for the shared attrs ? [16:39:40] <_jpl_> You have separate tables, but you can query the assets table and see all records from all child tables, though you only get asset fields. [16:39:41] that problem sound similar to our LDAP Datamanager [16:40:21] when you query an LDAP directory you don't know what element-class to create as GHOST (defaultState) [16:41:19] <_jpl_> As far as I can tell, we'll still need something like an asset_type field (which we'd have to manually maintain) in order to know what type of object any particular asset record represents. [16:41:26] you need to query for the objectClasses first, maintain a mapping between the object-class-set and your element-class and give back instances of the right class for each record [16:41:56] the drawback with that solution is, that you loose some of the lazy-loading features of peak [16:42:15] allthough we haven't yet found a better solution [16:43:01] if you have an asset_type field .. you need to query for this field before you create the instance of the ElementClass, and select which one to create based on e.g. ini-style configuration [16:43:14] <_jpl_> As far as the relational side of things, we don't have particularly complex foreign key relationships, so I was hoping using standard object references would work out alright in an OODB context. [16:43:42] i have some objections against having all my data in a big pickle ;-) [16:44:12] but that's my opinion [16:44:37] you'ld have to use ZODB4 to be compatible with peak.transactions .. [16:45:17] (or at least compatible with persistence .. i think transaction api's differ somehow in ZODB3/4) [16:45:52] <_jpl_> If we went this route we'd still keep a close link to the relational side for querying and real-time data analysis. For things that don't change much we'd probably export to Postgres daily, while things like events we'd keep live in Postgres. [16:46:35] so what do you save then, when you use the ZODB ? [16:47:00] <_jpl_> If we went with ZODB we'd probably have to stop using peak.model for now. :( We could write DMs to move data to/from ZODB and model classes, but it seems like we'd be doubling the work for little gain. [16:47:13] right [16:48:25] <_jpl_> We'd save a lot of work in writing all the object-relational data management code, which has already taken up a lot of valuable time for little gain. Interestingly, when we mentioned these frustrations to our VP he asked us why we weren't just using some sort of object store! [16:48:51] :) [16:49:27] * jack-e needs to hunt unicode bugs now .. i need to get the customers system back up running before they get up ;-) [16:49:59] <_jpl_> My only concern with ZODB is stability and querying capabilities. Stability is probably not really an issue, and IndexedCatalog seems to provide a powerful indexing/querying layer on top of ZODB. [16:51:23] i've never found a satisfying way to store object trees with associations in a senseful structure .. what belongs there, or there .. but that depends on the type of data you work with [16:51:29] * Maniac looks around for junction [16:52:00] <_jpl_> IndexedCatalog looks quite nice. Have you taken a look at it? [16:52:21] nope .. haven't done any ZODB (without Zope) work recently .. [16:53:01] <_jpl_> It doesn't do much for associations, though, but with ZODB references to other objects seem to work correctly. [16:53:43] you'll run into trouble when you move/delete objects probably .. [16:54:06] if you use hard-refs instead of path's as zope does [16:55:09] <_jpl_> Even if you confine creating/deleting to a specific location for each object type? [16:55:58] <_jpl_> meaning, a separate OOBTree for each type of object [16:56:14] <_jpl_> you'd only create or delete objects in their specific area of the db [16:56:43] but there will other object referencing them directly (like a.attr = b) [16:56:46] ? [16:57:06] or x.myref = '/objTypes/a/id1' [16:57:35] <_jpl_> Yes, but ZODB seems to ensure that you get the same object whether you load it as a.attr or from b's class's area of the db. [16:57:42] in both cases you need to query your whole database to find out referrers before you can delete an object [16:58:10] and resolve any conflicts .. that's what the relational db does for you if configured correctly [16:58:38] you can delete .. but then you have dead references which might break your system [16:58:45] <_jpl_> That's true. Luckily we don't delete much. :) [16:59:00] :) [16:59:35] <_jpl_> I can see how that could get really ugly and complicated. [16:59:43] yup [17:00:37] <_jpl_> But for our usage patterns, that would be some tiny fraction compared to the majority of creating/querying data [17:01:00] like i said .. that depends on your data-model [17:02:50] basically you could also use a DM to store your Element in a ZODB's OOBTREE [17:03:05] if you get the Transaction machinery togehter [17:04:55] <_jpl_> We could, but then we'd probably have to have shadow classes for each of our peak.model classes, which be what gets stored into ZODB (inheriting from ZODB's Persistent class). And in that case it seems like we'd just be doubling our work with little benefit. [17:05:37] ture [17:05:39] true [17:05:50] <_jpl_> I suppose there's also the option of using Berkeley DB directly, or even Metakit. [17:06:17] i had trouble with bsddb3 on debian lately (using ext3) [17:06:35] <_jpl_> There are plusses and minuses to every approach, no one approach solves all the problems unfortunately. [17:06:49] yeah .. thats REAL WORLD :) [17:07:36] <_jpl_> Ha, I never did like the real world much. :) [17:07:42] * jack-e too [17:08:49] have you looked at SQLObject ? [17:09:02] <_jpl_> Aside from storage our other big problem is picking one of the many approaches to building a web interface. Again, each solution has its plusses/minuses. [17:09:27] yes .. i tend to go with simple XML-Generation + XSLT [17:10:20] <_jpl_> I'm leaning toward Woven at the moment. [17:10:47] <_jpl_> Haven't looked seriously at SQLObject, no, [17:11:12] me too .. i just know that friends used it for a zope3 project as storage backend [17:11:23] (with some more/less glue) [17:11:42] aren't they about to rewrite woven ? [17:13:00] <_jpl_> There's nevow, which seems to be the successor, but there's no "official" word on it that I know of. [17:13:22] yeah .. that's what i often read in #twisted lately :) [17:14:04] <_jpl_> I've asked about it on #twisted, but never get a clear answer on its status or future. [17:14:33] you need to provide a complete managment - interface for your system ? [17:14:39] <_jpl_> And I'm not sure whether it would completely replace Woven (and if so, how different it would be), or if its more like a set of enhancements. [17:16:02] <_jpl_> Yes, asset management with the usual add/query/edit/delete features, plus job scheduling and monitoring, event management/monitoring, and various other things. [17:16:24] sounds like you want zope3 schema and add/edit-forms ;-) [17:16:34] one more framework :)))) [17:17:04] <_jpl_> I've been looking at X3 lately, but I don't have even a clue as to what its capable of. [17:17:31] <_jpl_> I'd be quite happy to use X3, actually, if I knew what it could do and how to do it. [17:18:06] <_jpl_> Knowing that it'll be using PyProtocols underneath the hood gives me some idea of what it could be capable of. [17:18:08] i've worked with X3 Milestone2-3 intensively .. it would help you a lot doing the Web-Stuff .. if they'ld have done all their refactoring after adaptergeddon i think [17:18:38] <_jpl_> It sounds like adaptergeddon is almost done. [17:19:04] yeah .. the core part .. but now all views, etc need to be refactored to use the newer simpler api afaik [17:19:23] (at least jim calls for volunteers ;-) [17:20:03] i would if someone would pay my income :)) [17:20:06] <_jpl_> True, that's not much of an indication of readiness. [17:20:06] <_jpl_> :) [17:20:50] i liked z3 alot .. but it tends too much to be nice to web-developers again i think [17:21:12] <_jpl_> What do you mean by too nice? [17:21:41] in the beginning there was a huge interest in being nice to other python-frameworks (opposite to z2) [17:22:19] but that was loosen't in favor to have simple TTW capabilities [17:22:26] IMHO !!! [17:23:37] _jpl_, the long term plan of glyph (and twisted) is to seperate out woven / nevow from twisted entirely [17:23:50] i believe the feeling is twisted != woven [17:24:06] which i'ld support [17:24:23] <_jpl_> What about short term? [17:24:33] twisted is far to big already [17:24:49] short term, they are focusing their energy on nevow, but they appear to be willing to maintain woven so far as what it is. [17:24:50] <_jpl_> That's probably a good thing, actually, since Woven could easily stand on its own. [17:25:06] i.e. new developement will be nevow [17:25:11] <_jpl_> hmm [17:25:33] also nevow has an actual appliation being driven by a large number of the twisted developers [17:25:47] (quotient) [17:25:56] <_jpl_> Right, I've been peeking at nevow via Quotient from time to time. [17:26:18] i find quotient incredibly hard to follow stuff like powerups, pools, avatars ug [17:27:00] what is quotient ? [17:27:09] http://stewstuff.com/doc/nevow.xhtml [17:27:25] * Maniac hopes everyone notices nevow == woven spelt backwards :) [17:27:36] jack-e, http://divmod.org [17:28:00] <_jpl_> Yes, the terminology in Quotient is pretty silly. [17:28:12] iQuotient is an open source product that combines a multi-protocol messaging [17:28:12] server with tools for information management and retrieval. [17:28:18] like some other twisted - packages - names ;-) [17:28:19] <_jpl_> Seems the developers play a few too many video games. [17:28:33] _jpl_, they have some sort of puns related to sandwiches and games [17:28:40] jelly, bananna etc. [17:28:42] <_jpl_> Brilliant. [17:28:48] :)) [17:28:59] manhole, trial [17:29:15] <_jpl_> Well, the pb/jelly/banana puns make *some* sense at least. [17:29:47] <_jpl_> I really like trial, by the way. I've switched over to using it rather than the built-in unittest module. [17:30:37] what does peak use? [17:30:53] unittest python-stdlib [17:30:53] <_jpl_> It's not a drastic improvement, but it does add some definite benefits. [17:31:30] i have no idear what twisted.trial has that unittest does not [17:31:46] <_jpl_> The 'trial' command line app, for one, is really useful. You can run "trial --recurse package" for example, to run all unit tests everywhere in a package. [17:32:19] anyone used peak.security yet? [17:32:28] _jpl_, ah [17:32:50] i took the test.py script from zope3 for that :) [17:33:05] <_jpl_> Basically the standard unittest module doesn't provide a test runner. You have to write your own program to locate, collect, run, and report all unit tests in your project. [17:33:38] ok [17:33:51] * Maniac wonders about peak.security [17:33:58] <_jpl_> Whereas trial does all that for you and gives you variuos options like how to display the output (including a handy colored version). [17:34:19] i have used it for a few tests with peak.web .. but i do not use it in my programs yet [17:34:34] jack-e, it is not specific to peak.web no? [17:34:48] <_jpl_> You can also put a 'skip = True' in a TestCase class in order to skip all the tests for that class -- really useful when you have a lot of broken tests in need of updating. [17:34:52] no .. except that peak.web requires you to use peak.security :) [17:35:18] i looked at the unit tests a little, but have no real picture how i would use it simply [17:35:38] <_jpl_> I've read the unit tests for peak.security a number of times, but haven't actually tried to use it yet. [17:36:45] mmm [17:36:54] peak web uses ist [17:36:58] s/ist/it [17:37:00] <_jpl_> It looks very promising, though [17:37:09] is there a sample in examples? [17:37:14] <_jpl_> I'd really *like* to use it, actually [17:37:19] i want to use it in my 'bot' [17:37:21] <_jpl_> No examples yet. [17:37:25] right [17:37:58] (rather than create my own type of security framework) [17:38:19] so go for it .. dive into the sources .. and get it running :)) [17:39:14] <_jpl_> pje responded to my query about it a couple of weeks ago, shedding a little more light on it in the process. [17:39:25] <_jpl_> Are you on the mailing list, Maniac? [17:41:35] _jpl_: of course! [17:41:47] _jpl_: i saw that message i think [17:42:04] <_jpl_> Ulrich, have you worked with SQLObjects much? [17:42:08] * Maniac can't figure out why mozilla thunderbird decided to not run [17:42:09] <_jpl_> It looks interesting. [17:42:58] nope .. not at all, i used pydo before i worked out the SQLEntityDM [17:43:55] but i heard good things about it [17:44:52] do you now sidney (dreamcatcher) ?? [17:45:01] <_jpl_> No [17:45:19] he's the one who used it with z3 [17:45:23] me heads home for the evening [17:45:32] nite maniac [17:45:40] * Maniac adds a / to that last comment [17:45:41] <_jpl_> Does SQLEntityDM handle references and enumerations and such? [17:46:48] you need to provide the referencing/referenced Columns by hand or use the schema-code generator .. [17:47:19] it s 1liner per reference e.g. state['owner'] = self.OwnerDM[row.owner_id] [17:48:34] there is a sample (outdated probably but it shows a large model created from a db): [17:48:56] http://cvs.net-labs.de/cgi-bin/viewcvs.cgi/libs/nllsandbox/database_components/HELLER.py?rev=1.1&content-type=text/vnd.viewcvs-markup [17:49:37] first all the autogenerated model.element-classes [17:50:01] after that the Query/EntityDM's (all information is created automaticall using the SQL-Schema inforfmation of SAPDB) [17:50:23] it's not 100% done .. but it works as far as i could see .. [17:51:31] creating proper names is probably one of the trickies parts of this job ;-) [17:54:34] <_jpl_> I'll take a look and see if it might be helpful to us. [17:55:47] (like i said .. it probably won't run with a current peak-checkout .. if you want to use it, i can get it uptodate in a couple of days when i find some time [17:55:49] ) [18:01:12] * jack-e heads home for some sleep as well .. [18:01:18] nite guys .. [18:01:53] jack-e is now known as jack-e|away [18:01:59] <_jpl_> tscheuss [18:02:19] night [18:12:20] ** hazmat has joined us [18:41:27] ** glyph has joined us [19:00:42] <_jpl_> Hi glyph [19:05:32] _jpl_: are you the author of PEAK [19:05:32] ? [19:08:56] <_jpl_> No! That's Phillip Eby. [19:09:01] <_jpl_> I'm just a user. [19:09:38] <_jpl_> Phillip is the one with the big brain. :)