[04:02:26] [connected at Mon Mar 14 04:02:26 2005] [04:02:26] <> *** Looking up your hostname... [04:02:26] <> *** Checking ident [04:02:26] <> *** Found your hostname [04:02:57] <> *** No identd (auth) response [04:02:57] <> *** Your host is sterling.freenode.net[freebsd.widexs.nl/6667], running version dancer-ircd-1.0.35 [04:02:57] [I have joined #peak] [04:02:57] ** sterling.freenode.net set the topic to http://dirtsimple.org/2004/11/generic-functions-have-landed.html [05:27:50] ** debugger has joined us [06:14:19] ** apoirier has joined us [06:52:22] ** bear has left IRC (Read error: 110 (Connection timed out)) [08:10:09] ** vlado has joined us [09:31:04] ** Maniac has left IRC (Read error: 54 (Connection reset by peer)) [10:03:34] ** bear has joined us [10:57:40] ** vlado_ has joined us [10:58:36] ** vlado has left IRC (Read error: 104 (Connection reset by peer)) [11:13:12] ** vlado_ has left IRC ("Leaving") [12:01:22] ** hazmat has left IRC (Read error: 60 (Operation timed out)) [12:07:17] ** hazmat has joined us [12:17:30] ** etrepum has left IRC (Read error: 60 (Operation timed out)) [12:33:54] ** hazmat has left IRC (Read error: 110 (Connection timed out)) [12:51:23] ** hazmat has joined us [12:58:29] ** etrepum has joined us [13:00:24] ** sprout has joined us [14:00:18] ** apoirier has left IRC ("Using KVIrc 3.0.1") [16:39:56] ** hazmat has left IRC (Read error: 60 (Operation timed out)) [16:42:23] ** hazmat has joined us [17:17:43] ** bear has left IRC (Read error: 145 (Connection timed out)) [17:27:50] ** bear has joined us [19:35:47] ** hazmat has left us [19:37:52] bear is now known as bear_afk [20:19:02] ahh [20:19:12] python code highlighting works [20:40:42] ** debugger has left IRC () [21:14:59] arg: nice [21:15:29] arg: I wrote a really stupid docutils plugin that does it with "pycode::" a long time ago, how did you do it? [21:19:01] bear_afk is now known as bear [21:32:35] t.p.htmlizer [21:32:51] now i have do go an de-hackify all the stuff i rushed through [21:33:14] fill out the XML apis and do the RSS feeds, plus whatever web admin stuff im going to do for now [21:33:18] then i can release a 0.1 [21:36:37] ** sprout has left IRC ("Snak 4.13 IRC For Mac - http://www.snak.com") [21:37:11] http://argv0.net:8080/ [21:37:32] theres what i had last night [21:37:44] the layout needs work, obviously [21:46:42] did you write it as a docutils plugin? [21:47:12] this is an interesting error: [21:47:12] unsupported operand type(s) for |: 'str' and 'function' [21:47:14] hehe [21:47:18] haha yeah [21:47:20] the comments link [21:47:23] yea [21:47:26] that actually works now [21:47:45] I'm not sure I've ever seen a traceback wrt the | operator [21:47:52] no, im actually havent dont the ReST stuff yet, so i havent dove into docutils yet [21:48:16] the rest stuff isn't that hard [21:48:19] yeah [21:48:30] im working on a simple way to add hooks [21:49:05] and i'll add something like xmlrpc->xmlwrapper->rst2html [21:49:22] xmlwrapper will try to fix up obiously unmarked-up posts [21:49:34] basically just add an enclosing

tag [21:50:12] oh, ahaha, i know what the | operator was [21:50:15] what i'm using now is kinda ghetto: [21:50:16] http://bob.pythonmac.org/rest-plugin.txt [21:50:19] i was using that infix hack [21:50:39] like foo |isa| bar [21:50:52] and i just moved a bunch of code around.. som somethings prolly a string now [21:51:20] I would be more inclined to write isa(foo, bar) :) [21:51:31] yeah i would be too [21:51:34] i was drunk last night [21:51:44] that would explain it [21:51:54] i changed it back, hence the traceback :) [21:51:56] I too have written "interesting" code drunk [21:52:13] oh man, i've destroyed good code drunk too [21:52:16] thank god for subversion [21:52:34] yeah no shit, i check in constantly [21:52:42] I want a wiki that uses svn as the backend [21:52:48] trac doesn't, for some reason [21:52:49] yeah that would be nice [21:53:03] a commit hook could expire the cache [21:53:06] we have a homegrown document management thing at work that uses perforce [21:53:12] or generate the content even [21:53:34] eventually i think ill get rid of sqlobject too [21:53:52] raw SQLite or what? [21:53:58] and use either twisted.enterprise or something I write [21:54:04] twisted.enterprise is bogus [21:54:04] or perhaps peak [21:54:07] yeah i know [21:54:22] shit, SQLObject is decent just for schema generation [21:54:31] its hardly worth using for more than a wrapper for deferToThread [21:55:19] but once that finalizes and i decide how i'll do persistent storage for plugins, i may get rid of it [21:55:31] using subversion instead of a database might be interesting too [21:55:38] yes [21:55:39] like, I'm not sure you need relational anything [21:55:41] im thinking that [21:55:43] you really dont [21:56:00] its amazing such a simple model is so complicated to implement... its a friggin list [21:56:14] if this were 1994 i'd have done this before lunch in Visual Basic for Foxpro [21:56:20] :) [21:56:26] it's not hard to implement, it's just that people insist on doing it the hard way [21:56:27] it wouldn't be half as fun though [21:56:29] yeah [21:56:48] do you guys use erlang at all? [21:56:53] single xml files for storage would be nice [21:56:58] nope, no erlang [21:57:18] I've used peak.model to do the xml thing [21:57:23] yeah me too [21:57:31] i love peak.model [21:57:42] yeah me too [21:57:45] peak took me a good while to get my head around [21:57:59] most people i cant get to commit to reading the twisted tutorial [21:58:04] nevertheless grokking peak [21:58:13] it's kinda like twisted, except the bad feeling goes away after you grok peak.. where it doesn't so much with twisted [21:58:34] there's so many things that need better APIs in twisted [21:58:36] yeah [21:58:55] I almost want to just .. rewrite it [21:58:56] peak.events is wonderful too [21:59:19] the whole yield event, or anyOf(evts..) [21:59:28] events.until() etc [21:59:41] I need to implement a high performance stupid web server.. I've been prototyping in twisted, but I'm pretty sure it can't stay in twisted [21:59:49] peak feels one level higher than twisted [22:00:00] yeah [22:00:17] twisted makes you play silly games to get references from one end of an app to another [22:00:39] maybe I should reprototype on top of peak.events [22:00:44] whereas binding.Obtain/Delegate [22:00:53] pje already has a lot of twisted stuff in peak.events [22:01:09] you can use a deferred in most places you'd use a native peak.event [22:01:09] yeah [22:01:38] my requirements are pretty simple, I need to parse URL encoded shit.. lots of it, real fast [22:01:52] not so much real fast, but real concurrent [22:02:06] ive been wanted to write a process pool class [22:02:18] *wanting [22:02:30] and then I need to get that information into a database and feed it somewhere else [22:02:53] I think I'm going to have a pool of web servers, each with their own local SQLite database, and then that gets pumped into PostgreSQL in batch for mining [22:03:21] yeah, that sounds workable [22:03:33] because I certainly can't feed directly into postgresql piecemeal [22:03:36] and I don't want to use MySQL [22:03:40] yeah i've tried that [22:03:44] i feel dirty using mysql [22:03:50] so dirty [22:04:05] have you read the google MapReduce paper? [22:04:15] I don't think so [22:04:24] its how they process lots of data concurrently [22:04:33] you can kinda guess the model [22:04:56] I'm not too worried about processing, that doesn't need to be near-real-time [22:05:00] yeah.. [22:05:01] I just need to not choke on input [22:05:30] do you reply to the requests or are you just sucking data in? [22:06:22] the reply is simple.. on the first request in a session I need to generate a session id for the client, and possibly also a GUID if the client doesn't already know one [22:07:08] the client knows its own session id and guid after that, so the servers just need to make sure they don't have collisions, which is easy [22:07:13] * bear pops in with a hopefully non-silly question [22:07:21] shoot [22:07:25] because I can just toss the server's ident into the session id [22:07:30] can you get the data from the stream, parse it really fast and put it into a queue [22:07:41] then have another thread consume that queue for database insertion? [22:07:55] yeah, that's more or less what I'm going to do [22:07:59] is that what your talking about above? [22:08:00] memory is so cheap nowadays [22:08:15] yeah [22:08:39] you should fill up memory, then send a temporary redirect to the next machine when you commit it all [22:08:40] the cheap servers we bought to start out with only have 1gb each, but I don't think that's going to be a problem [22:08:54] the processing servers will probably need a bump [22:08:58] so the only twisted http part you need is getting the stream from the url - after that it's in the other process' queue [22:09:03] but the web servers don't [22:09:20] brb, doorbell [22:10:49] the twisted http part is a server, not a client [22:11:09] oops - that's what I get for jumping into the middle [22:11:19] but yeah.. I'm just not sure I trust twisted to scale to a massive number of requests [22:11:42] I was looking at erlang / yaws, but getting the data out of erlang looks like a PITA [22:12:12] if your looking at erlang, then how about ruby? [22:12:27] ruby doesn't do anything for me that Python doesn't [22:12:32] erlang does [22:12:54] k [22:13:15] erlang is basically designed for just this kind of shit [22:13:16] ruby is all OO instead of pythons OO-if-you-don't-look-hard [22:13:27] I'm looking for concurrency, not OO [22:13:29] but I guess for web stuff OO is not the answer [22:13:35] * bear nods [22:14:01] I really don't care what the syntax looks like, what matters is scalability [22:14:21] this particular component is simple enough to write in C if I have to, but I don't want to [22:14:27] watch out, you could be talking yourself into a C app :) [22:14:45] * bear is 2 seconds too slow to finish typing [22:14:46] i was going to say you might want a vector [22:15:07] I was looking at ACE earlier but I couldn't find any documentation that didn't suck [22:15:24] if you don't trust twisted then I wouldn't trust ACE either [22:15:37] ugh, i have an ACE book, i wouldn't bother [22:15:38] the twisted folks wrote it because of how bad ACE sucked IIRC [22:15:52] well, sort of.. [22:16:06] I don't trust twisted because I know how it's implemented, and I know what Python isn't good at [22:16:22] but if you dont need to do anything with the urls, you can fill up the big vector then dump it out to disk and have a different process put it into the database [22:16:35] I could fix Twisted's flaws by rewriting what I need in whole or part [22:17:15] I do need to parse them a little bit to figure out whether I need to generate a new GUID or session ID [22:17:18] if you are worried about speed why not mmap the file between two processes - one fills the other feeds [22:17:25] that avoids the copy on write issues [22:19:46] I'm not too worried about IPC, just having Twisted handle thousands of connections at the same time [22:20:02] and not fall over [22:20:04] yeah, maybe you do want ACE :) [22:20:32] it doesn't necessarily need to handle them all really fast, it just can't blow up [22:20:33] you may end up worrying more about thread limits than twisted's ability to handle the connections [22:22:01] I won't have more than three threads in the server process [22:22:25] * bear kicks himself [22:22:40] you mean process limits? [22:22:48] you were talking about twisted's over all reliability rather than any issues with thread handling or max threads - sorry [22:23:12] I was talking about its ability to handle lots and lots of TCP connections at the same time without falling over [22:23:35] it's worked every time I've used it, but I've never dealt with more than a few hundred at once [22:24:04] like < 500 [22:24:16] would be interesting to see what upper limit of tcp connections twisted has [22:24:30] well I'm sure I'll find out [22:24:31] if the OS/process starts to bog down before python does [22:25:02] I'll either be using epoll or kqueue, depending on which of Linux, FreeBSD, or NetBSD sucks less [22:25:24] so I expect pretty good performance out of the OS [22:25:40] sounds like you'll be testing the twisted app against all three then [22:25:49] yeah [22:26:58] maybe itamar's BOOST reactor would make a difference [22:30:08] oh yeah, i remember seeing that in the sandbox [22:31:01] it's on his site too [22:34:19] at this point it's just research, I'm not prematurely optimizing anything.. but when I have to, I want to have a pretty good idea of where to jump ship if the boat is gonna sink :) [22:34:31] and where the life boat is [22:34:59] I could probably launch with an unoptimized twisted implementation and be OK, but it's definitely not going to last [22:35:20] stackless might help [22:35:33] I could write something twisted-ish with stackless [23:08:14] ** bear_ has joined us [23:08:48] ** bear has left IRC (Nick collision from services.) [23:09:01] bear_ is now known as bear [23:54:33] ** bear has left IRC (Read error: 60 (Operation timed out)) [23:56:01] ** bear has joined us