[PEAK] PEAK design pointers

[Phillip J. Eby]

At 08:45 PM 3/5/04 +0000, Paul Moore wrote:

>What I'm trying to write is an application (more precisely, a
>framework) for database monitoring. I have a central repository
>database, and a large number (100+) of target databases. My
>application needs to connect to each target, run a query, collect the
>results and save them in a table in the repository. I'll ultimately
>have a number of different instances of this application, each
>collecting the results of a different query, and possibly handling the
>results in slightly different ways.
>
>Objects (components?) I see a need for include:
>
>     * A "target database"
>     * A "repository"

I assume these are DBAPI SQL databases?


>     * A "thread pool" (too many targets to query in sequence - I need
>       to do the queries in the background).

I'll probably be adding one of these to PEAK eventually, but you might want 
to just use Twisted's threadpool support, i.e. 
'reactor.deferToThread()'.  The resulting Deferred objects are usable with 
'peak.events' to control "tasks".


>     * A "report" (to pack up the results of a run for publishing)
>     * A "mailer" (reports are typically sent by mail)
>
>But I can't work out how to connect them all together (well, I can,
>but that's when all the nice modularity and flexibility breaks
>down...)

Well, if I understand you correctly, I'd probably write something that 
would spawn an events.Task for each of the databases to be monitored, and 
create another Task that accepted the results of each of the other tasks in 
order to prepare the consolidated report and mail it out.

But, I don't really know enough about what you're trying to even know if 
that's really appropriate, or if something else would be better.

The other thing I imagine I'd do is tie the components together with 
ZConfig, creating a top-level schema with sections to describe the 
individual databases, etc.  Then, my app would literally be assembled by a 
config document, like this:

<Repository>
URL pgsql:blah:blah at blah
SummaryTable the_summary_table
</Repository>

<DB Foo>
URL sybase:blah:blah at blah/blah
LoadDataFrom sometable
</DB>

<DB Bar>
URL dcoracle2:blah:blah at blah
LoadDataFrom othertable
</DB>


All of the DB things would get put into a list attribute on the 
application-level object, and the repository would be put in the repository 
attribute, etc.  The class that DB sections mapped to would have a binding 
to automatically start their data collection task, and of course you could 
have lots of other configuration settings on them.


>But as I say, my question is less about the specifics of my
>application, and more about how to *think* about this sort of design.

If you're thinking about the kind of problems I think you're thinking 
about, this is probably the way to go.  That is, design in terms of 
pluggable components, and let ZConfig do the selection, configuration, and 
assembly of those components for you.