[TransWarp] Tips on using model, storage, config, and naming (was Re: First attempt to use the storage package in PEAK)

[Phillip J. Eby]

At 10:31 AM 12/29/02 +0200, Roché Compaan wrote:
>Jeez, you think of everything ;-)

I try.  Sometimes I drive Ty nuts with my, "but what about X?" questions, 
though.  :)


>But I'm glad you put so much effort
>into a proper validation framework - I found that without it you quickly
>end up with spaghetti code especially when you have to validate changes
>accross fields and accross objects. I wont buy into a framework that
>validates fields atomically.

Yeah, that's pretty useless for meaningful domain models.  The work that 
I'm doing on the MOF will be an interesting test case in itself for 
validation: MOF has a lot of meaningful semantic constraints, like that 
something can't subclass itself or contain itself, directly or indirectly.

But the MOF definition itself includes the notion of constraints that are 
either "immediate" (checked in real-time) or "deferred" (checked manually 
or perhaps at a transaction boundary), and I think it will be best to 
implement these capabilities directly in peak.model, since such constraints 
are part of the MOF model, and thus part of the UML and CWM metamodels.



> > Ideally, if one could specify constraints in "relatively declarative" 
> form,
> > it would make validating constraints that much easier.  Actually, I 
> kind of
> > wonder if maybe I shouldn't be looking into OCL, because then one could 
> put
> > that into the UML model for an application from the start.
>
>OCL would be great, but expressing constraints in OCL shouldn't be a
>requirement. I think one should be able to express constraints in python
>as well, not free from, but in a form that can be mapped back to OCL.

Right.  The simple way to do that would be to create object classes 
representing OCL constructs, that can be created straightforwardly in a 
Python expression.  Then, if that works out well enough, it should be 
possible to create an OCL translator that creates the same objects.  That 
overall approach worked pretty decently for the design and implementation 
of SkinScript in ZPatterns.

But first, I need to learn OCL.  :)



>Thanks for all the great tips on the config and naming packages. I love
>the fact that configuration variables can be aliased, are in one place
>and that components can easily discover them.

Eh?  I think I know what you mean by "aliased", although technically 
they're not aliased; it's just that you can define one set of properties in 
terms of another.

But "in one place" is something I'd disagree with; you can put them in one 
place if you want, but every object can define its own value for a 
property, e.g.:

class MyComponent(binding.Base):

     someVar = binding.bindTo("otherVar", 
provides=PropertyName('some.property'))

Instances of MyComponent will supply the value of 'someVar' in response to 
a request for 'some.property', so any child component of a MyComponent 
instance will use that value as well.  You will see some instances of this 
pattern's usage in PEAK source code.  You'll also see some things like this:

     someVar = binding.Acquire(PropertyName('other.property'))

What this does, is that if 'someVar' isn't manually set, it will look up 
'other.property' in its parent components.  But the component and its child 
components will use the value of the 'someVar' attribute as their own value 
of 'other.property'.  This means that if you were to do this:

aComponent = MyComponent(someVar="some value")

Then you are effectively setting the value of 'other.property' to "some 
value" for 'aComponent' and its children.

As for whether components can "easily discover them", that's an interesting 
comment.  There isn't any discovery capability for properties; you can look 
one up if you know its name, but there's no way to find out what properties 
do or don't exist.  So, for example, you can't iterate through all possible 
URL scheme names.  The property namespace is potentially infinite.  You 
could do this, for example:

[peak.naming.schemes]
foo.* = # some code here to create a new scheme handler based on the scheme 
name

This would let you use URL schemes of the form "foo.bar:", "foo.baz:", 
etc.  This kind of capability is incompatible with true "discovery", 
although Ty and I have batted around the idea of making it possible to at 
least discover that a rule for 'foo.*' exists.  This would really just be 
*rule* discovery, not *property* discovery.


>  I still have a hell of a
>time maintaining instances of Zope apps where mailhost and database
>connections store configuration variables in themselves.

Yes, that was definitely a "misuse case" for PEAK, in the sense of 
something we definitely didn't want.  Storing that kind of configuration in 
a persistent object as part of an application is a twisty entanglement of 
problem domain and solution domain.


>At the moment PEAK addresses almost all my concerns as an application
>developer and I can really see that it will make maintaining
>applications orders of magnitude easier.

That was certainly the "big idea" behind PEAK, as distinct from TransWarp 
(whose primary mission was application generation).  I'm glad that you're 
already finding it useful.


>There is one big concern that
>is not necessarily the responsibility of a framework like PEAK and has
>more bearing on the implementation of the problem domain namely, 3rd
>party customisation. I mention it in the hope that you can already see a
>pattern that will work well with PEAK.

The config package gives you all the hooks you need to deal with this sort 
of thing, but of course you have to implement your basic application or 
framework in a suitable way.  This could be as simple as using:

config.setupModule()

at the end of a module's code.  This allows people to create "patch 
modules" which can "advise" your module's code, wrapping functions around 
your functions, redefining class hierarchies, etc.  It also allows people 
to "subclass" your module, creating their own instance of it that is 
similarly modified.

You can also define plenty of hook points using configuration variables, e.g.:

class Something(binding.Base):

     OtherClass = binding.bindToProperty('myapp.Something.CollaboratorClass')


[myapp]
Something.CollaboratorClass = importString('some.package:SomeClass')

And of course this is then overrideable in an end-user's application 
configuration file.



>   The best solutions  I've seen so
>far is CMF Skins and by far the best one is the hyperdb used by Roundup
>(a python based issue tracker to which I contributed some code a while
>back - http://roudnup.sf.net/). You most probably know CMF Skins so I'll

Actually, no, I'm not that familiar with it.  I read something a long time 
ago but I've never really used it so it has already dropped out of my brain 
cache.  :)


>When you install Roundup the core is installed in
>lib/python/site-packages.  Creating an instance of an issue tracker is a
>two stage process.  First you add an instance with a command-line script
>to anywhere on the file system. This installs a default class schema and
>html template in the directory you specified.

One way we expect this sort of thing to work with PEAK is that an 
application "instance" would look like this:

#!/bin/env basicApp

...configuration file of arbitrary format goes here...


And the 'basicApp' interpreter would look like this:

#!/bin/env PEAK_App

[some.properties]
#... ".ini" format configuration goes here

[peak.running]
app_config.format = # code to load the schema needed to parse the instance 
config file


Finally, there would be a 'PEAK_App' script, written in Python, provided as 
part of PEAK, which would basically set PEAK_CONFIG to point to argv[1] 
(i.e., the 'basicApp' script), and then invoke the 
'peak.running.app_config.format' schema on argv[2] (the application 
instance config) to create a root component for the application, and 
execute it (probably via a 'run()' method or some such).

The idea here is that you'll use a more user-oriented configuration format 
(such as Zope 3's ZConfig syntax) with an application-specific schema for 
its content.  The configuration file itself becomes the executable for the 
application instance.  Applications themselves are a PEAK .ini-format 
configuration file that specifies the "meta configuration", if you will.

You'll see that this concept allows you to "inherit" from an application 
definition, as well.  For example, you could create this "extendedApp" script:

#!/bin/env PEAK_App

[Load Settings From]
file="/path/to/basicApp"

#... various settings to override those in basicApp


And then you could use "extendedApp" as the interpreter for instances of 
this new extended application definition.

In principle, there is no limit to how deep the inclusions or meta-levels 
can be.  In practice, however, I think that two metalevels of configuration 
(instance and application), with "n" levels of inclusion and the use of a 
PEAK_CONFIG-specified environment configuration, should be plenty of 
flexibility.  :)


>Now I only have to modify the html template for the user, initialise the
>instance and start using my customised instance.

This would of course be possible with the above techniques as well.


>Although the hyperdb is quite flexible wrt to custom schemas and user
>interfaces the downside is that it has relatively few hooks where custom
>methods can be plugged in.
>
>Whereas CMF Skins allows for generous customisation of the user
>interface and provides you with a perfect place to add custom logic
>through scripts and external methods it only solves part of the problem.
>One still needs something that can accommodate 3rd party modifications
>to class schemas that won't be heavily disrupted by upgrades of the app.

I think that simplest way to do this sort of thing in PEAK would probably 
be to have a property that specifies where to import the "model.Model" that 
describes the application's problem-domain object model.  The 
solution-domain components would have 'bindToProperty()' attributes to 
retrieve the model.  Something like this:

class someDM(storage.EntityDM):

     model = binding.bindToProperty('myapp.Model')
     defaultClass = binding.bindTo('model/TheClassIWant')


[myapp]
Model = importString('some.package:MyDomainModel')


Of course, it may be simpler to do it using a wildcard rule, and something 
like:

class someDM(storage.EntityDM):

     defaultClass = binding.bindToProperty('myapp.Model.TheClassIWant')

[myapp]
Model.* = # code to load the class named after the 'myapp.Model.' prefix



>I am still thinking of ways that will work best with PEAK but haven't
>come up with something that satisfies me yet. I think the way you
>illustrated how to include the MySQLConnection without modifying code in
>the storage package might be the answer but I'll have to play with my
>Contact example some more to see how well the patterns works with domain
>entities.

Perhaps the examples I've laid out above will help.


>I've attached the mysql driver if you want to include it - it was
>fairly simple to write since MySQLdb implements the DB API.

I'm afraid MySQL gives Ty and I the creeps, and we don't feel comfortable 
including a driver for it with PEAK for a number of reasons.  Because we do 
try to "think of everything", we are concerned about its locking and 
transactional semantics and how they would interact with PEAK's caching, 
locking, and transactional patterns.  We couldn't say whether or not it 
will work correctly, and so are totally uncomfortable including it in an 
"enterprise application kit".

In the environment where I work, *PostgreSQL* is considered a lightweight 
database for prototyping purposes, and Sybase or Oracle are what you use 
for production applications.  One of our apps has a 30GB database on a RAID 
array, and its dedicated database server hardware is a 64-bit, 
four-processor machine, with an identical warm standby server continuously 
replicated from the first.  PEAK really is intended to work at this sort of 
"enterprise" scale, and using MySQL for production in such an environment 
would be quite insane, IMO.

But Gadfly and SQLite don't pretend to do or be more than what they are, 
and their limitations are both more obvious and more clearly documented, so 
I don't have a problem with including them so that people can do database 
prototyping.  I worry that too many people use MySQL in production 
situations where they *really* shouldn't be, and I don't want to encourage 
that trend.  Most people, however, wouldn't be tempted to use Gadfly or 
SQLite for production unless they really could live with the 
limitations.  But many are tempted by MySQL's claims that speed is all that 
matters and transactions aren't important.  That's why it gives people 
whose companies spend hundreds of thousands of dollars on RAID, 
replication, and hardware redundancy for mission-critical, 24/7 worldwide 
apps, a really creepy feeling about having *anything* to do with it.  :)

So, while I'm sure that you would never use it for an application 
environment it wasn't suited for, I'm not comfortable with making any 
implication that PEAK supports or endorses its use in any way, because 
PEAK's intended audience *does* include people who would abuse MySQL, and 
in the process would give other people an excuse to trash PEAK, Python, and 
Open Source in general for their lack of attention to "enterprise-level" 
concerns.