[TransWarp] Functionality Gaps

[Phillip J. Eby]

These are some notes on functionality gaps I encountered in PEAK while 
working on the 'bulletins' example app.  This is not necessarily a complete 
list; I expect to add more to it eventually.  :(


Awkward Access to Services
--------------------------

Complex applications involve a lot of services, especially data managers 
(DMs).  DMs that do O-R mapping tend to refer to each other, so they need 
to be able to find one another.  Using a spatial mapping (e.g. 
bindTo('../anotherDM')) leads to a rigid component structure and 
duplication.  Using a configuration lookup means needing to either define 
an arbitrary property or interface to use as a key, and then it still needs 
to be repeated in several places.  Because this is an artificial key, it's 
still awkward to use.  I think the solution to this would be to create a 
new IConfigKey implementation that took a class or classes as 
parameters.  The idea being that you might say things like:

     SomeDM = binding.New('some.dmclass', offerAs=[storage.DMFor(SomeClass)])
     OtherDM = binding.New('other.dmclass', 
offerAs=[storage.DMFor(OtherClass)])

in an application class to instantiate and offer the components.  Then, in 
SomeDM, you might say:

     myOtherDM = binding.bindTo(storage.DMFor(OtherClass))

to create inter-dm links.  Because this is based on the classes themselves, 
the linkages are tied directly to the application's semantics, rather than 
being an artificial key based on a coding convention.  I'm not fond of 
'DMFor()' as a name for this, and I also don't know what to call the 
concept in general.

The second way in which service access is awkward, is accessing a DM 
service from an Element.  If you want an Element to create another Element, 
it needs to call a DM's newItem() method.  Currently, since Elements aren't 
components and don't really support bindings, you'd have to do something like:

     def someElementMethod(self, blah...):

         dm = binding.lookupComponent(self, storage.DMFor(OtherClass))
         other = dm.newItem(OtherClass)

Ugh.  This isn't really fixed by the 'DMFor()', either.  The issue here is 
that the created OtherClass instance won't be saved unless it's associated 
with a DM that can take care of it.  For one-to-one relationships (and 
one-to-many if there is a "pointer" for each item on the "many" side), this 
can actually be handled automatically at object _save() time (using 
'OtherDM.oidFor()'), but for any other relationship, the object *must* have 
its DM established at creation time.

Probably, the simplest way to do this, once we have a 'storage.DMFor()' 
standard, would be to add a new API like:

     def someElementMethod(self, blah...):
         other = storage.newElement(self, OtherClass)

Then, the 'newElement()' method would do the 'binding.lookupComponent(self, 
storage.DMFor(OtherClass))' part.  This at least is now a straightforward 
idiom.

To implement 'DMFor()', we'll need to take into consideration the fact that 
some DM's are polymorphic.  For example, DMs that use pickling techniques 
(e.g. XMI, ZODB, etc.) will likely be used for any of several objects.  It 
seems that we need configuration lookup for these items to work such that 
looking up 'storage.DMFor(X)' will be able to return 'storage.DMFor(Y)' if 
X is a subclass of Y and nothing is registered for 
'storage.DMFor(X)'.  This will require some refactoring of the 
'config.PropertyMap' class, and probably EigenRegistry as well.  :(


Data Managers
-------------

Being able to create storage.EntityDM's sure beats the heck out of not 
having EntityDM's.  But they also leave a lot to be desired in certain areas.

First, there really isn't a good way to handle OID's that are represented 
*in* the model.  For example, in the Bulletins domain model, users have a 
login ID that is also used to retrieve instances.  This pattern actually 
occurs in different ways for all of the Bulletins types, and is probably 
the common case for most business application objects (usually only 
log-like event records are lacking in model-defined primary keys).  Maybe 
there needs to be an EntityDM subclass that fills in more of these standard 
patterns, that you use for non-pickling DMs.

Second, we have negligible cross-DB SQL compatibility.  Because the DBAPI 
allows multiple parameter styles, for example, but also because type 
conversion is difficult.  Also, knowledge of tables, keys, conversions, 
etc. is scattered across a given codebase, such that the code will tend to 
be sensitive even to minor data model changes.  This isn't good.  For any 
application more complex than the Bulletins example, we're going to need 
some kind of abstraction.  The simplest thing that comes to mind is to have 
Table (really, "tabular data") objects that have some select/insert/update 
methods.  We're also going to need some better abstractions for DB-specific 
API variations (e.g. the DBAPI paramstyle), and these are probably going to 
have to be added to the SQLConnection type.  This is a fairly sizeable 
chunk of work, and it'll may have to wait till the 0.6 release cycle.  :(

Third, a standard idiom doesn't exist currently for a DM to handle 
non-existent objects, or to handle the addition of an object with the same 
OID as an existing object.  Currently, retrieving a non-existent OID hands 
you a "broken" object that may or may not work properly.  This is sort of 
like an "infinite Rack" in ZPatterns terms.  I don't know what to do about 
this.  Adding an existence check could result in quite a bit of processing 
overhead.  It seems like we might need two ways to get an item from a DM: a 
way used by other DMs to get "ghosts" (where a broken reference really 
should be considered broken), and a way used by application code when 
looking up keys that come from an external source.