[TransWarp] Basic "storage jar" design

[Phillip J. Eby]

At 03:34 PM 6/29/02 -0500, Phillip J. Eby wrote:
>* oidFor(ob) -- Called by save() operations of other jars to get foreign 
>key values for objects referenced in their states.  Implementation: if 
>ob._p_jar is self, return ob._p_oid, unless _p_oid is None, in which case 
>save the object using oid = ob._p_oid = self.new(ob), and return the 
>oid.  If the _p_jar is NOT self, return self.thunk(ob) to try to translate 
>the reference or create a stub.

Oops.  Actually, if _p_oid is None, it suffices to call self.commit(ob) and 
then return ob._p_oid, rather than duplicate its functionality.  If ob's 
"changed" flag isn't set, and it's not in the "committed" set, then we need 
to set the changed flag and call self.register(ob) before calling 
self.commit(ob).  This is so that fresh new "default" objects which are 
then saved in another object will get saved, even if they haven't been 
changed from their default state.

Also, I didn't include anything in the API to create a new object and point 
its _p_jar to the jar, so it may be non-obvious as to how you could end up 
with an object that has _p_jar==self and _p_oid==None.  It would probably 
be done by a Specialist's newItem() method, in the event it didn't 
immediately assign an ID to the object.  The Specialist would create an 
"empty" instance of the object and point its _p_jar to the target storage.

Perhaps I should add this to the API:

* newItem(klass=None) -- return an instance of klass, initialized to 
default values, set up to be stored in this jar.  If klass is None, use a 
jar-defined default class (or if no default class is defined, raise 
NotImplementedError).

and add this to the abstract methods:

* defaultState(ob) -- return a state of default values to be loaded in 
'ob'.  Default raises NotImplementedError, which means that newItem() will 
be unsupported unless this method is overridden.  This method is allowed to 
set ob._p_oid, if creation-time ID allocation is desired.


These methods would make it clearer how to support creating new instances, 
while taking care of most of the boilerplate code.  Specialists would just 
go straight to newItem() to get the job done.  One important side note: as 
things are right now, you have to *modify* the new item or save a 
persistent reference (e.g. set a foreign key to point to it) to get it to 
be saved.  This is different from ZPatterns, where creation was creation 
was creation.  It's also different from ZODB, which only saves a new object 
if you store a reference to it someplace, no matter how much you modify it.



>* flush() -- Called by multi-row query jars that are about to issue a 
>query against states managed by this jar, to ensure that any changed 
>objects are written to the backing DB, thus preventing queries against 
>stale data.

Alternate key jars might also use this, against their corresponding primary 
key jar.



>Abstract Methods and Attributes
>-------------------------------
>
>...
>
>* ghost(oid, state=None) -- given an oid and optional state, return a 
>ghost (empty instance) of the correct class.  If 'state' is supplied, load 
>it into the object with ob.__setstate__() before returning it.  Note that 
>if 'state' is needed to determine the correct class, but it isn't 
>supplied, your implementation can always call self.load(oid) first, 
>examine the state, then create the class instance and stick the state in 
>it.  It's not a ghost at that point, but what else can you do if you need 
>the state?  The reason this method *must* accept an optional state, even 
>if it doesn't need it, is so that multi-row queries and alternate key 
>lookups can provide their results to preloadState(), preventing a 
>re-retrieval of the same data from the underlying DB.

It might make for simpler code to have the higher-level routines handle the 
__setstate__ if they have a state and the object is in fact a ghost (i.e., 
doesn't have its state set).  On the other hand, it seems possible that 
we'll sometimes have "partial" states passed to "preloadState()", and the 
ghost() method can rectify them.  This concept should probably be documented.



>Transaction.IDataManager Methods
>--------------------------------
>
>(implementations supplied by AbstractJar, similar in nature to the ones in 
>the TW.Database.DataModel.Database class)
>
>...
>
>* commit(ob,txn=None) -- If object's change flag isn't set, just 
>return.  Otherwise, add it to the 'committed' set, remove it from the 
>'dirty' set, and reset its 'changed' flag, after calling self.save() to 
>save it.  (Unless _p_oid is None, in which case do oid = ob._p_oid = 
>self.new(ob); self.cache[oid]=ob instead of calling self.save().)

This should say *then* calling self.save() to save it.  Otherwise, in the 
unlikely event that the process of saving somehow triggers a multi-row 
query or alternate key lookup, which then issues a flush() operation 
against this jar again, we can avoid infinite recursion so long as the 
process of saving doesn't change the object.  (Which would be a bug, so the 
docs for the save() method should reflect this too.)



>Whew!  I think that ought to take care of 95%+ of the boilerplate code 
>that I can think of right now, while

Obviously, I missed a few things on the first go-round.  :)


>I imagine that query jars and alternate key jars will probably have their 
>own boilerplate, with the former being a subclass of AbstractJar and the 
>latter being a different base class.

After giving it a little thought, it seems that alternate key jars based on 
strictly immutable keys can keep a WeakValue cache of the actual objects, 
since this won't extend anything's caching lifetime.  And the corresponding 
primary key jars may want to keep the alternate key caches up-to-date, if 
the alternate key is an important, commonly used one (like UUIDs will be in 
WarpCORE 2-based systems).  This could be implemented by an appropriate 
cache binding in the primary key jar (to a proxy that maintained the 
regular cache plus the alternate key cache(s)), and an override of the 
setstate() method to re-cache the object so that the alternate key cache(s) 
would be updated whenever the object's state was loaded.


>I love the smell of storage jars in the morning...  they smell like 
>persistence.  :)  Only it's afternoon now, hours after I started working 
>on this, so I think I'll go do something else now before my wrists give 
>out.  :)

And now it's off to dinner.  :)