[TransWarp] Basic "storage jar" design

[Roché Compaan]

Hi Phillip

I didn't understand enough of your previous post "PEAK persistence based
on ZODB4, continued" because my brain exploded every second paragraph. I
wasn't too concerned because it seemed that you and yourself first
needed to talk through it :)  The design for the "storage jar" you
outline in this post is beautiful, in its simplicity and its ability to
integrate with and reuse ZODB persistence.

However, I still have some head wounds that needs plaster :)

On Sat, 2002-06-29 at 22:34, Phillip J. Eby wrote:
> Abstract "Storage Jar"
> ======================
> 
> This is a basic design for an abstract implementation of the "storage jar" 
> concept for PEAK/ZODB4.  It can be used as the basis for either a primary 
> key-driven object jar, or a query jar, with appropriate method 
> overrides.  "Alternate key" jars won't have much use for this as a base 
> class, since they don't manage object states but just offer a convenient 
> front-end to retrieving an object from its primary key jar (possibly using 
> the preloadState() mechanism described below).

What will a query jar do?  I assume they will remember query results to
prevent re-querying the underlying database?

> * oidFor(ob) -- Called by save() operations of other jars to get foreign 
> key values for objects referenced in their states.  Implementation: if 
> ob._p_jar is self, return ob._p_oid, unless _p_oid is None, in which case 
> save the object using oid = ob._p_oid = self.new(ob), and return the 
> oid.  If the _p_jar is NOT self, return self.thunk(ob) to try to translate 
> the reference or create a stub.

So if I need to save an instance of "Person" which references an
instance of "Deparment" I can call "oidFor(ADepartment)" on the
DepartmentJar to get the department's id.  When will _p_jar not be self?
Won't all objects returned by the DepartmentJar have their _p_jar set to
the DepartmentJar? 

> Abstract Methods and Attributes
> -------------------------------
> 
> (to be redefined as needed in concrete subclasses of AbstractJar)
> 
> * ghost(oid, state=None) -- given an oid and optional state, return a ghost 
> (empty instance) of the correct class.  If 'state' is supplied, load it 
> into the object with ob.__setstate__() before returning it.  Note that if 
> 'state' is needed to determine the correct class, but it isn't supplied, 
> your implementation can always call self.load(oid) first, examine the 
> state, then create the class instance and stick the state in it.  It's not 
> a ghost at that point, but what else can you do if you need the state?  The 
> reason this method *must* accept an optional state, even if it doesn't need 
> it, is so that multi-row queries and alternate key lookups can provide 
> their results to preloadState(), preventing a re-retrieval of the same data 
> from the underlying DB.

So if an object's state is set to "loaded" by __setstate__ you still
have an empty instance.  The only difference being that it's state is
set.  When does data retrieval happen for this instance, especially
since its "loaded" state will prevent it.  What am I missing?

I understand "ghost" as a state (in memory but state is not loaded) but
I don't quite follow what "ghost" as a method gives you.

"__getitem__" returns an object from the cache or a ghost if its not in
the cache.

"load" will do the actual data retrieval or provide default states.

I can absolutely see the sense in separating direct data retrieval from
getting objects from the cache, into two separate methods.  I having
trouble understanding how the application or Jar will know when to
do which.

> 
> * new(ob) -- save new object 'ob' and return its oid (by generating it or 
> extracting it from state)

What about foreign key constraints in the underlying db?  Not that I
really use them - I think it is the application's responsibility to 
govern relationships between objects. 

> Whew!  I think that ought to take care of 95%+ of the boilerplate code that 
> I can think of right now, while providing all the required interfaces, 
> including what's needed for queries and alternate key jars to help storage 
> jars avoid re-loading of state that's already available, and what's needed 
> to let query jars ensure that their searches are always against up-to-date 
> data in the transaction.

I'm really glad that you put so much thought into avoiding the
re-loading of state since that can either be the big strenth or major
downfall of a persistence framework.

> I love the smell of storage jars in the morning...  they smell like 
> persistence.  :)  Only it's afternoon now, hours after I started working on 
> this, so I think I'll go do something else now before my wrists give out.  :)

For those who don't know, "Jar" comes straight from your fridge.  When
you want to preserve food, you pickle it and put it in a Jar.  The same
goes for objects that you want to persist: you pickle it and put it in a
Jar.  Sometimes it helps to explain what was obvious once an has since
been forgotten.

-- 
Roché Compaan
Upfront Systems                 http://www.upfrontsystems.co.za