[PEAK] A ridiculously simple mapping layer

[Robert Brewer]

Phillip J. Eby wrote:
> It seems I've been overlooking an obvious, dirt-simple, 
> slap-your-forehead, why-didn't-I-think-of-that way to
> specify an object mapping layer, the bridge from abstract
> model to concrete implementation, because I was too 
> busy trying to figure out how to create registries
> of classes to tables or some such.

I ended up writing my own framework (dejavu) to solve exactly this sort
of problem, because I couldn't find such functionality in any existing
framework; most ORM's assume you have complete control over the storage
mechanism, and will never need to step outside your abstract model. So I
love your idea! :)

> And all along, it could just be something as 
> simple as this:
> 
>      # === mymodel.py =============
> 
>      class Invoice(model.Element):
>          # etc...
> 
>      class Customer(model.Element):
>          # blah blah
> 
> 
>      # === mystorage.py =============
> 
>      class PhysicalSchema1(storage.Schema):
>          Invoice = mapTable('invoices', someField=someConverter, ...)
>          Customer = mapTable('customers', 
> otherField=thus_and_such,...)
>          # etc.
> 
>      class PhysicalSchema2(PhysicalSchema1):
>          # customer is mapped differently in this version of 
> the schema,
>          # but everything else is the same
>          Customer = mapTable('customers', 
> otherField=sturm_und_drang,...)
> 
> 
>      # === myapp.ini =============
> 
>      [Storage Mappings]
>      mymodel = mystorage.PhysicalSchema2   # or 1, or whatever...
> 
> 
>      # === make_invoice.py =============
>      from peak.api import *
>      import mymodel
> 
>      root = config.makeRoot(iniFiles=[("peak","peak.ini"), 
> "myapp.ini"])
> 
>      # create a workspace for the model
>      ws = storage.newWorkspace(root, mymodel)
> 
>      storage.beginTransaction(ws)
> 
>      # look up a customer
>      cust = ws.Customer.get(custno='1234')
> 
>      # create a new invoice
>      inv = ws.Invoice(customer=cust)
> 
>      # etc...
> 
>      storage.commitTransaction(ws)

How interesting that you would choose the form 'ws.Customer' over
'Customer.ws'; that is, why not just stick a reference to the workspace
into each Customer instance when it's created? I assume you're buying
something with your syntax, but I can't figure out what it is just yet.
Do tell. ;)

FWIW, I do the physicalschema-1-vs-2 with a custom StorageManager (what
you might consider a custom DM): basically an adapter over the
Someone-Else's-Code I have to support. (Part of) *my* "myapp.ini":


####
####  Storage Managers
####

[StorageManager_MControl]
Class: dejavu.storage.storeado.StorageManagerADO_MSAccess
Connect: "DSN=MControl;Driver=Microsoft Access Driver
(*.mdb);Dbq=D:\data\mcontrol.mdb;"
Threaded: True

[StorageManager_Blackbaud]
Class: raisersedge.StorageManagerRE76MSSQL

[StorageManager_REMixer]
Class: raisersedge.StorageManagerREMixer
Income Store: Blackbaud
Expense Store: MControl

####
####  Caches
####

[Cache_MControl]
Units:
StorageManager: MControl

[Cache_MControl_Burned]
Units: Action, Agents, Camps, Characteristic, City, Community,
Currencies, Fields, FieldRep, GiftStatuses, Interests, OccupationTypes,
Pastor, PaymentTypes, Plans, ProjectStatuses, ProjectType,
RecipientTypes, Region, Settings, TripStatuses, VehicleMakes
Class: dejavu.sandbox.BurnedCache
StorageManager: MControl

[Cache_Engines]
Units: UnitCollection, UnitEngine, UnitEngineRule
Class: dejavu.sandbox.PersistentCache
StorageManager: MControl
Lifetime: 10 minutes

[Cache_Blackbaud]
Units: Address, BombFlag, ConstituentCode, Contact, Directory,
DirectoryAddress, DirectoryNote, Education, MailingList, Participant,
Relation, Segment, TripParticipant
StorageManager: Blackbaud

[Cache_Blackburn]
Units: AddressTypes, ConstituentCodeTypes, ContactTypes, NoteTypes,
RelationContactTypes, RelationTypes, AppealSegments, Countries, Fund,
Genders, MailingListTypes, Schools, SchoolTypes, States, Suffixes,
Titles
Class: dejavu.sandbox.BurnedCache
StorageManager: Blackbaud

[Cache_Transactions]
Units: Batch, Ledger, Transaction
StorageManager: REMixer

[Cache_Misc]
Units: FieldDashboardSumSet
Class: dejavu.sandbox.PersistentCache
Lifetime: 15 minutes
# Notice no StorageManager--these only persist in memory.

> While not all of these mechanisms are interchangeable at the 
> top-level of 
> code (since one would have to specify the file to load, when 
> to save it, 
> etc., while others do not), any code that receives an already-opened 
> workspace object is not affected by these mapping issues.  
> Only the code 
> that manages the workspace object's lifecycle has to know 
> anything about 
> the overall persistence model.  (Assuming that we have a 
> query language 
> that can be applied to the classes exposed by a workspace, 
> and it operates 
> on storage interfaces supplied by the implementation-specific 
> subclasses.)

I did a lot of refactoring over the weekend, so these issues are much on
my mind. A big concern for me recently has been transparent persistence
mechanisms; that is, as you say, "only the code that manages the
workspace object's lifecycle has to know anything about the overall
persistence model." I went a step further and made a Sandbox class which
gets doled out to UI consumers as needed (for example, web UI's obtain
it from the Application object upon each page request). The cool bit is
that the sandbox doesn't control persistence, each object cache within
the sandbox can have its own persistence mechanism: some persist
per-request (and thereby avoid a lot of multithread concurrency issues),
some persist forever, some get swept out on a schedule, some use the
Borg pattern to be shared among consumers. More subclasses to follow. :)
For example (now that I've got the basic architecture), I need to build
a traditional queue-on-read, lock-on-write cache. And nothing's stopping
me from writing an admin tool which flushes a given cache and replaces
it with another (with a different persistence style) on the fly. Tuning
deployed apps when concurrency problems arise? Going to be a lot
simpler, now. :) I think you'll find the same if you can isolate it as
you describe.


Robert Brewer
MIS
Amor Ministries
fumanchu at amor.org