[PEAK] DM refactoring and asynchrony (was Re: PEAK and Twisted, revisited)

[Robert Brewer]

First, I want to introduce myself, since I just joined this list. My
name is Robert Brewer and I've spent the last 9 months or so both
learning Python, and using it to rewrite a VB4 web app that I started
five years ago (and has been in production ever since). During the
rewrite, I basically wrote a poor man's PEAK. :) Now that I know the
*real* PEAK exists, I'm quite interested to learn about its internals,
since I certainly haven't solved all the problems in my framework
(called 'Dejavu': http:/www.aminus.org/rbre/python/dejavu if anyone is
interested). Anyway, on to the email:


Phillip J. Eby wrote:

> This unifying characteristic makes the FO viewpoint a perfect 
> "intermediate representation" or "lingua franca" between different 
> levels of a system.  In fact, because of the ease of expressing
complex 
> constraints in FO systems, and because FO models and constraints can
be 
> specified or rendered as user-friendly English (or other language) 
> expressions, I believe it is in fact preferable to such modelling 
> languages as UML for gathering end-user requirements and defining the 
> domain model for an application.
> 
> The hard part of the design, that I'm chewing on in my spare time, is 
> how to create a Pythonic FO notation, that won't end up looking like 
> Prolog or Lisp!

I'm quite pleased with the technique I settled on for Dejavu (which is
OO, not FO): I realized that lambdas are the perfect "limiting language"
for filtering on object attributes. There are certain actions you cannot
naturally perform within a lambda (such as assignment), and this nicely
constrains them for use as a query construct. For example, to recall all
'Transaction' objects with an Amount attribute greater than 127, you'd
write:

f = logic.Expression(lambda x: x.Amount > 127)
trans = namespace.recall(Transaction, f)

If a Dejavu StorageManager (like a DM) needs to evaluate that, it might
convert the logic.Expression to an SQL string and return matching
objects. The current downside is that I'm using CPython-specific
bytecode hacks both in the formation of the Expression object (mostly
early binding of globals, and distilling constants down), and in the
derivation of the SQL. However, for my purposes, the "Pythonic syntax"
far outweighs the portability concern. By the way, the 'logic' module
(and codewalk.py, upon which it depends) is one folder up on my website
from the above.

> If instead one creates form tools that are based on an FO perspective,

> however, this issue vanishes.  One isn't doing 'x.y' in the first
place, 
> but rather searching for 'x has y of ?', and expecting a collection of

> matching facts.

This could be implemented within my lambda syntax quite easily. All Unit
classes (Unit is the superclass for all Dejavu persistent objects)
currently get a 'has' method to facilitate associations between
Units--the equivalent SQL might be "WHERE EXISTS(SELECT * FROM farClass
WHERE farClass.parent = ID)". In Dejavu, if a DM manages both the near
and far classes, it's free to use that SQL. What I'm getting at is if
you control the core objects, it's no great leap to implement "x has y
of ?" as "x.has(y, 3)" or something similar.

> > This is both more and less granular than a DM.  When you have
related 
> > objects in two DMs, right now you must write code in both to save
and 
> > load the linking attribute(s).

Tell me about it! :)

> The same thing in an FO 
> approach would 
> > correspond to a single fact type, pairing the two objects.  
> So, there 
> > would be only *one* implementation of that fact type, handling both 
> > sides of the relationship.

As I describe above, I leave it to the DM to decide whether or not it
can optimize such relationships. The Python code is the master model.
FWIW, I have a namespace.associate() method, which adds methods to both
the near and far classes to look up each other. For example, to retrieve
all Ledger objects for a given Transaction, you write:

for ledger in aTransaction.Ledger():
    ledger.do_something()

> ...encompass as many facts as you want.  It solves the OO "reporting 
> problem", in fact, because one can create queries of arbitrary 
> complexity without introducing duplicated implementation for the
> more elementary components of those queries.  So, an application can
> define its own compound fact types, and an EC will implement the 
> operations by delegation to the elementary fact types if there isn't
> a custom implementation defined.  Such custom implementations can be 
> useful for providing specially hand-tuned SQL or other algorithms,
> if needed for some critical aspect of an application.

That OO "reporting problem" sounds familiar... :) One thing I've been
playing with on and off since I started is creating new Unit classes
on the fly in the DM; i.e. - you use the same query syntax and form a
Unit dynamically from "whatever columns the DM returns". Seems hackish,
though.

> That is, ...it would have to always provide a 
> sequence, rather than a single object, even if the role in 
> principle is required to always be exactly one object.

Meh. My Namespace class (which I roughly equate to ec?) has darned few
methods, but the two most-used are:
1) .unit(cls, kwfilters), which returns a single Unit or None, and
2) .recall(cls, expression), which returns an iterator.

> This still isn't perfect, in that this API doesn't yet 
> explicitly deal 
> with asynchronous results, or how to "listen" to assertions or 
> retractions, or how to deal with queries when there are 
> assertions or 
> > retractions that haven't been "flushed" to the physical 
> storage scheme.  

Heh. I use that exact term ("flush") in my servers.

Of course, Dejavu isn't anywhere near the complexity of PEAK, or even
address the exact same domains (I make plenty of assumptions about my
data that PEAK can't), but it's interesting to discuss similar problems
across models. I look forward to learning more about about PEAK as time
permits.

Thanks!


Robert Brewer
MIS
Amor Ministries
fumanchu at amor.org