[PEAK] DecoratorTools, Aspects, and the Trellis
Phillip J. Eby
pje at telecommunity.com
Tue Jul 10 20:57:04 EDT 2007
So at this point I have a rather splendidly-working prototype of the
new Trellis update algorithm, sitting at around 200 lines of code in
the svn.eby-sarna.com repository. However, I don't trust it has 100%
code coverage, because I wrote it basically from scratch, ran it
against an older test suite, and then added some tests to exercise
some of the new functionality.
In the process of doing this, I found that there are lots of ways in
which you could do things that would foul up the implementation, that
were not covered by tests. In other words, I found potential bugs by
inspection, that needed their own proofs of impossibility. I've
added tests for the things I've thought of so far, and fixed the bugs
as I found them, but I suspect there may be one or two more lurking
beneath the surface.
So I'm going to do one more draft of the code, but this time using
TDD, both to be absolutely sure I know how the thing works, and also
to possibly improve the efficiency a bit, although I expect the
algorithm in its simplest form to be a straightforward translation to
C if needed. I've already done some memory-related optimization,
such that Trellis' Cell objects consume less memory than their
PyCells counterparts, and about 1/4th the memory consumed by the
competing "Cellulose" library.
I expect that even a modestly-sized Trellis-based program, using
cells to simultaneously manage its GUI, persistence, and network I/O
will be capable of having thousands or even tens of thousands of
cells in use at any one time, so both space and speed efficiency are
important for a production system.
The other big step that remains to be done on the core Trellis API is
to create the class and function decorators I've sketched here
previously. In the course of planning how to implement them, it's
occurred to me that there are some recurring patterns in how I've
done these sorts of decorators for Chandler, Contextual, and other
projects, and that I can and should add code for these patterns to
the DecoratorTools package, so I don't have to keep recoding them in
an ad-hoc way.
The PEAK-Rules package contains an "Aspect" implementation that's
used to dynamically attach additional information to existing
objects, sort of like you might slap a sticker on the side of an
object in real life. The advantage of using an Aspect over poking
extra attributes onto the object is twofold: 1. you don't have to
come up with a name that doesn't collide with anything else, and 2.
you don't have to write code to *add* the aspect to the object. Just
invoking the aspect constructor is enough to either return the
existing aspect, or create one and attach it somewhere.
And it has occurred to me recently that this bit really belongs in
DecoratorTools rather than PEAK-Rules, because with it, you can write
method decorators that can effectively add "extra attributes" to the
class' instances. Like, oh, say... a map of attribute names to
trellis.Cell objects. :)
The second thing that occurred to me, just today, is that the
decoration of the *class itself* could be done using aspects --
aspects attached to the class, rather than the instance. For
example, in the Trellis case, we also need information about the
rules and default values of the cell attributes.
Unfortunately, regular class decoration as currently implemented in
DecoratorTools is pretty much limited to sticking attributes on the
class or doing things to it after it's already created -- which is
much more limited in what can be done with the class. For example,
it's no use trying to change __slots__ after the class is created,
nor can you change a class' metaclass after creation, unless it was
created using a metaclass other than 'type'.
Granted, for a lot of uses it's not a terrible tragedy. In my
examples, I've been using base classes like trellis.Component to
indicate that some sort of mixin is required, in order for the rules
to work. The trickier bit is ensuring that people don't use
decorators like trellis.rules(), and expect them to work on a
My current solution for things like Contextual's "context.replaces()"
decorator is to add a class decorator that checks the created class
for suitability. This ensures that you'll at least get an error, and
won't be able to proceed without either getting rid of the decorator,
or adding the right base class. It could be nice, however, if you
could just have the decorator add in the mixin.
On the other hand, I can think of some folks who could complain that
not including an explicit base or metaclass is likely to make code
harder to debug or understand, especially if someone's not familiar
with *all* the decorators in a body of code.
So we can probably live with that for the Trellis. The
trellis.Component mixin isn't in fact going to have a lot of stuff
added to the class anyway; just a default __init__ method to handle
keyword arguments, and a metaclass __call__ implementation that will
make sure that any eager cell attributes are activated once __init__
has completely finished running.
So, I'm thinking that the idiom I have now of adding class decorators
to check this, could be simplified a little by adding a class
decoration function like ``class_requires(message, func, *args)``,
that raises TypeError(message) unless func(cls, *args) returns
true. I could then use that in place of the custom code in
Contextual, and Trellis needs it too.
The next pattern I find myself repeating is the need to process
inherited metadata from a class' base classes, either by base order
or MRO order (or the reverse), in order to figure out the combined
metadata for the new class. This usually involves a lot of private
attributes rudely shoved into the class dictionaries, and a loop that
peers into the base classes, looking for whether those attributes are there.
It would be a lot nicer to write code like:
for base_aspect in CellMetadata.by_mro(cls):
# combine some sort of metadata from the base_aspect object
Of course, there are some non-trivial problems to be solved
here. Unlike attributes, aspects can't be inherited (unless I make
them able to be).
Second, at one of the times when we most want to use these class
aspects, the class itself (including its bases, mro, etc.) doesn't
exist yet! So there has to be a way to access and set up the aspect
before the class exists, then have it do any needed setup later.
However, the solution to the second problem also points to a way of
solving the first. If class aspects can be set up lazily, then we
don't have to worry what happens with base classes not having an
aspect, because we can simply create them as we go -- even for
classes that never had the aspect to start with!
(For example, the 'object' class can't be decorated in any normal
way, but an aspect can certainly be attached to it afterwards.)
In order to implement this, class aspects will have to be looked up
in a weakref-keyed dictionary, if they're not already in the class'
dictionary. (Because new-style class dictionaries can't be directly
modified once the class exists.) But, for pre-creation aspect
access, one can create the aspect inside the class dictionary before
the class exists, then add a class decorator to invoke the
So, class aspects could have a method like 'setup_for_class(self,
cls)', that would get called to allow the aspect to initialize
itself, using any data it wants to from the class object. It can
also request aspects of the class' bases or mro, and those aspects
can then lazily come into being -- and have *their* setup_for_class()
methods called, if need be.
So, instead of the pattern I use now, where I have to loop over base
class info and fill in the metadata gaps for classes without it
(which can be tricky at times), I can instead just let every base
have an aspect with appropriate registry attributes, each of which
knows just how to assemble itself from its nearby bases, or its whole
mro if need be.
Yep, that sounds about right. Boy I wish I'd had this idea of
aspects, and especially class aspects, when I was writing the PEAK
core a zillion years ago when Python *2.2* was the latest and
greatest. I would never have created all those special attributes
like __objectsToBeAssembled__ and __class_offers__ and so on. But
then, if I'd had Contextual and the Trellis back then, I wouldn't
have needed to write a good chunk of the PEAK core in the first place. :)
So, once I have aspects moved from PEAK-Rules to DecoratorTools, and
add these new "class aspect" things, I'll be able to start on the
Trellis class-level API, so you can declare the rules and default
values and whatnot.
I'll start on that tomorrow.
More information about the PEAK