[PEAK] PEAK-Rules for Python3

[PJ Eby]

On Fri, Apr 3, 2015 at 3:32 PM, Cara <ceridwen.mailing.lists at gmail.com> wrote:
> Despite what I said last time, I did a little bit more on this.

Thanks!


> This is also what I found.  Note that trying to run the tests on 2.7
> will fail because of the missing ez_setup.

Not sure what you mean about missing ez_setup, but I'll look into it.
I don't use the newer versions of setuptools myself yet, so I haven't
run into that problem yet.  ;-)


> I'm aware of the problem and the PEPs that try to solve it, but I didn't
> know it was the issue here.  I'd like to see the functionality added,
> but CPython development is CPython development.  I can throw in a word
> on python-dev if you want, for all that would do.

At this point, it's not necessary.  I had the discussion myself, and
the resolution seems to be that there might be a new protocol added
later, but for now, just go ahead and hook __build_class__ to get the
old capabilities back, then change it to something standardized later.
PyPy in any event has __build_class__ in its Python 3, so it won't be
a problem there, either.


>> This isn't actually possible, anyway: you would not be able to define
>> generic function methods in a class body unless you added an explicit
>> metaclass or class decorator to the class, and rewrote the API or
>> dropped the feature.  (I guess I should say, it isn't possible while
>> retaining the features of class-level AddOns or of defining gf methods
>> in class bodies.)
>
> I'm not sure I understand why this is.

PEAK-Rules' decorators for methods defined in classes add in the
current class as a restriction on applicability of the method; to do
this, they need the class object, but that object can only be obtained
*after* the class is defined.  They do this by *invisbly* decorating
the class, so they can be notified when the class is created.  This is
why the whole PEP business was needed: the mechanism used for this
invisible class decoration in Python 2, went away in Python 3.


> I've gotten the Python 3 port to the point where everything installs and
> most of the tests work.  Extremes, SymbolType, and Importing pass all
> their tests in both 2 and 3.  There's an AmbiguousMethods error in
> PEAK-Rules,
>
> ("peak.rules.core.AmbiguousMethods: ([Method(<built-in function
> issubclass>, (<type 'type'>, <type 'type'>), 2, None), Method(<built-in
> function issubclass>, (<type 'type'>, <type 'type'>), 3, None)], (<class
> 'peak.rules.core.TypeEngine'>, <class 'peak.rules.core.TypeEngine'>),
> {})"
>
> which I think comes from this section of code that I don't
> understand in core.py:
>
> ClassType = type
> when(implies, (type,      (ClassType, type) ))(issubclass)
> when(implies, (ClassType,  ClassType        ))(issubclass)
> when(implies, (istype,     istype           ))(lambda s1,s2:
>     s1==s2 or (s1.type is not s2.type and s1.match and not s2.match))
> when(implies, (istype, type) )(lambda s1,s2: s1.match and
>     implies(s1.type,s2))
>
> I added the ClassType = type assignment to bypass shallow NameErrors.

In the event that `ClassType is type`, only one of the
when(...)(issubclass) lines is needed, and the other is redundant.
i.e. a simple:

    when(implies, (type, type))(issubclass)

is all that's necessary.

Basically, this code says, "when you want to know if 'a' implies 'b',
and both 'a' and 'b' are instances of 'type', call `issubclass(a, b)`
to find out."  In Python 2, classic classes are ClassType instances,
not `type` instances.  And you can have new-style classes that
subclass a ClassType.  That's why two declarations are needed: the
first says, "if you have a new-style class, and want to check if it
inherits from either a classic or new-style class, use issubclass."
The second says, "If you have a classic class, and you want to check
if it inherits from a classic class, you can also use issubclass."
(But a classic class can never be a subclass of a new-style class, so
that's not listed anywhere.)

The `implies()` function in the core is used to tell whether some
criterion implies another criterion.  In the core, the main criteria
are class objects and `istype` objects (hard type checks).  But the
full predicate system has many more kinds of things, including
expressions and range checks and whatnot, so it adds more rules to the
`implies()` generic function later to cover those other kinds of
objects as they are introduced.

Yes, PEAK-Rules is written using its own generic functions.  ;-)


> There's a collection of errors in Python 3 for the AddOns and
> DecoratorTools tests that I suspect relate to the class creation and
> decorators issued you mentioned: the tests involving Demo in the doc
> tests and TypeErrors in testMixedMetas, testSingleExplicitMeta, and
> testOrder, plus some shallow errors.  From what you said, the changes to
> metaclass handling make any kind of a direct port impossible because the
> functionality isn't there on Python 3?

Yeah, DecoratorTools needs some significant work to port to Python 3,
in order for the class-decorating and metaclass-related features to
work, and PEAK-Rules actually does use some of them, directly or via
AddOns.  I'm not sure how many of the test failures actually represent
sticking points for PEAK-Rules, though.


> There's one error in testAlreadyTracing that appears on both Python 2
> and 3:
>
> Traceback (most recent call last):
>   File "/home/cara/code/peak-rules/DecoratorTools/test_decorators.py",
> line 116, in testAlreadyTracing
>     'decorate_assignment', 'enclosing_frame', '<lambda>',
> 'failUnlessEqual',
> AssertionError: Lists differ: ['tes[66 chars]>', 'assertEqual',
> '_getAssertEqualityFunc', '[363 chars]ual'] != ['tes[66 chars]>',
> 'failUnlessEqual', 'decorate_assignment', [171 chars]ual']

Yeah, as of 2.7 it looks like part of unittest was refactored, and
that test was based on the call stack in pre-2.7.  I've already seen
it in 2.7 and consider it a shallow failure; if it fails the same way
on 3.x then it's probably nothing to worry about.


> And then there's a large collection of errors in BytecodeAssembler on
> Python 2 I haven't assessed,

Huh?  I just ran "python27 setup.py test" on a fresh checkout of
BytecodeAssembler and it comes back clean.

> plus it segfaults (no surprise there) on
> Python 3.

You'll probably need to comment out some tests to find out where it breaks.


>I posted the log for Python 3 here:
> http://pastebin.com/UhCfyHJf .

Interesting.  So, looks like SymbolTypes and Extremes are good, and
most of DecoratorTools' failures are expected or shallow, and that
many of AddOns' failures are just the direct effect of DecoratorTools'
expected failures.  The unsortable types thing is a shallow failure of
sorts (no pun intended): dir() doesn't work right with non-string keys
and the test needs to do the verification it's doing in some other
way.

I'm actually very encouraged by this, though, since all the
DecoratorTools failures except the unittest call stack thing are due
to the metaclass issue.  So if I write a workaround for that,
DecoratorTools should work, and then AddOns ought to work, and maybe
PEAK-Rules will work, right up to the point where it needs
BytecodeAssembler, anyway.  ;-)

So...  I've checked in some changes to DecoratorTools, and I have here
a proposed patch to make the class stuff work on Python 3.  Add the
following to the end of decorators.py, and let me know if it works:

try:
    old_build_class = __build_class__
except NameError:
    pass
else:
    def apply_decorators(cls, advice):
        if metaclass_is_decorator(advice):
            cls = advice.callback(
                apply_decorators(cls, advice.previousMetaclass)
            )
        return cls

    def py3_build_class(func, name, *bases, **kw):
        cls = old_build_class(func, name, *bases, **kw)
        advice = cls.__dict__.get('__metaclass__', None)
        cls = apply_decorators(cls, advice)
        if '__metaclass__' in cls.__dict__
            try:
                del cls.__metaclass__
            except TypeError:
                pass
        return cls

    __builtins__.__build_class__ = py3_build_class


This is basically a monkeypatch to post-process class decorators on
Python 3, by detecting DecoratorTools' metaclass-decorator protocol
and only applying the decoration part, not the metaclass part.  I
would be very interested in seeing what test failures are left after
you add this, and the code I just checked in to fix the
unittest-related failure.  If this patch above works, then we should
also see a lot fewer failures for AddOns as well.