[PEAK] PEAK-Rules and PyPy

[Alain Poirier]

Le 17 oct. 2012 à 00:59, PJ Eby <pje at telecommunity.com> a écrit :

> On Tue, Oct 16, 2012 at 12:48 PM, Alain Poirier
> <alain.poirier at net-ng.com> wrote:
>> Le 16 oct. 2012 à 18:26, Marcin Tustin <marcin.tustin at gmail.com> a écrit :
>> 
>>> Please see: http://docs.python.org/library/__builtin__.html?highlight=__builtins__
>> 
>> Thanks Marcin. I see from the 'CPython implementation detail' note, '__builtins__'
>> can be a module or a dict. Ok, so I added a check to use '__builtins__.__dict__' if
>> '__builtins__' is a module and now I've got:
>> 
>>>>>> from peak.rules import predicates
>> Traceback (most recent call last):
>>  File "<console>", line 1, in <module>
>>  File "/private/tmp/p/site-packages/DecoratorTools-1.8-py2.7.egg/peak/util/decorators.py", line 617, in tracer
>>    frm.f_locals[k] = callback(frm,k,v,old_locals)
>>  File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line 276, in callback
>>    register_for_class(None)
>>  File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line 270, in register_for_class
>>    _register_rule(f, pred, context, cls)
>>  File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line 395, in _register_rule
>>    rules.add(parse_rule(Dispatching(gf).engine, pred, context, cls))
>>  File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line 685, in parse_rule
>>    def parse_rule(engine, predicate, context, cls):
>>  File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line 953, in parse_upgrade
>>    predicate, context, cls
>>  File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line 685, in parse_rule
>>    def parse_rule(engine, predicate, context, cls):
>>  File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/predicates.py", line 606, in _parse_string
>>    maybe_bind(ctx.body, bindings), expr, ctx.actiontype, ctx.sequence
>> UnboundLocalError: local variable 'expr' referenced before assignment
>> 
>> which is strange because 'expr' is just defined some lines above.
>> 
>>> CPython implementation detail: Most modules have the name __builtins__ (note the 's') made available as part of their globals. The value of__builtins__ is normally either this module or the value of this modules’s __dict__ attribute. Since this is an implementation detail, it may not be used by alternate implementations of Python.
>>> 
>>> On Tue, Oct 16, 2012 at 12:22 PM, PJ Eby <pje at telecommunity.com> wrote:
>>> On Tue, Oct 16, 2012 at 11:18 AM, Alain Poirier
>>> <alain.poirier at net-ng.com> wrote:
>>>>  File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/predicates.py", line 595, in _parse_string
>>>>    b = CriteriaBuilder(engine.arguments, ctx.localdict, ctx.globaldict, __builtins__)
>>>>  File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/codegen.py", line 334, in __init__
>>>>    dict([(k,self.Const(v)) for k,v in ns.iteritems()]) for ns in namespaces
>>>> AttributeError: 'module' object has no attribute 'iteritems'
>>>> 
>>>> 
>>>> Do you think it could be possible to have a working PyPy version of Peak Rules?
>>> 
>>> It depends. It looks like the problem above is that __builtins__ in
>>> PyPy might be a module instead of a dictionary?  If so, that's
>>> probably a bug in PyPy that needs to be fixed.
>>> 
>>> There are likely to be other problems besides this one, but let's take
>>> them one at a time.  ;-)
> 
> 
> There are bigger fish to fry - the AddOns package (which is a
> dependency) fails all its tests because PyPy doesn't support
> non-string keys in type dictionaries.  DecoratorTools' test suite also
> fails, but AFAICT it's all due to changes in repr() of various
> built-in types (unless PyPy doesn't support classic classes), and
> should be shallow.  AddOns, however, is quite heavily used by
> PEAK-Rules.
> 
> An important BytecodeAssembler test also fails: PyPy has a
> slightly-incompatible bytecode interpreter, and PEAK-Rules abuses a
> weak link in CPython's bytecode to implement a "computed goto"
> operation in generated bytecode.  My guess is that the UnboundLocal
> errors are being caused by how PyPy handles (or more precisely,
> doesn't handle) the computed goto.
> 
> The specific issue is this: in CPython, the END_FINALLY bytecode takes
> a "why" value on top of the stack, plus an extra value as a jump
> offset.  PyPy, however, wants one value on the stack: a special
> interpreter-owned value (that's AFAICT can't be created directly in
> Python) that wraps the reason and the jump offset together.
> 
> Unless there's some way to create these special SContinueLoop objects
> from Python (and I'm guessing there's not), I'd have to write a
> replacement code generator for PyPy that uses a linear search instead
> -- which ironically may make PyPy's predicate dispatch slower than
> CPython's in some cases, unless the JIT can optimize it out enough.
> 
> [pause for some hacking]
> 
> After hacking around a bit, I have a quick and dirty patch (attached)
> that makes most of the tests pass, by fixing up the __builtins__, and
> handling the END_FINALLY issue by using a linear search in the
> innermost dispatch loops.  It isn't suitable for release at the
> moment, because it doesn't check for whether it's running under PyPy,
> and there are still some tests that fail due to shallow minor issues
> like the PyPy repr() differences and hash iteration differences.  Let
> me know if it works for you, and maybe I'll clean it up for release.

I tested today several applications with predicates dispatch rules, for
web stuffs and complex business workflows and they all ran fine, without
any problem (I didn't yet benchmark them). That's really great, thanks.

Now I can only hope for an official release ;)

> Technically, AddOns and BytecodeAssembler need some fixes too, but the
> parts of them that don't work the same under PyPy don't actually get
> used by PEAK-Rules, at least not in the test suite.

Best regards,
Alain