Additions are marked like this.

Python bytecodes, this module also offers an extensible mini-AST framework for

Please see the `BytecodeAssembler reference manual`_ for more details.

 

.. _BytecodeAssembler reference manual: http://peak.telecommunity.com/DevCenter/BytecodeAssembler#toc

 

 

Changes since version 0.5.2:

 

* Symbolic disassembly with full emulation of backward-compatible

  ``JUMP_IF_TRUE`` and ``JUMP_IF_FALSE`` opcodes on Python 2.7 -- tests now

  run clean on Python 2.7.

 

* Support for backward emulation of Python 2.7's ``JUMP_IF_TRUE_OR_POP`` and

  ``JUMP_IF_FALSE_OR_POP`` instructions on earlier Python versions; these

  emulations are also used in BytecodeAssembler's internal code generation,

  for maximum performance on 2.7+ (with no change to performance on older

  versions).

 

Changes since version 0.5.1:

 

* Initial support for Python 2.7's new opcodes and semantics changes, mostly

  by emulating older versions' behavior with macros. (0.5.2 is really just

  a quick-fix release to allow packages using BytecodeAssembler to run on 2.7

  without having to change any of their code generation; future releases will

  provide proper support for the new and changed opcodes, as well as a test

  suite that doesn't show spurious differences in the disassembly listings

  under Python 2.7.)

 

Changes since version 0.5:

 

* Fix incorrect stack size calculation for ``MAKE_CLOSURE`` on Python 2.5+

 

Changes since version 0.3:

 

* New node types:

 

  * ``For(iterable, assign, body)`` -- define a "for" loop over `iterable`

 

  * ``UnpackSequence(nodes)`` -- unpacks a sequence that's ``len(nodes)`` long,

    and then generates the given nodes.

 

  * ``LocalAssign(name)`` -- issues a ``STORE_FAST``, ``STORE_DEREF`` or

    ``STORE_LOCAL`` as appropriate for the given name.

 

  * ``Function(body, name='<lambda>', args=(), var=None, kw=None, defaults=())``

    -- creates a nested function from `body` and puts it on the stack.

 

  * ``If(cond, then_, else_=Pass)`` -- "if" statement analogue

 

  * ``ListComp(body)`` and ``LCAppend(value)`` -- implement list comprehensions

 

  * ``YieldStmt(value)`` -- generates a ``YIELD_VALUE`` (plus a ``POP_TOP`` in

    Python 2.5+)

 

* ``Code`` objects are now iterable, yielding ``(offset, op, arg)`` triples,

  where `op` is numeric and `arg` is either numeric or ``None``.

 

* ``Code`` objects' ``.code()`` method can now take a "parent" ``Code`` object,

  to link the child code's free variables to cell variables in the parent.

 

* Added ``Code.from_spec()`` classmethod, that initializes a code object from a

  name and argument spec.

 

* ``Code`` objects now have a ``.nested(name, args, var, kw)`` method, that

  creates a child code object with the same ``co_filename`` and the supplied

  name/arg spec.

 

* Fixed incorrect stack tracking for the ``FOR_ITER`` and ``YIELD_VALUE``

  opcodes

 

* Ensure that ``CO_GENERATOR`` flag is set if ``YIELD_VALUE`` opcode is used

 

* Change tests so that Python 2.3's broken line number handling in ``dis.dis``

  and constant-folding optimizer don't generate spurious failures in this

  package's test suite.

 

 

Changes since version 0.2:

 

* Added ``Suite``, ``TryExcept``, and ``TryFinally`` node types

 

* Added a ``Getattr`` node type that does static or dynamic attribute access

  and constant folding

 

* Fixed ``code.from_function()`` not copying the ``co_filename`` attribute when

  ``copy_lineno`` was specified.

 

* The ``repr()`` of AST nodes doesn't include a trailing comma for 1-argument

  node types any more.

 

* Added a ``Pass`` symbol that generates no code, a ``Compare()`` node type

  that does n-way comparisons, and ``And()`` and ``Or()`` node types for doing

  logical operations.

 

* The ``COMPARE_OP()`` method now accepts operator strings like ``"<="``,

  ``"not in"``, ``"exception match"``, and so on, as well as numeric opcodes.

  See the standard library's ``opcode`` module for a complete list of the

  strings accepted (in the ``cmp_op`` tuple). ``"<>"`` is also accepted as an

  alias for ``"!="``.

 

* Added code to verify that forward jump offsets don't exceed a 64KB span, and

  support absolute backward jumps to locations >64KB.

 

Changes since version 0.1:

 

* Constant handling has been fixed so that it doesn't confuse equal values of

  differing types (e.g. ``1.0`` and ``True``), or equal unhashable objects

  (e.g. two empty lists).

 

* Removed ``nil``, ``ast_curry()`` and ``folding_curry()``, replacing them with

  the ``nodetype()`` decorator and ``fold_args()``; please see the docs for

  more details.

 

* Added stack tracking across jumps, globally verifying stack level prediction

  consistency and automatically rejecting attempts to generate dead code. It

  should now be virtually impossible to accidentally generate bytecode that can

  crash the interpreter. (If you find a way, let me know!)

 

fully supported. Also note the following limitations:

* Jumps to as-yet-undefined labels cannot span a distance greater than 65,535

  bytes.

* The ``dis()`` function in Python 2.3 has a bug that makes it show incorrect

  greater than 255. (To work around this, the test_suite uses a later version

  of ``dis()``, but do note that it may affect your own tests if you use

  ``dis()`` with Python 2.3 and use widely separated line numbers.)

  

.. _toc:

    

Finally, code objects are also iterable, yielding ``(offset, opcode, arg)``

tuples, where `arg` is ``None`` for opcodes with no arguments, and an integer

otherwise::

 

    >>> import peak.util.assembler as op

    >>> list(c) == [

    ... (0, op.LOAD_CONST, 1),

    ... (3, op.RETURN_VALUE, None)

    ... ]

    True

 

This can be useful for testing or otherwise inspecting code you've generated.

 

 

Symbolic Disassembler

=====================

 

Python's built-in disassembler can be verbose and hard to read when inspecting

complex generated code -- usually you don't care about bytecode offsets or

line numbers as much as you care about labels, for example.

 

So, BytecodeAssembler provides its own, simplified disassembler, which we'll

be using for more complex listings in this manual::

 

    >>> from peak.util.assembler import dump

 

Some sample output, that also showcases some of BytecodeAssembler's

`High-Level Code Generation`_ features::

 

    >>> c = Code()

    >>> from peak.util.assembler import Compare, Local

    >>> c.return_(Compare(Local('a'), [('<', Local('b')), ('<', Local('c'))]))

    >>> dump(c.code())

                    LOAD_FAST 0 (a)

                    LOAD_FAST 1 (b)

                    DUP_TOP

                    ROT_THREE

                    COMPARE_OP 0 (<)

                    JUMP_IF_FALSE L1

                    POP_TOP

                    LOAD_FAST 2 (c)

                    COMPARE_OP 0 (<)

                    JUMP_FORWARD L2

            L1: ROT_TWO

                    POP_TOP

            L2: RETURN_VALUE

 

As you can see, the line numbers and bytecode offsets have been dropped,

making it esier to see where the jumps go. (This also makes doctests more

robust against Python version changes, as ``dump()`` has some extra code to

make conditional jumps appear consistent across the major changes that were

made to conditional jump instructions between Python 2.6 and 2.7.)

 

    >>> where = c.here() # get a location near the start of the code

    >>> c.DUP_TOP()

    >>> c.POP_TOP()

    >>> dump(c.code())

                    LOAD_CONST 1 (42)

            L1: DUP_TOP

                    POP_TOP

                    JUMP_ABSOLUTE L1

    >>> c.LOAD_CONST(99)

    >>> forward = c.JUMP_IF_TRUE() # create a jump and a forward reference

    >>> c.POP_TOP()

    >>> dump(c.code())

                    LOAD_CONST 1 (99)

                    JUMP_IF_TRUE L1

                    LOAD_CONST 2 (42)

                    POP_TOP

            L1: LOAD_CONST 3 (23)

                    RETURN_VALUE

    >>> import sys

    >>> if sys.version>='2.5':

    ... c.BUILD_TUPLE(2) # In Python 2.5+, free vars must be in a tuple

    >>> c.LOAD_CONST(None) # in real code, this'd be a Python code constant

    >>> c.MAKE_CLOSURE(0,2) # no defaults, 2 free vars in the new function

 

    >>> c.stack_size # This will be 1, no matter what Python version

    1

 

The ``COMPARE_OP`` method takes an argument which can be a valid comparison

integer constant, or a string containing a Python operator, e.g.::

 

    >>> c = Code()

    >>> c.LOAD_CONST(1)

    >>> c.LOAD_CONST(2)

    >>> c.COMPARE_OP('not in')

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (1)

                  3 LOAD_CONST 2 (2)

                  6 COMPARE_OP 7 (not in)

 

The full list of valid operator strings can be found in the standard library's

``opcode`` module. ``"<>"`` is also accepted as an alias for ``"!="``::

 

    >>> c.LOAD_CONST(3)

    >>> c.COMPARE_OP('<>')

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (1)

                  3 LOAD_CONST 2 (2)

                  6 COMPARE_OP 7 (not in)

                  9 LOAD_CONST 3 (3)

                 12 COMPARE_OP 3 (!=)

Note that although some values of different types may compare equal to each

other, ``Code`` objects will not substitute a value of a different type than

the one you requested::

 

    >>> c = Code()

    >>> c(1, True, 1.0, 1L) # equal, but different types

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (1)

                  3 LOAD_CONST 2 (True)

                  6 LOAD_CONST 3 (1.0)

                  9 LOAD_CONST 4 (1L)

If the value wrapped in a ``Const`` is not hashable, it is compared by identity

rather than value. This prevents equal mutable values from being reused by

accident, e.g. if you plan to mutate the "constant" values later::

 

    >>> c = Code()

    >>> c(Const([]), Const([])) # equal, but not the same object!

    >>> dis(c.code())

      0 0 LOAD_CONST 1 ([])

                  3 LOAD_CONST 2 ([])

 

Thus, although ``Const`` objects hash and compare based on equality for

hashable types::

 

    >>> hash(Const(3)) == hash(3)

    True

    >>> Const(3)==Const(3)

    True

 

They hash and compare based on object identity for non-hashable types::

 

    >>> c = Const([])

    >>> hash(c) == hash(id(c.value))

    True

    >>> c == Const(c.value) # compares equal if same object

    True

    >>> c == Const([]) # but is not equal to a merely equal object

    False

 

 

``Suite`` and ``Pass``

----------------------

 

On occasion, it's helpful to be able to group a sequence of opcodes,

expressions, or statements together, to be passed as an argument to other node

types. The ``Suite`` node type accomplishes this::

 

    >>> from peak.util.assembler import Suite, Pass

 

    >>> c = Code()

    >>> c.return_(Suite([Const(42), Code.DUP_TOP, Code.POP_TOP]))

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (42)

                  3 DUP_TOP

                  4 POP_TOP

                  5 RETURN_VALUE

 

And ``Pass`` is a shortcut for an empty ``Suite``, that generates nothing::

 

    >>> Suite([])

    Pass

 

    >>> c = Code()

    >>> c(Pass)

    >>> c.return_(None)

    >>> dis(c.code())

      0 0 LOAD_CONST 0 (None)

                  3 RETURN_VALUE

 

The ``LocalAssign`` node type takes a name, and stores a value in a local

variable::

 

    >>> from peak.util.assembler import LocalAssign

    >>> c = Code()

    >>> c(42, LocalAssign('x'))

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (42)

                  3 STORE_FAST 0 (x)

 

set), local variables will be referenced using ``LOAD_NAME`` and ``STORE_NAME``

instead of ``LOAD_FAST`` and ``STORE_FAST``, and if the referenced local name

is a "cell" or "free" variable, ``LOAD_DEREF`` and ``STORE_DEREF`` are used

instead::

    >>> c( LocalAssign('x'), LocalAssign('y'), LocalAssign('z') )

                  9 STORE_NAME 0 (x)

                 12 STORE_DEREF 0 (y)

                 15 STORE_DEREF 1 (z)

 

 

Obtaining Attributes

--------------------

 

The ``Getattr`` node type takes an expression and an attribute name. The

attribute name can be a constant string, in which case a ``LOAD_ATTR`` opcode

is used, and constant folding is done if possible::

 

    >>> from peak.util.assembler import Getattr

 

    >>> c = Code()

    >>> c(Getattr(Local('x'), '__class__'))

    >>> dis(c.code())

      0 0 LOAD_FAST 0 (x)

                  3 LOAD_ATTR 0 (__class__)

 

 

    >>> Getattr(Const(object), '__class__') # const expression, const result

    Const(<type 'type'>)

 

Or the attribute name can be an expression, in which case a ``getattr()`` call

is compiled instead::

 

    >>> c = Code()

    >>> c(Getattr(Local('x'), Local('y')))

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (<built-in function getattr>)

                  3 LOAD_FAST 0 (x)

                  6 LOAD_FAST 1 (y)

                  9 CALL_FUNCTION 2

    >>> dis(c.code())

      0 0 LOAD_CONST 0 (None)

                  3 RETURN_VALUE

 

    >>> c = Code()

 

 

``If`` Conditions

-----------------

 

The ``If()`` node type generates conditional code, roughly equivalent to a

Python if/else statement::

 

    >>> from peak.util.assembler import If

    >>> c = Code()

    >>> c( If(Local('a'), Return(42), Return(55)) )

    >>> dump(c.code())

                    LOAD_FAST 0 (a)

                    JUMP_IF_FALSE L1

                    POP_TOP

                    LOAD_CONST 1 (42)

                    RETURN_VALUE

            L1: POP_TOP

                    LOAD_CONST 2 (55)

                    RETURN_VALUE

 

However, it can also be used like a Python 2.5+ conditional expression

(regardless of the targeted Python version)::

 

    >>> c = Code()

    >>> c( Return(If(Local('a'), 42, 55)) )

    >>> dump(c.code())

                    LOAD_FAST 0 (a)

                    JUMP_IF_FALSE L1

                    POP_TOP

                    LOAD_CONST 1 (42)

                    JUMP_FORWARD L2

            L1: POP_TOP

                    LOAD_CONST 2 (55)

            L2: RETURN_VALUE

 

 

Note that ``If()`` does *not* do constant-folding on its condition; even if the

condition is a constant, it will be tested at runtime. This avoids issues with

using mutable constants, e.g.::

 

    >>> c = Code()

    >>> c(If(Const([]), 42, 55))

    >>> dump(c.code())

                    LOAD_CONST 1 ([])

                    JUMP_IF_FALSE L1

                    POP_TOP

                    LOAD_CONST 2 (42)

                    JUMP_FORWARD L2

            L1: POP_TOP

                    LOAD_CONST 3 (55)

    >>> c.LOAD_CONST(99)

    >>> forward = c.JUMP_IF_FALSE()

    >>> c( 1, Code.POP_TOP, forward, Return(3) )

    >>> dump(c.code())

                    LOAD_CONST 1 (99)

                    JUMP_IF_FALSE L1

                    LOAD_CONST 2 (1)

                    POP_TOP

            L1: LOAD_CONST 3 (3)

                    RETURN_VALUE

    >>> c(99, skip.JUMP_IF_FALSE, 1, Code.POP_TOP, skip, Return(3))

    >>> dump(c.code())

                    LOAD_CONST 1 (99)

                    JUMP_IF_FALSE L1

                    LOAD_CONST 2 (1)

                    POP_TOP

            L1: LOAD_CONST 3 (3)

                    RETURN_VALUE

More Conditional Jump Instructions

----------------------------------

 

In Python 2.7, the traditional ``JUMP_IF_TRUE`` and ``JUMP_IF_FALSE``

instructions were replaced with four new instructions that either conditionally

or unconditionally pop the value being tested. This was done to improve

performance, since virtually all conditional jumps in Python code pop the

value on one branch or the other.

 

To provide better cross-version compatibility, BytecodeAssembler emulates the

old instructions on Python 2.7 by emitting a ``DUP_TOP`` followed by a

``POP_JUMP_IF_FALSE`` or ``POP_JUMP_IF_TRUE`` instruction.

 

However, since this decreases performance, BytecodeAssembler *also* emulates

Python 2.7's ``JUMP_IF_FALSE_OR_POP`` and ``JUMP_IF_FALSE_OR_TRUE`` opcodes

on *older* Pythons::

 

    >>> c = Code()

    >>> l1, l2 = Label(), Label()

    >>> c(Local('a'), l1.JUMP_IF_FALSE_OR_POP, Return(27), l1)

    >>> c(l2.JUMP_IF_TRUE_OR_POP, Return(42), l2, Code.RETURN_VALUE)

    >>> dump(c.code())

                    LOAD_FAST 0 (a)

                    JUMP_IF_FALSE L1

                    POP_TOP

                    LOAD_CONST 1 (27)

                    RETURN_VALUE

            L1: JUMP_IF_TRUE L2

                    POP_TOP

                    LOAD_CONST 2 (42)

                    RETURN_VALUE

            L2: RETURN_VALUE

 

This means that you can immediately begin using the "or-pop" variations, in

place of a jump followed by a pop, and BytecodeAssembler will use the faster

single instruction automatically on Python 2.7+.

 

BytecodeAssembler *also* supports using Python 2.7's conditional jumps

that do unconditional pops, but currently cannot emulate them on older Python

versions, so at the moment you should use them only when your code requires

Python 2.7.

 

(Note: for ease in doctesting across Python versions, the ``dump()`` function

*always* shows the code as if it were generated for Python 2.6 or lower, so

if you need to check the *actual* bytecodes generated, you must use Python's

``dis.dis()`` function instead!)

 

 

N-Way Comparisons

-----------------

 

You can generate N-way comparisons using the ``Compare()`` node type::

 

    >>> from peak.util.assembler import Compare

 

    >>> c = Code()

    >>> c(Compare(Local('a'), [('<', Local('b'))]))

    >>> dis(c.code())

      0 0 LOAD_FAST 0 (a)

                  3 LOAD_FAST 1 (b)

                  6 COMPARE_OP 0 (<)

 

3-way comparisons generate code that's a bit more complex. Here's a three-way

comparison (``a<b<c``)::

 

    >>> c = Code()

    >>> c.return_(Compare(Local('a'), [('<', Local('b')), ('<', Local('c'))]))

    >>> dump(c.code())

                    LOAD_FAST 0 (a)

                    LOAD_FAST 1 (b)

                    DUP_TOP

                    ROT_THREE

                    COMPARE_OP 0 (<)

                    JUMP_IF_FALSE L1

                    POP_TOP

                    LOAD_FAST 2 (c)

                    COMPARE_OP 0 (<)

                    JUMP_FORWARD L2

            L1: ROT_TWO

                    POP_TOP

            L2: RETURN_VALUE

 

And a four-way (``a<b>c!=d``)::

 

    >>> c = Code()

    >>> c.return_(

    ... Compare( Local('a'), [

    ... ('<', Local('b')), ('>', Local('c')), ('!=', Local('d'))

    ... ])

    ... )

    >>> dump(c.code())

                    LOAD_FAST 0 (a)

                    LOAD_FAST 1 (b)

                    DUP_TOP

                    ROT_THREE

                    COMPARE_OP 0 (<)

                    JUMP_IF_FALSE L1

                    POP_TOP

                    LOAD_FAST 2 (c)

                    DUP_TOP

                    ROT_THREE

                    COMPARE_OP 4 (>)

                    JUMP_IF_FALSE L1

                    POP_TOP

                    LOAD_FAST 3 (d)

                    COMPARE_OP 3 (!=)

                    JUMP_FORWARD L2

            L1: ROT_TWO

                    POP_TOP

            L2: RETURN_VALUE

 

 

Sequence Unpacking

------------------

 

The ``UnpackSequence`` node type takes a sequence of code generation targets,

and generates an ``UNPACK_SEQUENCE`` of the correct length, followed by the

targets::

 

    >>> from peak.util.assembler import UnpackSequence

    >>> c = Code()

    >>> c((1,2), UnpackSequence([LocalAssign('x'), LocalAssign('y')]))

    >>> dis(c.code()) # x, y = 1, 2

      0 0 LOAD_CONST 1 (1)

                  3 LOAD_CONST 2 (2)

                  6 BUILD_TUPLE 2

                  9 UNPACK_SEQUENCE 2

                 12 STORE_FAST 0 (x)

                 15 STORE_FAST 1 (y)

 

 

Yield Statements

----------------

 

The ``YieldStmt`` node type generates the necessary opcode(s) for a ``yield``

statement, based on the target Python version. (In Python 2.5+, a ``POP_TOP``

must be generated after a ``YIELD_VALUE`` in order to create a yield statement,

as opposed to a yield expression.) It also sets the code flags needed to make

the resulting code object a generator::

 

    >>> from peak.util.assembler import YieldStmt

    >>> c = Code()

    >>> c(YieldStmt(1), YieldStmt(2), Return(None))

    >>> list(eval(c.code()))

    [1, 2]

 

 

 

    NotAConstant: Local('x')

    NotAConstant: Global('y')

Folding Function Calls

----------------------

 

    Const(<type 'int'>)

    Const(<type 'dict'>)

Logical And/Or

--------------

 

You can evaluate logical and/or expressions using the ``And`` and ``Or`` node

types::

 

    >>> from peak.util.assembler import And, Or

 

    >>> c = Code()

    >>> c.return_( And([Local('x'), Local('y')]) )

    >>> dump(c.code())

                    LOAD_FAST 0 (x)

                    JUMP_IF_FALSE L1

                    POP_TOP

                    LOAD_FAST 1 (y)

            L1: RETURN_VALUE

 

    >>> c = Code()

    >>> c.return_( Or([Local('x'), Local('y')]) )

    >>> dump(c.code())

                    LOAD_FAST 0 (x)

                    JUMP_IF_TRUE L1

                    POP_TOP

                    LOAD_FAST 1 (y)

            L1: RETURN_VALUE

 

 

True or false constants are folded automatically, avoiding code generation

for intermediate values that will never be used in the result::

 

    >>> c = Code()

    >>> c.return_( And([1, 2, Local('y')]) )

    >>> dis(c.code())

      0 0 LOAD_FAST 0 (y)

                  3 RETURN_VALUE

 

    >>> c = Code()

    >>> c.return_( And([1, 2, Local('y'), 0]) )

    >>> dump(c.code())

                    LOAD_FAST 0 (y)

                    JUMP_IF_FALSE L1

                    POP_TOP

                    LOAD_CONST 1 (0)

            L1: RETURN_VALUE

 

    >>> c = Code()

    >>> c.return_( Or([1, 2, Local('y')]) )

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (1)

                  3 RETURN_VALUE

 

    >>> c = Code()

    >>> c.return_( Or([False, Local('y'), 3]) )

    >>> dump(c.code())

                    LOAD_FAST 0 (y)

                    JUMP_IF_TRUE L1

                    POP_TOP

                    LOAD_CONST 1 (3)

            L1: RETURN_VALUE

 

 

want to implement more sophisticated callables.

 

To make it easy to create diverse target types, a ``nodetype()`` decorator is

provided::

    >>> from peak.util.assembler import nodetype

 

It allows you to create code generation target types using functions. Your

function should take one or more arguments, with a ``code=None`` optional

argument in the last position. It should check whether ``code is None`` when

called, and if so, return a tuple of the preceding arguments. If ``code``

is not ``None``, then it should do whatever code generating tasks are required.

For example::

    ... return block1, block2

    >>> TryFinally = nodetype()(TryFinally)

 

Note: although the nodetype() generator can be used above the function

definition in either Python 2.3 or 2.4, it cannot be done in a doctest under

Python 2.3, so this document doesn't attempt to demonstrate that. Under

2.4, you would do something like this::

 

    @nodetype()

    def TryFinally(...):

 

and code that needs to also work under 2.3 should do something like this::

 

    nodetype()

    def TryFinally(...):

 

But to keep the examples here working with doctest, we'll be doing our

``nodetype()`` calls after the end of the function definitions, e.g.::

    ... return value,

    >>> ExprStmt = nodetype()(ExprStmt)

    >>> dump(c.code())

                    SETUP_FINALLY L1

                    LOAD_CONST 1 (1)

                    POP_TOP

                    POP_BLOCK

                    LOAD_CONST 0 (None)

            L1: LOAD_CONST 2 (2)

                    POP_TOP

                    END_FINALLY

 

The ``nodetype()`` decorator is virtually identical to the ``struct()``

decorator in the DecoratorTools package, except that it does not support

``*args``, does not create a field for the ``code`` argument, and generates a

``__call__()`` method that reinvokes the wrapped function to do the actual

code generation.

 

Among the benefits of this decorator are:

 

* It gives your node types a great debugging format::

 

    >>> tf = TryFinally(ExprStmt(1), ExprStmt(2))

    >>> tf

    TryFinally(ExprStmt(1), ExprStmt(2))

 

* It makes named fields accessible::

 

    >>> tf.block1

    ExprStmt(1)

 

    >>> tf.block2

    ExprStmt(2)

 

* Hashing and comparison work as expected (handy for algorithms that require

  comparing or caching AST subtrees, such as common subexpression

  elimination)::

 

    >>> ExprStmt(1) == ExprStmt(1)

    True

    >>> ExprStmt(1) == ExprStmt(2)

    False

 

 

Please see the `struct decorator documentation`_ for info on how to customize

node types further.

 

.. _struct decorator documentation: http://peak.telecommunity.com/DevCenter/DecoratorTools#the-struct-decorator

 

Note: hashing only works if all the values you return in your argument tuple

are hashable, so you should try to convert them if possible. For example, if

an argument accepts any sequence, you should probably convert it to a tuple

before returning it. Most of the examples in this document, and the node types

supplied by ``peak.util.assembler`` itself do this.

 

 

Constant Folding in Custom Targets

----------------------------------

or code generation time. For example, this ``And`` node type (a simpler

version of the one included in ``peak.util.assembler``) folds constants during

code generation, by not generating unnecessary branches when it can

    ... return tuple(values),

    ... continue # true constants can be skipped

    ... code(value, end.JUMP_IF_FALSE_OR_POP)

    ... else: # and false constants end the chain right away

    ... return code(value, end)

    >>> And = nodetype()(And)

    >>> dump(c.code())

                    LOAD_FAST 0 (x)

                    JUMP_IF_FALSE L1

                    POP_TOP

                    LOAD_CONST 1 (False)

            L1: RETURN_VALUE

 

The above example only folds constants at code generation time, however. You

can also do constant folding at AST construction time, using the

``fold_args()`` function. For example::

 

    >>> from peak.util.assembler import fold_args

 

    >>> def Getattr(ob, name, code=None):

    ... try:

    ... name = const_value(name)

    ... except NotAConstant:

    ... return Call(Const(getattr), [ob, name])

    ... if code is None:

    ... return fold_args(Getattr, ob, name)

    ... code(ob)

    ... code.LOAD_ATTR(name)

    >>> Getattr = nodetype()(Getattr)

 

    >>> const_value(Getattr(1, '__class__'))

    <type 'int'>

 

The ``fold_args()`` function tries to evaluate the node immediately, if all of

its arguments are constants, by creating a temporary ``Code`` object, and

running the supplied function against it, then doing an ``eval()`` on the

generated code and wrapping the result in a ``Const``. However, if any of the

arguments are non-constant, the original arguments (less the function) are

returned. This causes a normal node instance to be created instead of a

``Const``.

 

This isn't a very *fast* way of doing partial evaluation, but it makes it

really easy to define new code generation targets without writing custom

constant-folding code for each one. Just ``return fold_args(ThisType, *args)``

instead of ``return args``, if you want your node constructor to be able to do

eager evaluation. If you need to, you can check your parameters in order to

decide whether to call ``fold_args()`` or not; this is in fact how ``Call``

implements its ``fold`` argument and the suppression of folding when

the call has no arguments.

 

(By the way, this same ``Getattr`` node type is also available

    >>> tuple(inspect.getargspec(f1))

    >>> tuple(inspect.getargspec(f2))

    >>> c1.co_filename is f1.func_code.co_filename

    True

    >>> tuple(inspect.getargspec(f3))

    >>> tuple(inspect.getargspec(f4))

You can also use the ``from_spec(name='<lambda>', args=(), var=None, kw=None)``

classmethod to explicitly set a name and argument spec for a new code object::

 

    >>> c = Code.from_spec('a', ('b', ('c','d'), 'e'), 'f', 'g')

    >>> c.co_name

    'a'

 

    >>> c.co_varnames

    ['b', '.1', 'e', 'f', 'g', 'c', 'd']

 

    >>> c.co_argcount

    3

    

    >>> tuple(inspect.getargs(c.code()))

    (['b', ['c', 'd'], 'e'], 'f', 'g')

 

    >>> tuple(inspect.getargspec(f))

    Its value is automatically updated by most opcodes, but if you are doing

    something sufficiently tricky (as in the ``Switch`` demo, below) you may

    need to explicitly set it.

 

    The ``stack_size`` automatically becomes ``None`` after any unconditional

    jump operations, such as ``JUMP_FORWARD``, ``BREAK_LOOP``, or

    ``RETURN_VALUE``. When the stack size is ``None``, the only operations

    that can be performed are the resolving of forward references (which will

    set the stack size to what it was when the reference was created), or

    manually setting the stack size.

    A tuple of strings naming a function's "free" variables. Defaults to an

    value is updated automatically as you generate code or change

    the ``stack_size`` attribute.

 

 

 

Stack Size Tracking and Dead Code Detection

===========================================

 

``Code`` objects automatically track the predicted stack size as code is

generated, by updating the ``stack_size`` attribute as each operation occurs.

A history is kept so that backward jumps can be checked to ensure that the

current stack height is the same as at the jump's target. Similarly, when

forward jumps are resolved, the stack size at the jump target is checked

against the stack size at the jump's origin. If there are multiple jumps to

the same location, they must all have the same stack size at the origin and

the destination.

 

In addition, whenever any unconditional jump code is generated (i.e.

``JUMP_FORWARD``, ``BREAK_LOOP``, ``CONTINUE_LOOP``, ``JUMP_ABSOLUTE``, or

``RETURN_VALUE``), the predicted ``stack_size`` is set to ``None``. This

means that the ``Code`` object does not know what the stack size will be at

the current location. You cannot issue *any* instructions when the predicted

stack size is ``None``, as you will receive an ``AssertionError``::

 

    >>> c = Code()

    >>> fwd = c.JUMP_FORWARD()

    >>> print c.stack_size # forward jump marks stack size as unknown

    None

 

    >>> c.LOAD_CONST(42)

    Traceback (most recent call last):

      ...

    AssertionError: Unknown stack size at this location

 

Instead, you must resolve a forward reference (or define a previously-jumped to

label). This will propagate the stack size at the source of the jump to the

current location, updating the stack size::

 

    >>> fwd()

    >>> c.stack_size

    0

 

Note, by the way, that this means it is impossible for you to generate static

"dead code". In other words, you cannot generate code that isn't reachable.

You should therefore check if ``stack_size`` is ``None`` before generating

code that might be unreachable. For example, consider this ``If``

implementation::

 

    >>> def If(cond, then, else_=Pass, code=None):

    ... if code is None:

    ... return cond, then, else_

    ... else_clause = Label()

    ... end_if = Label()

    ... code(cond, else_clause.JUMP_IF_FALSE_OR_POP, then)

    ... code(end_if.JUMP_FORWARD, else_clause, Code.POP_TOP, else_)

    ... code(end_if)

    >>> If = nodetype()(If)

 

It works okay if there's no dead code::

 

    >>> c = Code()

    >>> c( If(Local('a'), 42, 55) )

    >>> dump(c.code())

                    LOAD_FAST 0 (a)

                    JUMP_IF_FALSE L1

                    POP_TOP

                    LOAD_CONST 1 (42)

                    JUMP_FORWARD L2

            L1: POP_TOP

                    LOAD_CONST 2 (55)

 

But it breaks if you end the "then" block with a return::

 

    >>> c = Code()

    >>> c( If(23, Return(42), 55) )

    Traceback (most recent call last):

      ...

    AssertionError: Unknown stack size at this location

 

What we need is something like this instead::

 

    >>> def If(cond, then, else_=Pass, code=None):

    ... if code is None:

    ... return cond, then, else_

    ... else_clause = Label()

    ... end_if = Label()

    ... code(cond, else_clause.JUMP_IF_FALSE_OR_POP, then)

    ... if code.stack_size is not None:

    ... end_if.JUMP_FORWARD(code)

    ... code(else_clause, Code.POP_TOP, else_, end_if)

    >>> If = nodetype()(If)

 

As you can see, the dead code is now eliminated::

 

    >>> c = Code()

    >>> c( If(Local('a'), Return(42), 55) )

    >>> dump(c.code())

                    LOAD_FAST 0 (a)

                    JUMP_IF_FALSE L1

                    POP_TOP

                    LOAD_CONST 1 (42)

                    RETURN_VALUE

            L1: POP_TOP

                    LOAD_CONST 2 (55)

When you issue a ``SETUP_EXCEPT`` or ``SETUP_FINALLY``, the code's maximum

stack size is raised to ensure that it's at least 3 items higher than

In the case of ``SETUP_EXCEPT``, the *current* stack size is increased by 3

after a ``POP_BLOCK``, because the code that follows will be an exception

handler and will thus always have exception items on the stack::

    >>> dump(c.code())

                    SETUP_EXCEPT L1

                    POP_BLOCK

                    JUMP_FORWARD L2

            L1: POP_TOP

                    POP_TOP

                    POP_TOP

            L2: LOAD_CONST 0 (None)

                    RETURN_VALUE

    >>> dump(c.code())

                    SETUP_EXCEPT L1

                    POP_BLOCK

                    JUMP_FORWARD L2

            L1: POP_TOP

                    POP_TOP

                    POP_TOP

            L2: LOAD_CONST 0 (None)

                    RETURN_VALUE

 

(Labels have a ``POP_BLOCK`` attribute that you can pass in when generating

code.)

 

And, for generating typical try/except blocks, you can use the ``TryExcept``

node type, which takes a body, a sequence of exception-type/handler pairs,

and an optional "else" clause::

 

    >>> from peak.util.assembler import TryExcept

    >>> c = Code()

    >>> c.return_(

    ... TryExcept(

    ... Return(1), # body

    ... [(Const(KeyError),2), (Const(TypeError),3)], # handlers

    ... Return(4) # else clause

    ... )

    ... )

    >>> dump(c.code())

                    SETUP_EXCEPT L1

                    LOAD_CONST 1 (1)

                    RETURN_VALUE

                    POP_BLOCK

                    JUMP_FORWARD L4

            L1: DUP_TOP

                    LOAD_CONST 2 (<...exceptions.KeyError...>)

                    COMPARE_OP 10 (exception match)

                    JUMP_IF_FALSE L2

                    POP_TOP

                    POP_TOP

                    POP_TOP

                    POP_TOP

                    LOAD_CONST 3 (2)

                    JUMP_FORWARD L5

            L2: POP_TOP

                    DUP_TOP

                    LOAD_CONST 4 (<...exceptions.TypeError...>)

                    COMPARE_OP 10 (exception match)

                    JUMP_IF_FALSE L3

                    POP_TOP

                    POP_TOP

                    POP_TOP

                    POP_TOP

                    LOAD_CONST 5 (3)

                    JUMP_FORWARD L5

            L3: POP_TOP

                    END_FINALLY

            L4: LOAD_CONST 6 (4)

                    RETURN_VALUE

            L5: RETURN_VALUE

    >>> dump(c.code())

                    SETUP_FINALLY L1

                    POP_BLOCK

                    LOAD_CONST 0 (None)

            L1: END_FINALLY

For your convenience, the ``TryFinally`` node type can also be used to generate

try/finally blocks::

 

    >>> from peak.util.assembler import TryFinally

    >>> c = Code()

    >>> c( TryFinally(ExprStmt(1), ExprStmt(2)) )

    >>> dump(c.code())

                    SETUP_FINALLY L1

                    LOAD_CONST 1 (1)

                    POP_TOP

                    POP_BLOCK

                    LOAD_CONST 0 (None)

            L1: LOAD_CONST 2 (2)

                    POP_TOP

                    END_FINALLY

 

    >>> dump(c.code())

                    SETUP_LOOP L3

                    LOAD_CONST 1 (5)

            L1: JUMP_IF_FALSE L2

                    LOAD_CONST 2 (1)

                    BINARY_SUBTRACT

                    JUMP_ABSOLUTE L1

            L2: POP_TOP

                    POP_BLOCK

                    LOAD_CONST 3 (42)

                    RETURN_VALUE

            L3: LOAD_CONST 0 (None)

                    RETURN_VALUE

    >>> c.LOAD_CONST(57)

    >>> fwd = c.JUMP_IF_TRUE()

    >>> fwd()

    >>> dump(c.code())

                    LOAD_CONST 1 (57)

                    SETUP_LOOP L3

                    JUMP_IF_TRUE L2

            L1: JUMP_ABSOLUTE L1

            L2: BREAK_LOOP

                    POP_BLOCK

    >>> c.LOAD_CONST(57)

    >>> fwd = c.JUMP_IF_TRUE()

    >>> fwd()

    >>> dump(c.code())

                    LOAD_CONST 1 (57)

                    SETUP_LOOP L4

            L1: SETUP_FINALLY L3

                    JUMP_IF_TRUE L2

                    CONTINUE_LOOP L1

            L2: POP_BLOCK

                    LOAD_CONST 0 (None)

            L3: END_FINALLY

                    POP_BLOCK

 

``for`` Loops

-------------

 

There is a ``For()`` node type available for generating simple loops (without

break/continue support). It takes an iterable expression, an assignment

clause, and a loop body::

 

    >>> from peak.util.assembler import For

    >>> y = Call(Const(range), (3,))

    >>> x = LocalAssign('x')

    >>> body = Suite([Local('x'), Code.PRINT_EXPR])

 

    >>> c = Code()

    >>> c(For(y, x, body)) # for x in range(3): print x

    >>> c.return_()

    >>> dump(c.code())

                    LOAD_CONST 1 ([0, 1, 2])

                    GET_ITER

            L1: FOR_ITER L2

                    STORE_FAST 0 (x)

                    LOAD_FAST 0 (x)

                    PRINT_EXPR

                    JUMP_ABSOLUTE L1

            L2: LOAD_CONST 0 (None)

                    RETURN_VALUE

 

The arguments are given in execution order: first the "in" value of the loop,

then the assignment to a loop variable, and finally the body of the loop. The

distinction between the assignment and body, however, is only for clarity and

convenience (to avoid needing to glue the assignment to the body with a

``Suite``). If you already have a suite or only need one node for the entire

loop body, you can do the same thing with only two arguments::

 

    >>> c = Code()

    >>> c(For(y, Code.PRINT_EXPR))

    >>> c.return_()

    >>> dump(c.code())

                    LOAD_CONST 1 ([0, 1, 2])

                    GET_ITER

            L1: FOR_ITER L2

                    PRINT_EXPR

                    JUMP_ABSOLUTE L1

            L2: LOAD_CONST 0 (None)

                    RETURN_VALUE

 

Notice, by the way, that ``For()`` does NOT set up a loop block for you, so if

you want to be able to use break and continue, you'll need to wrap the loop in

a labelled SETUP_LOOP/POP_BLOCK pair, as described in the preceding sections.

 

 

List Comprehensions

-------------------

 

In order to generate correct list comprehension code for the target Python

version, you must use the ``ListComp()`` and ``LCAppend()`` node types. This

is because Python versions 2.4 and up store the list being built in a temporary

variable, and use a special ``LIST_APPEND`` opcode to append values, while 2.3

stores the list's ``append()`` method in the temporary variable, and calls it

to append values.

 

The ``ListComp()`` node wraps a code body (usually a ``For()`` loop) and

manages the creation and destruction of a temporary variable (e.g. ``_[1]``,

``_[2]``, etc.). The ``LCAppend()`` node type wraps a value or expression to

be appended to the innermost active ``ListComp()`` in progress::

 

    >>> from peak.util.assembler import ListComp, LCAppend

    >>> c = Code()

    >>> simple = ListComp(For(y, x, LCAppend(Local('x'))))

    >>> c.return_(simple)

    >>> eval(c.code())

    [0, 1, 2]

 

    >>> c = Code()

    >>> c.return_(ListComp(For(y, x, LCAppend(simple))))

    >>> eval(c.code())

    [[0, 1, 2], [0, 1, 2], [0, 1, 2]]

 

 

Closures and Nested Functions

=============================

 

Free and Cell Variables

-----------------------

 

To implement closures and nested scopes, your code objects must use "free" or

"cell" variables in place of regular "fast locals". A "free" variable is one

that is defined in an outer scope, and a "cell" variable is one that's defined

in the current scope, but will also be used by nested functions.

 

The simplest way to set up free or cell variables is to use a code object's

``makefree(names)`` and ``makecells(names)`` methods::

 

    >>> c = Code()

    >>> c.co_cellvars

    ()

    >>> c.co_freevars

    ()

 

    >>> c.makefree(['x', 'y'])

    >>> c.makecells(['z'])

 

    >>> c.co_cellvars

    ('z',)

    >>> c.co_freevars

    ('x', 'y')

 

When a name has been defined as a free or cell variable, the ``_DEREF`` opcode

variants are used to generate ``Local()`` and ``LocalAssign()`` nodes::

 

    >>> c((Local('x'), Local('y')), LocalAssign('z'))

      0 0 LOAD_DEREF 1 (x)

                  3 LOAD_DEREF 2 (y)

                  6 BUILD_TUPLE 2

                  9 STORE_DEREF 0 (z)

 

If you have already written code in a code object that operates on the relevant

locals, the code is retroactively patched to use the ``_DEREF`` opcodes::

    >>> c = Code()

    >>> c((Local('x'), Local('y')), LocalAssign('z'))

    >>> dis(c.code())

      0 0 LOAD_FAST 0 (x)

                  3 LOAD_FAST 1 (y)

                  6 BUILD_TUPLE 2

                  9 STORE_FAST 2 (z)

    >>> c.makefree(['x', 'y'])

    >>> c.makecells(['z'])

 

    >>> dis(c.code())

      0 0 LOAD_DEREF 1 (x)

                  3 LOAD_DEREF 2 (y)

                  6 BUILD_TUPLE 2

                  9 STORE_DEREF 0 (z)

 

This means that you can defer the decision of which locals are free/cell

variables until the code is ready to be generated. In fact, by passing in

a "parent" code object to the ``.code()`` method, you can get BytecodeAssembler

to automatically call ``makefree()`` and ``makecells()`` for the correct

variable names in the child and parent code objects, as we'll see in the next

section.

 

 

Nested Code Objects

-------------------

 

To create a code object for use in a nested scope, you can use the parent code

object's ``nested()`` method. It works just like the ``from_spec()``

classmethod, except that the ``co_filename`` of the parent is copied to the

child::

 

    >>> p = Code()

    >>> p.co_filename = 'testname'

 

    >>> c = p.nested('sub', ['a','b'], 'c', 'd')

 

    >>> c.co_name

    'sub'

    

    >>> c.co_filename

    'testname'

 

    >>> tuple(inspect.getargs(c.code(p)))

    (['a', 'b'], 'c', 'd')

 

Notice that you must pass the parent code object to the child's ``.code()``

method to ensure that free/cell variables are properly set up. When the

``code()`` method is given another code object as a parameter, it automatically

converts any locally-read (but not written) to "free" variables in the child

code, and ensures that those same variables become "cell" variables in the

supplied parent code object::

 

    >>> p.LOAD_CONST(42)

    >>> p(LocalAssign('a'))

    >>> dis(p.code())

      0 0 LOAD_CONST 1 (42)

                  3 STORE_FAST 0 (a)

    

    >>> c = p.nested()

    >>> c(Local('a'))

 

    >>> dis(c.code(p))

      0 0 LOAD_DEREF 0 (a)

 

    >>> dis(p.code())

      0 0 LOAD_CONST 1 (42)

                  3 STORE_DEREF 0 (a)

 

Notice that the ``STORE_FAST`` in the parent code object was automatically

patched to a ``STORE_DEREF``, with an updated offset if applicable. Any

future use of ``Local('a')`` or ``LocalAssign('a')`` in the parent or child

code objects will now refer to the free/cell variable, rather than the "local"

variable::

 

    >>> p(Local('a'))

    >>> dis(p.code())

      0 0 LOAD_CONST 1 (42)

                  3 STORE_DEREF 0 (a)

                  6 LOAD_DEREF 0 (a)

 

    >>> c(LocalAssign('a'))

    >>> dis(c.code(p))

      0 0 LOAD_DEREF 0 (a)

                  3 STORE_DEREF 0 (a)

 

 

``Function()``

--------------

 

The ``Function(body, name='<lambda>', args=(), var=None, kw=None, defaults=())``

node type creates a function object from the specified body and the optional

name, argument specs, and defaults. The ``Function()`` node generates code to

create the function object with the appropriate defaults and closure (if

applicable), and any needed free/cell variables are automatically set up in the

parent and child code objects. The newly generated function will be on top of

the stack at the end of the generated code::

 

    >>> from peak.util.assembler import Function

    >>> c = Code()

    >>> c.co_filename = '<string>'

    >>> c.return_(Function(Return(Local('a')), 'f', ['a'], defaults=[42]))

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (42)

                  3 LOAD_CONST 2 (<... f ..., file "<string>", line -1>)

                  6 MAKE_FUNCTION 1

                  9 RETURN_VALUE

 

Now that we've generated the code for a function returning a function, let's

run it, to get the function we defined::

 

    >>> f = eval(c.code())

    >>> f

    <function f at ...>

 

    >>> tuple(inspect.getargspec(f))

    (['a'], None, None, (42,))

 

    >>> f()

    42

 

    >>> f(99)

    99

 

Now let's create a doubly nested function, with some extras::

 

    >>> c = Code()

    >>> c.co_filename = '<string>'

    >>> c.return_(

    ... Function(Return(Function(Return(Local('a')))),

    ... 'f', ['a', 'b'], 'c', 'd', [99, 66])

    ... )

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (99)

                  3 LOAD_CONST 2 (66)

                  6 LOAD_CONST 3 (<... f ..., file "<string>", line -1>)

                  9 MAKE_FUNCTION 2

                 12 RETURN_VALUE

 

    >>> f = eval(c.code())

    >>> f

    <function f at ...>

 

    >>> tuple(inspect.getargspec(f))

    (['a', 'b'], 'c', 'd', (99, 66))

 

    >>> dis(f)

      0 0 LOAD_CLOSURE 0 (a)

                  ... LOAD_CONST 1 (<... <lambda> ..., file "<string>", line -1>)

                  ... MAKE_CLOSURE 0

                  ... RETURN_VALUE

 

    >>> dis(f())

      0 0 LOAD_DEREF 0 (a)

                  3 RETURN_VALUE

 

    >>> f(42)()

    42

 

    >>> f()()

    99

 

As you can see, ``Function()`` not only takes care of setting up free/cell

variables in all the relevant scopes, it also chooses whether to use

``MAKE_FUNCTION`` or ``MAKE_CLOSURE``, and generates code for the defaults.

 

(Note, by the way, that the `defaults` argument should be a sequence of

generatable expressions; in the examples here, we used numbers, but they could

have been arbitrary expression nodes.)

    >>> dis(simple_code(13,414))

    >>> dis(simple_code(13,572,120))

    >>> c = Code() # 0

    >>> c.LOAD_CONST(1) # 1

    >>> c.POP_TOP() # 0

    >>> c.LOAD_CONST(2) # 1

    >>> c.LOAD_CONST(3) # 2

    >>> c.stack_history

    [0, ..., 1, 0, ..., 1]

    >>> c.BINARY_ADD() # 1

    >>> c.LOAD_CONST(4) # 2

    >>> c.stack_history

    [0, ..., 1, 0, 1, ..., 2, ..., 1]

Code iteration::

 

    >>> c.DUP_TOP()

    >>> c.return_(Code.POP_TOP)

    >>> list(c) == [

    ... (0, op.LOAD_GLOBAL, 0),

    ... (3, op.LOAD_ATTR, 1),

    ... (6, op.DELETE_FAST, 0),

    ... (9, op.DUP_TOP, None),

    ... (10, op.POP_TOP, None),

    ... (11, op.RETURN_VALUE, None)

    ... ]

    True

 

Code patching::

 

    >>> c = Code()

    >>> c.LOAD_CONST(42)

    >>> c.STORE_FAST('x')

    >>> c.LOAD_FAST('x')

    >>> c.DELETE_FAST('x')

    >>> c.RETURN_VALUE()

 

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (42)

                  3 STORE_FAST 0 (x)

                  6 LOAD_FAST 0 (x)

                  9 DELETE_FAST 0 (x)

                 12 RETURN_VALUE

 

 

    >>> c.co_varnames

    ['x']

    >>> c.co_varnames.append('y')

 

    >>> c._patch(

    ... {op.LOAD_FAST: op.LOAD_FAST,

    ... op.STORE_FAST: op.STORE_FAST,

    ... op.DELETE_FAST: op.DELETE_FAST},

    ... {0: 1}

    ... )

 

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (42)

                  3 STORE_FAST 1 (y)

                  6 LOAD_FAST 1 (y)

                  9 DELETE_FAST 1 (y)

                 12 RETURN_VALUE

 

    >>> c._patch({op.RETURN_VALUE: op.POP_TOP})

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (42)

                  3 STORE_FAST 1 (y)

                  6 LOAD_FAST 1 (y)

                  9 DELETE_FAST 1 (y)

                 12 POP_TOP

 

Converting locals to free/cell vars::

 

    >>> c = Code()

    >>> c.LOAD_CONST(42)

    >>> c.STORE_FAST('x')

    >>> c.LOAD_FAST('x')

 

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (42)

                  3 STORE_FAST 0 (x)

                  6 LOAD_FAST 0 (x)

 

    >>> c.co_freevars = 'y', 'x'

    >>> c.co_cellvars = 'z',

 

    >>> c._locals_to_cells()

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (42)

                  3 STORE_DEREF 2 (x)

                  6 LOAD_DEREF 2 (x)

 

    >>> c.DELETE_FAST('x')

    >>> c._locals_to_cells()

    Traceback (most recent call last):

      ...

    AssertionError: Can't delete local 'x' used in nested scope

 

    >>> c = Code()

    >>> c.LOAD_CONST(42)

    >>> c.STORE_FAST('x')

    >>> c.LOAD_FAST('x')

 

    >>> c.co_freevars

    ()

    >>> c.makefree(['x'])

    >>> c.co_freevars

    ('x',)

 

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (42)

                  3 STORE_DEREF 0 (x)

                  6 LOAD_DEREF 0 (x)

 

    >>> c = Code()

    >>> c.LOAD_CONST(42)

    >>> c.STORE_FAST('x')

    >>> c.LOAD_FAST('x')

    >>> c.makecells(['x'])

    >>> c.co_freevars

    ()

    >>> c.co_cellvars

    ('x',)

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (42)

                  3 STORE_DEREF 0 (x)

                  6 LOAD_DEREF 0 (x)

    

    >>> c = Code()

    >>> c.LOAD_CONST(42)

    >>> c.STORE_FAST('x')

    >>> c.LOAD_FAST('x')

    >>> c.makefree('x')

    >>> c.makecells(['y'])

    >>> c.co_freevars

    ('x',)

    >>> c.co_cellvars

    ('y',)

    >>> dis(c.code())

      0 0 LOAD_CONST 1 (42)

                  3 STORE_DEREF 1 (x)

                  6 LOAD_DEREF 1 (x)

 

    >>> c = Code()

    >>> c.co_flags &= ~op.CO_OPTIMIZED

    >>> c.makecells(['q'])

    Traceback (most recent call last):

      ...

    AssertionError: Can't use cellvars in unoptimized scope

    

 

 

Auto-free promotion with code parent:

 

    >>> p = Code()

    >>> c = Code()

    >>> c.LOAD_FAST('x')

    >>> dis(c.code(p))

      0 0 LOAD_DEREF 0 (x)

    >>> p.co_cellvars

    ('x',)

 

    >>> p = Code()

    >>> c = Code.from_function(lambda x,y,z=2: None)

    >>> c.LOAD_FAST('x')

    >>> c.LOAD_FAST('y')

    >>> c.LOAD_FAST('z')

    

    >>> dis(c.code(p))

      0 0 LOAD_FAST 0 (x)

                  3 LOAD_FAST 1 (y)

                  6 LOAD_FAST 2 (z)

    >>> p.co_cellvars

    ()

 

    >>> c.LOAD_FAST('q')

    >>> dis(c.code(p))

      0 0 LOAD_FAST 0 (x)

                  3 LOAD_FAST 1 (y)

                  6 LOAD_FAST 2 (z)

                  9 LOAD_DEREF 0 (q)

    >>> p.co_cellvars

    ('q',)

 

    >>> p = Code()

    >>> c = Code.from_function(lambda x,*y,**z: None)

    >>> c.LOAD_FAST('q')

    >>> c.LOAD_FAST('x')

    >>> c.LOAD_FAST('y')

    >>> c.LOAD_FAST('z')

    >>> dis(c.code(p))

      0 0 LOAD_DEREF 0 (q)

                  3 LOAD_FAST 0 (x)

                  6 LOAD_FAST 1 (y)

                  9 LOAD_FAST 2 (z)

    >>> p.co_cellvars

    ('q',)

 

    >>> p = Code()

    >>> c = Code.from_function(lambda x,*y: None)

    >>> c.LOAD_FAST('x')

    >>> c.LOAD_FAST('y')

    >>> c.LOAD_FAST('z')

    >>> dis(c.code(p))

      0 0 LOAD_FAST 0 (x)

                  3 LOAD_FAST 1 (y)

                  6 LOAD_DEREF 0 (z)

    >>> p.co_cellvars

    ('z',)

 

    >>> p = Code()

    >>> c = Code.from_function(lambda x,**y: None)

    >>> c.LOAD_FAST('x')

    >>> c.LOAD_FAST('y')

    >>> c.LOAD_FAST('z')

    >>> dis(c.code(p))

      0 0 LOAD_FAST 0 (x)

                  3 LOAD_FAST 1 (y)

                  6 LOAD_DEREF 0 (z)

    >>> p.co_cellvars

    ('z',)

 

 

Stack tracking on jumps::

 

    >>> c = Code()

    >>> else_ = Label()

    >>> end = Label()

    >>> c(99, else_.JUMP_IF_TRUE_OR_POP, end.JUMP_FORWARD)

    >>> c(else_, Code.POP_TOP, end)

    >>> dump(c.code())

                    LOAD_CONST 1 (99)

                    JUMP_IF_TRUE L1

                    POP_TOP

                    JUMP_FORWARD L2

            L1: POP_TOP

 

    >>> c.stack_size

    0

    >>> if sys.version>='2.7':

    ... print c.stack_history == [0, 1, 1, 1, 0, 0, 0, None, None, 1]

    ... else:

    ... print c.stack_history == [0, 1, 1, 1, 1, 1, 1, 0, None, None, 1]

    True

    

 

    >>> c = Code()

    >>> fwd = c.JUMP_FORWARD()

    >>> c.LOAD_CONST(42) # forward jump marks stack size unknown

    Traceback (most recent call last):

      ...

    AssertionError: Unknown stack size at this location

 

    >>> c.stack_size = 0

    >>> c.LOAD_CONST(42)

    >>> fwd()

    Traceback (most recent call last):

      ...

    AssertionError: Stack level mismatch: actual=1 expected=0

 

    >>> from peak.util.assembler import For

    >>> c = Code()

    >>> c(For((), Code.POP_TOP, Pass))

    >>> c.return_()

    >>> dump(c.code())

                    BUILD_TUPLE 0

                    GET_ITER

            L1: FOR_ITER L2

                    POP_TOP

                    JUMP_ABSOLUTE L1

            L2: LOAD_CONST 0 (None)

                    RETURN_VALUE

 

    >>> c.stack_history

    [0, 1, 1, 1, 1, 2, 2, 2, 1, None, None, 0, 1, 1, 1]

 

 

Yield value::

 

    >>> import sys

    >>> from peak.util.assembler import CO_GENERATOR

    >>> c = Code()

    >>> c.co_flags & CO_GENERATOR

    0

    >>> c(42, Code.YIELD_VALUE)

    >>> c.stack_size == int(sys.version>='2.5')

    True

    >>> (c.co_flags & CO_GENERATOR) == CO_GENERATOR

    True

 

    

    

    >>> c = Code()

    >>> c.LOAD_CONST(1) # closure

    >>> if sys.version>='2.5': c.BUILD_TUPLE(1)

    >>> c.LOAD_CONST(2) # default

    >>> c.LOAD_CONST(3) # simulate being a function

    >>> c.MAKE_CLOSURE(1, 1)

    >>> if sys.version>='2.5': c.BUILD_TUPLE(2)

    >>> c.MAKE_CLOSURE(0, 2)

 

    >>> c.JUMP_FORWARD(where)

Constant folding for ``*args`` and ``**kw``::

Try/Except stack level tracking::

 

    >>> def class_or_type_of(expr):

    ... return Suite([expr, TryExcept(

    ... Suite([Getattr(Code.DUP_TOP, '__class__'), Code.ROT_TWO]),

    ... [(Const(AttributeError), Call(Const(type), (Code.ROT_TWO,)))]

    ... )])

 

    >>> def type_or_class(x): pass

    >>> c = Code.from_function(type_or_class)

    >>> c.return_(class_or_type_of(Local('x')))

    >>> dump(c.code())

                    LOAD_FAST 0 (x)

                    SETUP_EXCEPT L1

                    DUP_TOP

                    LOAD_ATTR 0 (__class__)

                    ROT_TWO

                    POP_BLOCK

                    JUMP_FORWARD L3

            L1: DUP_TOP

                    LOAD_CONST 1 (<...exceptions.AttributeError...>)

                    COMPARE_OP 10 (exception match)

                    JUMP_IF_FALSE L2

                    POP_TOP

                    POP_TOP

                    POP_TOP

                    POP_TOP

                    LOAD_CONST 2 (<type 'type'>)

                    ROT_TWO

                    CALL_FUNCTION 1

                    JUMP_FORWARD L3

            L2: POP_TOP

                    END_FINALLY

            L3: RETURN_VALUE

 

    >>> type_or_class.func_code = c.code()

    >>> type_or_class(23)

    <type 'int'>

    

 

 

    >>> WHY_CONTINUE = {'2.3':5}.get(sys.version[:3], 32)

    ... return expr, tuple(cases), default

    ... Call(Const(d.get), [expr]),

    ... cursize = code.stack_size - 1 # adjust for removed WHY_CONTINUE

    ... code.stack_size = cursize

    ... code(value)

    ... if code.stack_size is not None: # if the code can fall through,

    ... code(cleanup.JUMP_FORWARD) # jump forward to the cleanup

    >>> Switch = nodetype()(Switch)

    >>> dump(c.code())

                    SETUP_LOOP L2

                    LOAD_CONST 1 (<...method get of dict...>)

                    LOAD_FAST 0 (x)

                    CALL_FUNCTION 1

                    JUMP_IF_FALSE L1

                    LOAD_CONST 2 (...)

                    END_FINALLY

                    LOAD_CONST 3 (42)

                    RETURN_VALUE

                    LOAD_CONST 4 ('foo')

                    RETURN_VALUE

            L1: POP_TOP

                    LOAD_CONST 5 (27)

                    RETURN_VALUE

                    POP_BLOCK

            L2: LOAD_CONST 0 (None)

                    RETURN_VALUE

 

* Exhaustive tests of all opcodes' stack history effects

 

* Test wide jumps and wide argument generation in general