The PEAK Developers' Center   Diff for "BytecodeAssembler" UserPreferences
 
HelpContents Search Diffs Info Edit Subscribe XML Print View
Ignore changes in the amount of whitespace

Differences between version dated 2006-07-05 01:49:58 and 2010-08-02 17:38:15 (spanning 11 versions)

Deletions are marked like this.
Additions are marked like this.

bytecode instead of on these mechanical issues.
 
In addition to a low-level opcode-oriented API for directly generating specific
bytecodes, this module also offers an extensible mini-AST framework for
Python bytecodes, this module also offers an extensible mini-AST framework for
generating code from high-level specifications. This framework does most of
the work needed to transform tree-like structures into linear bytecode
instructions, and includes the ability to do compile-time constant folding.
 
Please see the `BytecodeAssembler reference manual`_ for more details.
 
.. _BytecodeAssembler reference manual: http://peak.telecommunity.com/DevCenter/BytecodeAssembler#toc
 
 
Changes since version 0.5.2:
 
* Symbolic disassembly with full emulation of backward-compatible
  ``JUMP_IF_TRUE`` and ``JUMP_IF_FALSE`` opcodes on Python 2.7 -- tests now
  run clean on Python 2.7.
 
* Support for backward emulation of Python 2.7's ``JUMP_IF_TRUE_OR_POP`` and
  ``JUMP_IF_FALSE_OR_POP`` instructions on earlier Python versions; these
  emulations are also used in BytecodeAssembler's internal code generation,
  for maximum performance on 2.7+ (with no change to performance on older
  versions).
 
Changes since version 0.5.1:
 
* Initial support for Python 2.7's new opcodes and semantics changes, mostly
  by emulating older versions' behavior with macros. (0.5.2 is really just
  a quick-fix release to allow packages using BytecodeAssembler to run on 2.7
  without having to change any of their code generation; future releases will
  provide proper support for the new and changed opcodes, as well as a test
  suite that doesn't show spurious differences in the disassembly listings
  under Python 2.7.)
 
Changes since version 0.5:
 
* Fix incorrect stack size calculation for ``MAKE_CLOSURE`` on Python 2.5+
 
Changes since version 0.3:
 
* New node types:
 
  * ``For(iterable, assign, body)`` -- define a "for" loop over `iterable`
 
  * ``UnpackSequence(nodes)`` -- unpacks a sequence that's ``len(nodes)`` long,
    and then generates the given nodes.
 
  * ``LocalAssign(name)`` -- issues a ``STORE_FAST``, ``STORE_DEREF`` or
    ``STORE_LOCAL`` as appropriate for the given name.
 
  * ``Function(body, name='<lambda>', args=(), var=None, kw=None, defaults=())``
    -- creates a nested function from `body` and puts it on the stack.
 
  * ``If(cond, then_, else_=Pass)`` -- "if" statement analogue
 
  * ``ListComp(body)`` and ``LCAppend(value)`` -- implement list comprehensions
 
  * ``YieldStmt(value)`` -- generates a ``YIELD_VALUE`` (plus a ``POP_TOP`` in
    Python 2.5+)
 
* ``Code`` objects are now iterable, yielding ``(offset, op, arg)`` triples,
  where `op` is numeric and `arg` is either numeric or ``None``.
 
* ``Code`` objects' ``.code()`` method can now take a "parent" ``Code`` object,
  to link the child code's free variables to cell variables in the parent.
 
* Added ``Code.from_spec()`` classmethod, that initializes a code object from a
  name and argument spec.
 
* ``Code`` objects now have a ``.nested(name, args, var, kw)`` method, that
  creates a child code object with the same ``co_filename`` and the supplied
  name/arg spec.
 
* Fixed incorrect stack tracking for the ``FOR_ITER`` and ``YIELD_VALUE``
  opcodes
 
* Ensure that ``CO_GENERATOR`` flag is set if ``YIELD_VALUE`` opcode is used
 
* Change tests so that Python 2.3's broken line number handling in ``dis.dis``
  and constant-folding optimizer don't generate spurious failures in this
  package's test suite.
 
 
Changes since version 0.2:
 
* Added ``Suite``, ``TryExcept``, and ``TryFinally`` node types
 
* Added a ``Getattr`` node type that does static or dynamic attribute access
  and constant folding
 
* Fixed ``code.from_function()`` not copying the ``co_filename`` attribute when
  ``copy_lineno`` was specified.
 
* The ``repr()`` of AST nodes doesn't include a trailing comma for 1-argument
  node types any more.
 
* Added a ``Pass`` symbol that generates no code, a ``Compare()`` node type
  that does n-way comparisons, and ``And()`` and ``Or()`` node types for doing
  logical operations.
 
* The ``COMPARE_OP()`` method now accepts operator strings like ``"<="``,
  ``"not in"``, ``"exception match"``, and so on, as well as numeric opcodes.
  See the standard library's ``opcode`` module for a complete list of the
  strings accepted (in the ``cmp_op`` tuple). ``"<>"`` is also accepted as an
  alias for ``"!="``.
 
* Added code to verify that forward jump offsets don't exceed a 64KB span, and
  support absolute backward jumps to locations >64KB.
 
Changes since version 0.1:
 
* Constant handling has been fixed so that it doesn't confuse equal values of
  differing types (e.g. ``1.0`` and ``True``), or equal unhashable objects
  (e.g. two empty lists).
 
* Removed ``nil`, ``ast_curry()`` and ``folding_curry()``, replacing them with
* Removed ``nil``, ``ast_curry()`` and ``folding_curry()``, replacing them with
  the ``nodetype()`` decorator and ``fold_args()``; please see the docs for
  more details.
 

* Various bug fixes
 
There are a few features that aren't tested yet, and not all opcodes may be
fully supported. Notably, the following features are still NOT reliably
supported yet:
fully supported. Also note the following limitations:
 
* Wide jump addressing (for generated bytecode>64K in size)
* Jumps to as-yet-undefined labels cannot span a distance greater than 65,535
  bytes.
 
* The ``dis()`` module in Python 2.3 has a bug that makes it show incorrect
* The ``dis()`` function in Python 2.3 has a bug that makes it show incorrect
  line numbers when the difference between two adjacent line numbers is
  greater than 255. This causes two shallow failures in the current test
  suite when it's run under Python 2.3.
 
  greater than 255. (To work around this, the test_suite uses a later version
  of ``dis()``, but do note that it may affect your own tests if you use
  ``dis()`` with Python 2.3 and use widely separated line numbers.)
  
If you find any other issues, please let me know.
 
Please also keep in mind that this is a work in progress, and the API may

Questions and discussion regarding this software should be directed to the
`PEAK Mailing List <http://www.eby-sarna.com/mailman/listinfo/peak>`_.
 
.. _toc:
.. contents:: **Table of Contents**
 
 

 
As you can see, ``Code`` instances automatically generate a line number table
that maps each ``set_lineno()`` to the corresponding position in the bytecode.
 
    
And of course, the resulting code objects can be run with ``eval()`` or
``exec``, or used with ``new.function`` to create a function::
 

    >>> f()
    42
 
Finally, code objects are also iterable, yielding ``(offset, opcode, arg)``
tuples, where `arg` is ``None`` for opcodes with no arguments, and an integer
otherwise::
 
    >>> import peak.util.assembler as op
    >>> list(c) == [
    ... (0, op.LOAD_CONST, 1),
    ... (3, op.RETURN_VALUE, None)
    ... ]
    True
 
This can be useful for testing or otherwise inspecting code you've generated.
 
 
Symbolic Disassembler
=====================
 
Python's built-in disassembler can be verbose and hard to read when inspecting
complex generated code -- usually you don't care about bytecode offsets or
line numbers as much as you care about labels, for example.
 
So, BytecodeAssembler provides its own, simplified disassembler, which we'll
be using for more complex listings in this manual::
 
    >>> from peak.util.assembler import dump
 
Some sample output, that also showcases some of BytecodeAssembler's
`High-Level Code Generation`_ features::
 
    >>> c = Code()
    >>> from peak.util.assembler import Compare, Local
    >>> c.return_(Compare(Local('a'), [('<', Local('b')), ('<', Local('c'))]))
    >>> dump(c.code())
                    LOAD_FAST 0 (a)
                    LOAD_FAST 1 (b)
                    DUP_TOP
                    ROT_THREE
                    COMPARE_OP 0 (<)
                    JUMP_IF_FALSE L1
                    POP_TOP
                    LOAD_FAST 2 (c)
                    COMPARE_OP 0 (<)
                    JUMP_FORWARD L2
            L1: ROT_TWO
                    POP_TOP
            L2: RETURN_VALUE
 
As you can see, the line numbers and bytecode offsets have been dropped,
making it esier to see where the jumps go. (This also makes doctests more
robust against Python version changes, as ``dump()`` has some extra code to
make conditional jumps appear consistent across the major changes that were
made to conditional jump instructions between Python 2.6 and 2.7.)
 
 
Opcodes and Arguments
=====================

    >>> c.POP_TOP()
    >>> c.JUMP_ABSOLUTE(where) # now jump back to it
 
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (42)
            >> 3 DUP_TOP
                  4 POP_TOP
                  5 JUMP_ABSOLUTE 3
    >>> dump(c.code())
                    LOAD_CONST 1 (42)
            L1: DUP_TOP
                    POP_TOP
                    JUMP_ABSOLUTE L1
 
But if you are jumping *forward*, you will need to call the jump or setup
method without any arguments. The return value will be a "forward reference"

    >>> c.LOAD_CONST(23)
    >>> c.RETURN_VALUE()
 
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (99)
                  3 JUMP_IF_TRUE 4 (to 10)
                  6 LOAD_CONST 2 (42)
                  9 POP_TOP
            >> 10 LOAD_CONST 3 (23)
                 13 RETURN_VALUE
    >>> dump(c.code())
                    LOAD_CONST 1 (99)
                    JUMP_IF_TRUE L1
                    LOAD_CONST 2 (42)
                    POP_TOP
            L1: LOAD_CONST 3 (23)
                    RETURN_VALUE
 
    >>> eval(c.code())
    23

    >>> c = Code()
    >>> c.co_cellvars = ('a','b')
 
    >>> import sys
    >>> c.LOAD_CLOSURE('a')
    >>> c.LOAD_CLOSURE('b')
    >>> c.LOAD_CONST(None) # in real code, this'd be a Python code constant
    >>> c.MAKE_CLOSURE(0,2) # no defaults, 2 free vars in the new function
    >>> if sys.version>='2.5':
    ... c.BUILD_TUPLE(2) # In Python 2.5+, free vars must be in a tuple
    >>> c.LOAD_CONST(None) # in real code, this'd be a Python code constant
    >>> c.MAKE_CLOSURE(0,2) # no defaults, 2 free vars in the new function
 
    >>> c.stack_size # This will be 1, no matter what Python version
    1
 
The ``COMPARE_OP`` method takes an argument which can be a valid comparison
integer constant, or a string containing a Python operator, e.g.::
 
    >>> c = Code()
    >>> c.LOAD_CONST(1)
    >>> c.LOAD_CONST(2)
    >>> c.COMPARE_OP('not in')
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (1)
                  3 LOAD_CONST 2 (2)
                  6 COMPARE_OP 7 (not in)
 
The full list of valid operator strings can be found in the standard library's
``opcode`` module. ``"<>"`` is also accepted as an alias for ``"!="``::
 
    >>> c.LOAD_CONST(3)
    >>> c.COMPARE_OP('<>')
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (1)
                  3 LOAD_CONST 2 (2)
                  6 COMPARE_OP 7 (not in)
                  9 LOAD_CONST 3 (3)
                 12 COMPARE_OP 3 (!=)
 
 
High-Level Code Generation

                  6 LOAD_CONST 3 (1.0)
                  9 LOAD_CONST 4 (1L)
 
 
Simple Containers
-----------------
 

    False
 
 
``Suite`` and ``Pass``
----------------------
 
On occasion, it's helpful to be able to group a sequence of opcodes,
expressions, or statements together, to be passed as an argument to other node
types. The ``Suite`` node type accomplishes this::
 
    >>> from peak.util.assembler import Suite, Pass
 
    >>> c = Code()
    >>> c.return_(Suite([Const(42), Code.DUP_TOP, Code.POP_TOP]))
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (42)
                  3 DUP_TOP
                  4 POP_TOP
                  5 RETURN_VALUE
 
And ``Pass`` is a shortcut for an empty ``Suite``, that generates nothing::
 
    >>> Suite([])
    Pass
 
    >>> c = Code()
    >>> c(Pass)
    >>> c.return_(None)
    >>> dis(c.code())
      0 0 LOAD_CONST 0 (None)
                  3 RETURN_VALUE
 
 
Local and Global Names
----------------------
 

      0 0 LOAD_FAST 0 (x)
                  3 LOAD_GLOBAL 0 (y)
 
 
As with simple constants and ``Const`` wrappers, these objects can be used to
construct more complex expressions, like ``{a:(b,c)}``::
 

                 16 ROT_THREE
                 17 STORE_SUBSCR
 
The ``LocalAssign`` node type takes a name, and stores a value in a local
variable::
 
    >>> from peak.util.assembler import LocalAssign
    >>> c = Code()
    >>> c(42, LocalAssign('x'))
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (42)
                  3 STORE_FAST 0 (x)
 
If the code object is not using "fast locals" (i.e. ``CO_OPTIMIZED`` isn't
set), local variables will be dereferenced using ``LOAD_NAME`` instead of
``LOAD_FAST``, and if the referenced local name is a "cell" or "free"
variable, ``LOAD_DEREF`` is used instead::
set), local variables will be referenced using ``LOAD_NAME`` and ``STORE_NAME``
instead of ``LOAD_FAST`` and ``STORE_FAST``, and if the referenced local name
is a "cell" or "free" variable, ``LOAD_DEREF`` and ``STORE_DEREF`` are used
instead::
 
    >>> from peak.util.assembler import CO_OPTIMIZED
    >>> c = Code()

    >>> c.co_cellvars = ('y',)
    >>> c.co_freevars = ('z',)
    >>> c( Local('x'), Local('y'), Local('z') )
    >>> c( LocalAssign('x'), LocalAssign('y'), LocalAssign('z') )
    >>> dis(c.code())
      0 0 LOAD_NAME 0 (x)
                  3 LOAD_DEREF 0 (y)
                  6 LOAD_DEREF 1 (z)
                  9 STORE_NAME 0 (x)
                 12 STORE_DEREF 0 (y)
                 15 STORE_DEREF 1 (z)
 
 
Obtaining Attributes
--------------------
 
The ``Getattr`` node type takes an expression and an attribute name. The
attribute name can be a constant string, in which case a ``LOAD_ATTR`` opcode
is used, and constant folding is done if possible::
 
    >>> from peak.util.assembler import Getattr
 
    >>> c = Code()
    >>> c(Getattr(Local('x'), '__class__'))
    >>> dis(c.code())
      0 0 LOAD_FAST 0 (x)
                  3 LOAD_ATTR 0 (__class__)
 
 
    >>> Getattr(Const(object), '__class__') # const expression, const result
    Const(<type 'type'>)
 
Or the attribute name can be an expression, in which case a ``getattr()`` call
is compiled instead::
 
    >>> c = Code()
    >>> c(Getattr(Local('x'), Local('y')))
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (<built-in function getattr>)
                  3 LOAD_FAST 0 (x)
                  6 LOAD_FAST 1 (y)
                  9 CALL_FUNCTION 2
 
 
Calling Functions and Methods

                  3 RETURN_VALUE
 
 
``If`` Conditions
-----------------
 
The ``If()`` node type generates conditional code, roughly equivalent to a
Python if/else statement::
 
    >>> from peak.util.assembler import If
    >>> c = Code()
    >>> c( If(Local('a'), Return(42), Return(55)) )
    >>> dump(c.code())
                    LOAD_FAST 0 (a)
                    JUMP_IF_FALSE L1
                    POP_TOP
                    LOAD_CONST 1 (42)
                    RETURN_VALUE
            L1: POP_TOP
                    LOAD_CONST 2 (55)
                    RETURN_VALUE
 
However, it can also be used like a Python 2.5+ conditional expression
(regardless of the targeted Python version)::
 
    >>> c = Code()
    >>> c( Return(If(Local('a'), 42, 55)) )
    >>> dump(c.code())
                    LOAD_FAST 0 (a)
                    JUMP_IF_FALSE L1
                    POP_TOP
                    LOAD_CONST 1 (42)
                    JUMP_FORWARD L2
            L1: POP_TOP
                    LOAD_CONST 2 (55)
            L2: RETURN_VALUE
 
 
Note that ``If()`` does *not* do constant-folding on its condition; even if the
condition is a constant, it will be tested at runtime. This avoids issues with
using mutable constants, e.g.::
 
    >>> c = Code()
    >>> c(If(Const([]), 42, 55))
    >>> dump(c.code())
                    LOAD_CONST 1 ([])
                    JUMP_IF_FALSE L1
                    POP_TOP
                    LOAD_CONST 2 (42)
                    JUMP_FORWARD L2
            L1: POP_TOP
                    LOAD_CONST 3 (55)
 
 
Labels and Jump Targets
-----------------------
 

    >>> c.LOAD_CONST(99)
    >>> forward = c.JUMP_IF_FALSE()
    >>> c( 1, Code.POP_TOP, forward, Return(3) )
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (99)
                  3 JUMP_IF_FALSE 4 (to 10)
                  6 LOAD_CONST 2 (1)
                  9 POP_TOP
            >> 10 LOAD_CONST 3 (3)
                 13 RETURN_VALUE
    >>> dump(c.code())
                    LOAD_CONST 1 (99)
                    JUMP_IF_FALSE L1
                    LOAD_CONST 2 (1)
                    POP_TOP
            L1: LOAD_CONST 3 (3)
                    RETURN_VALUE
 
However, there's an easier way to do the same thing, using ``Label`` objects::
 

    >>> skip = Label()
 
    >>> c(99, skip.JUMP_IF_FALSE, 1, Code.POP_TOP, skip, Return(3))
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (99)
                  3 JUMP_IF_FALSE 4 (to 10)
                  6 LOAD_CONST 2 (1)
                  9 POP_TOP
            >> 10 LOAD_CONST 3 (3)
                 13 RETURN_VALUE
    >>> dump(c.code())
                    LOAD_CONST 1 (99)
                    JUMP_IF_FALSE L1
                    LOAD_CONST 2 (1)
                    POP_TOP
            L1: LOAD_CONST 3 (3)
                    RETURN_VALUE
 
This approach has the advantage of being easy to use in complex trees.
``Label`` objects have attributes corresponding to every opcode that uses a

    AssertionError: Label previously defined
 
 
More Conditional Jump Instructions
----------------------------------
 
In Python 2.7, the traditional ``JUMP_IF_TRUE`` and ``JUMP_IF_FALSE``
instructions were replaced with four new instructions that either conditionally
or unconditionally pop the value being tested. This was done to improve
performance, since virtually all conditional jumps in Python code pop the
value on one branch or the other.
 
To provide better cross-version compatibility, BytecodeAssembler emulates the
old instructions on Python 2.7 by emitting a ``DUP_TOP`` followed by a
``POP_JUMP_IF_FALSE`` or ``POP_JUMP_IF_TRUE`` instruction.
 
However, since this decreases performance, BytecodeAssembler *also* emulates
Python 2.7's ``JUMP_IF_FALSE_OR_POP`` and ``JUMP_IF_FALSE_OR_TRUE`` opcodes
on *older* Pythons::
 
    >>> c = Code()
    >>> l1, l2 = Label(), Label()
    >>> c(Local('a'), l1.JUMP_IF_FALSE_OR_POP, Return(27), l1)
    >>> c(l2.JUMP_IF_TRUE_OR_POP, Return(42), l2, Code.RETURN_VALUE)
    >>> dump(c.code())
                    LOAD_FAST 0 (a)
                    JUMP_IF_FALSE L1
                    POP_TOP
                    LOAD_CONST 1 (27)
                    RETURN_VALUE
            L1: JUMP_IF_TRUE L2
                    POP_TOP
                    LOAD_CONST 2 (42)
                    RETURN_VALUE
            L2: RETURN_VALUE
 
This means that you can immediately begin using the "or-pop" variations, in
place of a jump followed by a pop, and BytecodeAssembler will use the faster
single instruction automatically on Python 2.7+.
 
BytecodeAssembler *also* supports using Python 2.7's conditional jumps
that do unconditional pops, but currently cannot emulate them on older Python
versions, so at the moment you should use them only when your code requires
Python 2.7.
 
(Note: for ease in doctesting across Python versions, the ``dump()`` function
*always* shows the code as if it were generated for Python 2.6 or lower, so
if you need to check the *actual* bytecodes generated, you must use Python's
``dis.dis()`` function instead!)
 
 
N-Way Comparisons
-----------------
 
You can generate N-way comparisons using the ``Compare()`` node type::
 
    >>> from peak.util.assembler import Compare
 
    >>> c = Code()
    >>> c(Compare(Local('a'), [('<', Local('b'))]))
    >>> dis(c.code())
      0 0 LOAD_FAST 0 (a)
                  3 LOAD_FAST 1 (b)
                  6 COMPARE_OP 0 (<)
 
3-way comparisons generate code that's a bit more complex. Here's a three-way
comparison (``a<b<c``)::
 
    >>> c = Code()
    >>> c.return_(Compare(Local('a'), [('<', Local('b')), ('<', Local('c'))]))
    >>> dump(c.code())
                    LOAD_FAST 0 (a)
                    LOAD_FAST 1 (b)
                    DUP_TOP
                    ROT_THREE
                    COMPARE_OP 0 (<)
                    JUMP_IF_FALSE L1
                    POP_TOP
                    LOAD_FAST 2 (c)
                    COMPARE_OP 0 (<)
                    JUMP_FORWARD L2
            L1: ROT_TWO
                    POP_TOP
            L2: RETURN_VALUE
 
And a four-way (``a<b>c!=d``)::
 
    >>> c = Code()
    >>> c.return_(
    ... Compare( Local('a'), [
    ... ('<', Local('b')), ('>', Local('c')), ('!=', Local('d'))
    ... ])
    ... )
    >>> dump(c.code())
                    LOAD_FAST 0 (a)
                    LOAD_FAST 1 (b)
                    DUP_TOP
                    ROT_THREE
                    COMPARE_OP 0 (<)
                    JUMP_IF_FALSE L1
                    POP_TOP
                    LOAD_FAST 2 (c)
                    DUP_TOP
                    ROT_THREE
                    COMPARE_OP 4 (>)
                    JUMP_IF_FALSE L1
                    POP_TOP
                    LOAD_FAST 3 (d)
                    COMPARE_OP 3 (!=)
                    JUMP_FORWARD L2
            L1: ROT_TWO
                    POP_TOP
            L2: RETURN_VALUE
 
 
Sequence Unpacking
------------------
 
The ``UnpackSequence`` node type takes a sequence of code generation targets,
and generates an ``UNPACK_SEQUENCE`` of the correct length, followed by the
targets::
 
    >>> from peak.util.assembler import UnpackSequence
    >>> c = Code()
    >>> c((1,2), UnpackSequence([LocalAssign('x'), LocalAssign('y')]))
    >>> dis(c.code()) # x, y = 1, 2
      0 0 LOAD_CONST 1 (1)
                  3 LOAD_CONST 2 (2)
                  6 BUILD_TUPLE 2
                  9 UNPACK_SEQUENCE 2
                 12 STORE_FAST 0 (x)
                 15 STORE_FAST 1 (y)
 
 
Yield Statements
----------------
 
The ``YieldStmt`` node type generates the necessary opcode(s) for a ``yield``
statement, based on the target Python version. (In Python 2.5+, a ``POP_TOP``
must be generated after a ``YIELD_VALUE`` in order to create a yield statement,
as opposed to a yield expression.) It also sets the code flags needed to make
the resulting code object a generator::
 
    >>> from peak.util.assembler import YieldStmt
    >>> c = Code()
    >>> c(YieldStmt(1), YieldStmt(2), Return(None))
    >>> list(eval(c.code()))
    [1, 2]
 
 
 
Constant Detection and Folding
==============================
 

    >>> const_value(Local('x'))
    Traceback (most recent call last):
      ...
    NotAConstant: Local('x',)
    NotAConstant: Local('x')
 
Tuples of constants are recursively replaced by constant tuples::
 

    >>> const_value( (1,Global('y')) )
    Traceback (most recent call last):
      ...
    NotAConstant: Global('y',)
    NotAConstant: Global('y')
 
As do any types not previously described here::
 

``globals()``, in other words.
 
 
Logical And/Or
--------------
 
You can evaluate logical and/or expressions using the ``And`` and ``Or`` node
types::
 
    >>> from peak.util.assembler import And, Or
 
    >>> c = Code()
    >>> c.return_( And([Local('x'), Local('y')]) )
    >>> dump(c.code())
                    LOAD_FAST 0 (x)
                    JUMP_IF_FALSE L1
                    POP_TOP
                    LOAD_FAST 1 (y)
            L1: RETURN_VALUE
 
    >>> c = Code()
    >>> c.return_( Or([Local('x'), Local('y')]) )
    >>> dump(c.code())
                    LOAD_FAST 0 (x)
                    JUMP_IF_TRUE L1
                    POP_TOP
                    LOAD_FAST 1 (y)
            L1: RETURN_VALUE
 
 
True or false constants are folded automatically, avoiding code generation
for intermediate values that will never be used in the result::
 
    >>> c = Code()
    >>> c.return_( And([1, 2, Local('y')]) )
    >>> dis(c.code())
      0 0 LOAD_FAST 0 (y)
                  3 RETURN_VALUE
 
    >>> c = Code()
    >>> c.return_( And([1, 2, Local('y'), 0]) )
    >>> dump(c.code())
                    LOAD_FAST 0 (y)
                    JUMP_IF_FALSE L1
                    POP_TOP
                    LOAD_CONST 1 (0)
            L1: RETURN_VALUE
 
    >>> c = Code()
    >>> c.return_( Or([1, 2, Local('y')]) )
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (1)
                  3 RETURN_VALUE
 
    >>> c = Code()
    >>> c.return_( Or([False, Local('y'), 3]) )
    >>> dump(c.code())
                    LOAD_FAST 0 (y)
                    JUMP_IF_TRUE L1
                    POP_TOP
                    LOAD_CONST 1 (3)
            L1: RETURN_VALUE
 
 
Custom Code Generation
======================
 

 
    >>> c = Code()
    >>> c( TryFinally(ExprStmt(1), ExprStmt(2)) )
    >>> dis(c.code())
      0 0 SETUP_FINALLY 8 (to 11)
                  3 LOAD_CONST 1 (1)
                  6 POP_TOP
                  7 POP_BLOCK
                  8 LOAD_CONST 0 (None)
            >> 11 LOAD_CONST 2 (2)
                 14 POP_TOP
                 15 END_FINALLY
    >>> dump(c.code())
                    SETUP_FINALLY L1
                    LOAD_CONST 1 (1)
                    POP_TOP
                    POP_BLOCK
                    LOAD_CONST 0 (None)
            L1: LOAD_CONST 2 (2)
                    POP_TOP
                    END_FINALLY
 
The ``nodetype()`` decorator is virtually identical to the ``struct()``
decorator in the DecoratorTools package, except that it does not support

 
    >>> tf = TryFinally(ExprStmt(1), ExprStmt(2))
    >>> tf
    TryFinally(ExprStmt(1,), ExprStmt(2,))
    TryFinally(ExprStmt(1), ExprStmt(2))
 
* It makes named fields accessible::
 
    >>> tf.block1
    ExprStmt(1,)
    ExprStmt(1)
 
    >>> tf.block2
    ExprStmt(2,)
    ExprStmt(2)
 
* Hashing and comparison work as expected (handy for algorithms that require
comparing or caching AST subtrees, such as common subexpression elimination)::
  comparing or caching AST subtrees, such as common subexpression
  elimination)::
 
    >>> ExprStmt(1) == ExprStmt(1)
    True

 
If you want to incorporate constant-folding into your AST nodes, you can do
so by checking for constant values and folding them at either construction
or code generation time. For example, this ``And`` node type folds constants
during code generation, by not generating unnecessary branches when it can
or code generation time. For example, this ``And`` node type (a simpler
version of the one included in ``peak.util.assembler``) folds constants during
code generation, by not generating unnecessary branches when it can
prove which way a branch will go::
 
    >>> from peak.util.assembler import NotAConstant

    ... if const_value(value):
    ... continue # true constants can be skipped
    ... except NotAConstant: # but non-constants require code
    ... code(value, end.JUMP_IF_FALSE, Code.POP_TOP)
    ... code(value, end.JUMP_IF_FALSE_OR_POP)
    ... else: # and false constants end the chain right away
    ... return code(value, end)
    ... code(values[-1], end)

 
    >>> c = Code()
    >>> c.return_( And([Local('x'), False, 27]) )
    >>> dis(c.code())
      0 0 LOAD_FAST 0 (x)
                  3 JUMP_IF_FALSE 4 (to 10)
                  6 POP_TOP
                  7 LOAD_CONST 1 (False)
            >> 10 RETURN_VALUE
    >>> dump(c.code())
                    LOAD_FAST 0 (x)
                    JUMP_IF_FALSE L1
                    POP_TOP
                    LOAD_CONST 1 (False)
            L1: RETURN_VALUE
 
The above example only folds constants at code generation time, however. You
can also do constant folding at AST construction time, using the

implements its ``fold`` argument and the suppression of folding when
the call has no arguments.
 
(By the way, this same ``Getattr`` node type is also available
 
 
Setting the Code's Calling Signature
====================================

 
    >>> import inspect
 
    >>> inspect.getargspec(f1)
    >>> tuple(inspect.getargspec(f1))
    (['a', 'b'], 'c', 'd', None)
 
    >>> inspect.getargspec(f2)
    >>> tuple(inspect.getargspec(f2))
    (['a', 'b'], 'c', 'd', None)
 
Note that these constructors do not copy any actual *code* from the code

    >>> c1 = Code.from_function(f1, copy_lineno=True)
    >>> c1.co_firstlineno
    1
    >>> c1.co_filename is f1.func_code.co_filename
    True
 
If you create a ``Code`` instance from a function that has nested positional
arguments, the returned code object will include a prologue to unpack the

unpacking process, and is designed so that the ``inspect`` module will
recognize it as an argument unpacking prologue::
 
    >>> inspect.getargspec(f3)
    >>> tuple(inspect.getargspec(f3))
    (['a', ['b', 'c'], ['d', ['e', 'f']]], None, None, None)
 
    >>> inspect.getargspec(f4)
    >>> tuple(inspect.getargspec(f4))
    (['a', ['b', 'c'], ['d', ['e', 'f']]], None, None, None)
 
You can also use the ``from_spec(name='<lambda>', args=(), var=None, kw=None)``
classmethod to explicitly set a name and argument spec for a new code object::
 
    >>> c = Code.from_spec('a', ('b', ('c','d'), 'e'), 'f', 'g')
    >>> c.co_name
    'a'
 
    >>> c.co_varnames
    ['b', '.1', 'e', 'f', 'g', 'c', 'd']
 
    >>> c.co_argcount
    3
    
    >>> tuple(inspect.getargs(c.code()))
    (['b', ['c', 'd'], 'e'], 'f', 'g')
 
 
Code Attributes
===============

    42
 
    >>> import inspect
    >>> inspect.getargspec(f)
    >>> tuple(inspect.getargspec(f))
    (['a', 'b', 'c'], None, None, None)
 
Although Python code objects want ``co_varnames`` to be a tuple, ``Code``

code that might be unreachable. For example, consider this ``If``
implementation::
 
    >>> def Pass(code=None):
    ... if code is None:
    ... return Pass
 
    >>> def If(cond, then, else_=Pass, code=None):
    ... if code is None:
    ... return cond, then, else_
    ... else_clause = Label()
    ... end_if = Label()
    ... code(cond, else_clause.JUMP_IF_FALSE, Code.POP_TOP, then)
    ... code(cond, else_clause.JUMP_IF_FALSE_OR_POP, then)
    ... code(end_if.JUMP_FORWARD, else_clause, Code.POP_TOP, else_)
    ... code(end_if)
    >>> If = nodetype()(If)

It works okay if there's no dead code::
 
    >>> c = Code()
    >>> c( If(23, 42, 55) )
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (23)
                  3 JUMP_IF_FALSE 7 (to 13)
                  6 POP_TOP
                  7 LOAD_CONST 2 (42)
                 10 JUMP_FORWARD 4 (to 17)
            >> 13 POP_TOP
                 14 LOAD_CONST 3 (55)
    >>> c( If(Local('a'), 42, 55) )
    >>> dump(c.code())
                    LOAD_FAST 0 (a)
                    JUMP_IF_FALSE L1
                    POP_TOP
                    LOAD_CONST 1 (42)
                    JUMP_FORWARD L2
            L1: POP_TOP
                    LOAD_CONST 2 (55)
 
But it breaks if you end the "then" block with a return::
 

    ... return cond, then, else_
    ... else_clause = Label()
    ... end_if = Label()
    ... code(cond, else_clause.JUMP_IF_FALSE, Code.POP_TOP, then)
    ... code(cond, else_clause.JUMP_IF_FALSE_OR_POP, then)
    ... if code.stack_size is not None:
    ... end_if.JUMP_FORWARD(code)
    ... code(else_clause, Code.POP_TOP, else_, end_if)
    ... code(else_clause, Code.POP_TOP, else_, end_if)
    >>> If = nodetype()(If)
 
As you can see, the dead code is now eliminated::
 
    >>> c = Code()
    >>> c( If(23, Return(42), 55) )
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (23)
                  3 JUMP_IF_FALSE 5 (to 11)
                  6 POP_TOP
                  7 LOAD_CONST 2 (42)
                 10 RETURN_VALUE
            >> 11 POP_TOP
                 12 LOAD_CONST 3 (55)
    >>> c( If(Local('a'), Return(42), 55) )
    >>> dump(c.code())
                    LOAD_FAST 0 (a)
                    JUMP_IF_FALSE L1
                    POP_TOP
                    LOAD_CONST 1 (42)
                    RETURN_VALUE
            L1: POP_TOP
                    LOAD_CONST 2 (55)
 
 
Blocks, Loops, and Exception Handling

    >>> c.POP_TOP()
    >>> else_()
    >>> c.return_()
    >>> dis(c.code())
      0 0 SETUP_EXCEPT 4 (to 7)
                  3 POP_BLOCK
                  4 JUMP_FORWARD 3 (to 10)
            >> 7 POP_TOP
                  8 POP_TOP
                  9 POP_TOP
            >> 10 LOAD_CONST 0 (None)
                 13 RETURN_VALUE
    >>> dump(c.code())
                    SETUP_EXCEPT L1
                    POP_BLOCK
                    JUMP_FORWARD L2
            L1: POP_TOP
                    POP_TOP
                    POP_TOP
            L2: LOAD_CONST 0 (None)
                    RETURN_VALUE
 
In the example above, an empty block executes with an exception handler that
begins at offset 7. When the block is done, it jumps forward to the end of

    ... Return()
    ... )
 
    >>> dis(c.code())
      0 0 SETUP_EXCEPT 4 (to 7)
                  3 POP_BLOCK
                  4 JUMP_FORWARD 3 (to 10)
            >> 7 POP_TOP
                  8 POP_TOP
                  9 POP_TOP
            >> 10 LOAD_CONST 0 (None)
                 13 RETURN_VALUE
    >>> dump(c.code())
                    SETUP_EXCEPT L1
                    POP_BLOCK
                    JUMP_FORWARD L2
            L1: POP_TOP
                    POP_TOP
                    POP_TOP
            L2: LOAD_CONST 0 (None)
                    RETURN_VALUE
 
(Labels have a ``POP_BLOCK`` attribute that you can pass in when generating
code.)
 
And, for generating typical try/except blocks, you can use the ``TryExcept``
node type, which takes a body, a sequence of exception-type/handler pairs,
and an optional "else" clause::
 
    >>> from peak.util.assembler import TryExcept
    >>> c = Code()
    >>> c.return_(
    ... TryExcept(
    ... Return(1), # body
    ... [(Const(KeyError),2), (Const(TypeError),3)], # handlers
    ... Return(4) # else clause
    ... )
    ... )
 
Labels have a ``POP_BLOCK`` attribute that you can pass in when generating
code.
    >>> dump(c.code())
                    SETUP_EXCEPT L1
                    LOAD_CONST 1 (1)
                    RETURN_VALUE
                    POP_BLOCK
                    JUMP_FORWARD L4
            L1: DUP_TOP
                    LOAD_CONST 2 (<...exceptions.KeyError...>)
                    COMPARE_OP 10 (exception match)
                    JUMP_IF_FALSE L2
                    POP_TOP
                    POP_TOP
                    POP_TOP
                    POP_TOP
                    LOAD_CONST 3 (2)
                    JUMP_FORWARD L5
            L2: POP_TOP
                    DUP_TOP
                    LOAD_CONST 4 (<...exceptions.TypeError...>)
                    COMPARE_OP 10 (exception match)
                    JUMP_IF_FALSE L3
                    POP_TOP
                    POP_TOP
                    POP_TOP
                    POP_TOP
                    LOAD_CONST 5 (3)
                    JUMP_FORWARD L5
            L3: POP_TOP
                    END_FINALLY
            L4: LOAD_CONST 6 (4)
                    RETURN_VALUE
            L5: RETURN_VALUE
 
 
Try/Finally Blocks

 
And it produces code that looks like this::
 
    >>> dis(c.code())
      0 0 SETUP_FINALLY 4 (to 7)
                  3 POP_BLOCK
                  4 LOAD_CONST 0 (None)
            >> 7 END_FINALLY
    >>> dump(c.code())
                    SETUP_FINALLY L1
                    POP_BLOCK
                    LOAD_CONST 0 (None)
            L1: END_FINALLY
 
The ``END_FINALLY`` opcode will remove 1, 2, or 3 values from the stack at
runtime, depending on how the "try" block was exited. In the case of simply

adjusts the maximum expected stack size to accomodate up to three values being
put on the stack by the Python interpreter for exception handling.
 
For your convenience, the ``TryFinally`` node type can also be used to generate
try/finally blocks::
 
    >>> from peak.util.assembler import TryFinally
    >>> c = Code()
    >>> c( TryFinally(ExprStmt(1), ExprStmt(2)) )
    >>> dump(c.code())
                    SETUP_FINALLY L1
                    LOAD_CONST 1 (1)
                    POP_TOP
                    POP_BLOCK
                    LOAD_CONST 0 (None)
            L1: LOAD_CONST 2 (2)
                    POP_TOP
                    END_FINALLY
 
 
Loops
-----

    ... Return()
    ... )
 
    >>> dis(c.code())
      0 0 SETUP_LOOP 19 (to 22)
                  3 LOAD_CONST 1 (5)
            >> 6 JUMP_IF_FALSE 7 (to 16)
                  9 LOAD_CONST 2 (1)
                 12 BINARY_SUBTRACT
                 13 JUMP_ABSOLUTE 6
            >> 16 POP_TOP
                 17 POP_BLOCK
                 18 LOAD_CONST 3 (42)
                 21 RETURN_VALUE
            >> 22 LOAD_CONST 0 (None)
                 25 RETURN_VALUE
    >>> dump(c.code())
                    SETUP_LOOP L3
                    LOAD_CONST 1 (5)
            L1: JUMP_IF_FALSE L2
                    LOAD_CONST 2 (1)
                    BINARY_SUBTRACT
                    JUMP_ABSOLUTE L1
            L2: POP_TOP
                    POP_BLOCK
                    LOAD_CONST 3 (42)
                    RETURN_VALUE
            L3: LOAD_CONST 0 (None)
                    RETURN_VALUE
 
    >>> eval(c.code())
    42

    >>> fwd()
    >>> c.BREAK_LOOP()
    >>> c.POP_BLOCK()()
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (57)
                  3 SETUP_LOOP 8 (to 14)
                  6 JUMP_IF_TRUE 3 (to 12)
            >> 9 JUMP_ABSOLUTE 9
            >> 12 BREAK_LOOP
                 13 POP_BLOCK
    >>> dump(c.code())
                    LOAD_CONST 1 (57)
                    SETUP_LOOP L3
                    JUMP_IF_TRUE L2
            L1: JUMP_ABSOLUTE L1
            L2: BREAK_LOOP
                    POP_BLOCK
 
In other words, ``CONTINUE_LOOP`` only really emits a ``CONTINUE_LOOP`` opcode
if it's inside some other kind of block within the loop, e.g. a "try" clause::

    >>> c.POP_BLOCK()
    >>> c.END_FINALLY()
    >>> c.POP_BLOCK()()
    >>> dump(c.code())
                    LOAD_CONST 1 (57)
                    SETUP_LOOP L4
            L1: SETUP_FINALLY L3
                    JUMP_IF_TRUE L2
                    CONTINUE_LOOP L1
            L2: POP_BLOCK
                    LOAD_CONST 0 (None)
            L3: END_FINALLY
                    POP_BLOCK
 
``for`` Loops
-------------
 
There is a ``For()`` node type available for generating simple loops (without
break/continue support). It takes an iterable expression, an assignment
clause, and a loop body::
 
    >>> from peak.util.assembler import For
    >>> y = Call(Const(range), (3,))
    >>> x = LocalAssign('x')
    >>> body = Suite([Local('x'), Code.PRINT_EXPR])
 
    >>> c = Code()
    >>> c(For(y, x, body)) # for x in range(3): print x
    >>> c.return_()
    >>> dump(c.code())
                    LOAD_CONST 1 ([0, 1, 2])
                    GET_ITER
            L1: FOR_ITER L2
                    STORE_FAST 0 (x)
                    LOAD_FAST 0 (x)
                    PRINT_EXPR
                    JUMP_ABSOLUTE L1
            L2: LOAD_CONST 0 (None)
                    RETURN_VALUE
 
The arguments are given in execution order: first the "in" value of the loop,
then the assignment to a loop variable, and finally the body of the loop. The
distinction between the assignment and body, however, is only for clarity and
convenience (to avoid needing to glue the assignment to the body with a
``Suite``). If you already have a suite or only need one node for the entire
loop body, you can do the same thing with only two arguments::
 
    >>> c = Code()
    >>> c(For(y, Code.PRINT_EXPR))
    >>> c.return_()
    >>> dump(c.code())
                    LOAD_CONST 1 ([0, 1, 2])
                    GET_ITER
            L1: FOR_ITER L2
                    PRINT_EXPR
                    JUMP_ABSOLUTE L1
            L2: LOAD_CONST 0 (None)
                    RETURN_VALUE
 
Notice, by the way, that ``For()`` does NOT set up a loop block for you, so if
you want to be able to use break and continue, you'll need to wrap the loop in
a labelled SETUP_LOOP/POP_BLOCK pair, as described in the preceding sections.
 
 
List Comprehensions
-------------------
 
In order to generate correct list comprehension code for the target Python
version, you must use the ``ListComp()`` and ``LCAppend()`` node types. This
is because Python versions 2.4 and up store the list being built in a temporary
variable, and use a special ``LIST_APPEND`` opcode to append values, while 2.3
stores the list's ``append()`` method in the temporary variable, and calls it
to append values.
 
The ``ListComp()`` node wraps a code body (usually a ``For()`` loop) and
manages the creation and destruction of a temporary variable (e.g. ``_[1]``,
``_[2]``, etc.). The ``LCAppend()`` node type wraps a value or expression to
be appended to the innermost active ``ListComp()`` in progress::
 
    >>> from peak.util.assembler import ListComp, LCAppend
    >>> c = Code()
    >>> simple = ListComp(For(y, x, LCAppend(Local('x'))))
    >>> c.return_(simple)
    >>> eval(c.code())
    [0, 1, 2]
 
    >>> c = Code()
    >>> c.return_(ListComp(For(y, x, LCAppend(simple))))
    >>> eval(c.code())
    [[0, 1, 2], [0, 1, 2], [0, 1, 2]]
 
 
Closures and Nested Functions
=============================
 
Free and Cell Variables
-----------------------
 
To implement closures and nested scopes, your code objects must use "free" or
"cell" variables in place of regular "fast locals". A "free" variable is one
that is defined in an outer scope, and a "cell" variable is one that's defined
in the current scope, but will also be used by nested functions.
 
The simplest way to set up free or cell variables is to use a code object's
``makefree(names)`` and ``makecells(names)`` methods::
 
    >>> c = Code()
    >>> c.co_cellvars
    ()
    >>> c.co_freevars
    ()
 
    >>> c.makefree(['x', 'y'])
    >>> c.makecells(['z'])
 
    >>> c.co_cellvars
    ('z',)
    >>> c.co_freevars
    ('x', 'y')
 
When a name has been defined as a free or cell variable, the ``_DEREF`` opcode
variants are used to generate ``Local()`` and ``LocalAssign()`` nodes::
 
    >>> c((Local('x'), Local('y')), LocalAssign('z'))
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (57)
                  3 SETUP_LOOP 15 (to 21)
            >> 6 SETUP_FINALLY 10 (to 19)
                  9 JUMP_IF_TRUE 3 (to 15)
                 12 CONTINUE_LOOP 6
            >> 15 POP_BLOCK
                 16 LOAD_CONST 0 (None)
            >> 19 END_FINALLY
                 20 POP_BLOCK
      0 0 LOAD_DEREF 1 (x)
                  3 LOAD_DEREF 2 (y)
                  6 BUILD_TUPLE 2
                  9 STORE_DEREF 0 (z)
 
If you have already written code in a code object that operates on the relevant
locals, the code is retroactively patched to use the ``_DEREF`` opcodes::
 
    >>> c = Code()
    >>> c((Local('x'), Local('y')), LocalAssign('z'))
    >>> dis(c.code())
      0 0 LOAD_FAST 0 (x)
                  3 LOAD_FAST 1 (y)
                  6 BUILD_TUPLE 2
                  9 STORE_FAST 2 (z)
 
    >>> c.makefree(['x', 'y'])
    >>> c.makecells(['z'])
 
    >>> dis(c.code())
      0 0 LOAD_DEREF 1 (x)
                  3 LOAD_DEREF 2 (y)
                  6 BUILD_TUPLE 2
                  9 STORE_DEREF 0 (z)
 
This means that you can defer the decision of which locals are free/cell
variables until the code is ready to be generated. In fact, by passing in
a "parent" code object to the ``.code()`` method, you can get BytecodeAssembler
to automatically call ``makefree()`` and ``makecells()`` for the correct
variable names in the child and parent code objects, as we'll see in the next
section.
 
 
Nested Code Objects
-------------------
 
To create a code object for use in a nested scope, you can use the parent code
object's ``nested()`` method. It works just like the ``from_spec()``
classmethod, except that the ``co_filename`` of the parent is copied to the
child::
 
    >>> p = Code()
    >>> p.co_filename = 'testname'
 
    >>> c = p.nested('sub', ['a','b'], 'c', 'd')
 
    >>> c.co_name
    'sub'
    
    >>> c.co_filename
    'testname'
 
    >>> tuple(inspect.getargs(c.code(p)))
    (['a', 'b'], 'c', 'd')
 
Notice that you must pass the parent code object to the child's ``.code()``
method to ensure that free/cell variables are properly set up. When the
``code()`` method is given another code object as a parameter, it automatically
converts any locally-read (but not written) to "free" variables in the child
code, and ensures that those same variables become "cell" variables in the
supplied parent code object::
 
    >>> p.LOAD_CONST(42)
    >>> p(LocalAssign('a'))
    >>> dis(p.code())
      0 0 LOAD_CONST 1 (42)
                  3 STORE_FAST 0 (a)
    
    >>> c = p.nested()
    >>> c(Local('a'))
 
    >>> dis(c.code(p))
      0 0 LOAD_DEREF 0 (a)
 
    >>> dis(p.code())
      0 0 LOAD_CONST 1 (42)
                  3 STORE_DEREF 0 (a)
 
Notice that the ``STORE_FAST`` in the parent code object was automatically
patched to a ``STORE_DEREF``, with an updated offset if applicable. Any
future use of ``Local('a')`` or ``LocalAssign('a')`` in the parent or child
code objects will now refer to the free/cell variable, rather than the "local"
variable::
 
    >>> p(Local('a'))
    >>> dis(p.code())
      0 0 LOAD_CONST 1 (42)
                  3 STORE_DEREF 0 (a)
                  6 LOAD_DEREF 0 (a)
 
    >>> c(LocalAssign('a'))
    >>> dis(c.code(p))
      0 0 LOAD_DEREF 0 (a)
                  3 STORE_DEREF 0 (a)
 
 
``Function()``
--------------
 
The ``Function(body, name='<lambda>', args=(), var=None, kw=None, defaults=())``
node type creates a function object from the specified body and the optional
name, argument specs, and defaults. The ``Function()`` node generates code to
create the function object with the appropriate defaults and closure (if
applicable), and any needed free/cell variables are automatically set up in the
parent and child code objects. The newly generated function will be on top of
the stack at the end of the generated code::
 
    >>> from peak.util.assembler import Function
    >>> c = Code()
    >>> c.co_filename = '<string>'
    >>> c.return_(Function(Return(Local('a')), 'f', ['a'], defaults=[42]))
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (42)
                  3 LOAD_CONST 2 (<... f ..., file "<string>", line -1>)
                  6 MAKE_FUNCTION 1
                  9 RETURN_VALUE
 
Now that we've generated the code for a function returning a function, let's
run it, to get the function we defined::
 
    >>> f = eval(c.code())
    >>> f
    <function f at ...>
 
    >>> tuple(inspect.getargspec(f))
    (['a'], None, None, (42,))
 
    >>> f()
    42
 
    >>> f(99)
    99
 
Now let's create a doubly nested function, with some extras::
 
    >>> c = Code()
    >>> c.co_filename = '<string>'
    >>> c.return_(
    ... Function(Return(Function(Return(Local('a')))),
    ... 'f', ['a', 'b'], 'c', 'd', [99, 66])
    ... )
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (99)
                  3 LOAD_CONST 2 (66)
                  6 LOAD_CONST 3 (<... f ..., file "<string>", line -1>)
                  9 MAKE_FUNCTION 2
                 12 RETURN_VALUE
 
    >>> f = eval(c.code())
    >>> f
    <function f at ...>
 
    >>> tuple(inspect.getargspec(f))
    (['a', 'b'], 'c', 'd', (99, 66))
 
    >>> dis(f)
      0 0 LOAD_CLOSURE 0 (a)
                  ... LOAD_CONST 1 (<... <lambda> ..., file "<string>", line -1>)
                  ... MAKE_CLOSURE 0
                  ... RETURN_VALUE
 
    >>> dis(f())
      0 0 LOAD_DEREF 0 (a)
                  3 RETURN_VALUE
 
    >>> f(42)()
    42
 
    >>> f()()
    99
 
As you can see, ``Function()`` not only takes care of setting up free/cell
variables in all the relevant scopes, it also chooses whether to use
``MAKE_FUNCTION`` or ``MAKE_CLOSURE``, and generates code for the defaults.
 
(Note, by the way, that the `defaults` argument should be a sequence of
generatable expressions; in the examples here, we used numbers, but they could
have been arbitrary expression nodes.)
 
 
----------------------

    >>> simple_code(1,1).co_stacksize
    1
 
    >>> dis(simple_code(13,414)) # FAILURE EXPECTED IN PYTHON 2.3
    >>> dis(simple_code(13,414))
     13 0 LOAD_CONST 0 (None)
    414 3 RETURN_VALUE
 

    >>> simple_code(13,14,100).co_stacksize
    100
 
    >>> dis(simple_code(13,572,120)) # FAILURE EXPECTED IN Python 2.3
    >>> dis(simple_code(13,572,120))
     13 0 LOAD_CONST 0 (None)
                  3 LOAD_CONST 0 (None)
    ...

                  3 LOAD_ATTR 1 (bar)
                  6 DELETE_FAST 0 (baz)
 
Code iteration::
 
    >>> c.DUP_TOP()
    >>> c.return_(Code.POP_TOP)
    >>> list(c) == [
    ... (0, op.LOAD_GLOBAL, 0),
    ... (3, op.LOAD_ATTR, 1),
    ... (6, op.DELETE_FAST, 0),
    ... (9, op.DUP_TOP, None),
    ... (10, op.POP_TOP, None),
    ... (11, op.RETURN_VALUE, None)
    ... ]
    True
 
Code patching::
 
    >>> c = Code()
    >>> c.LOAD_CONST(42)
    >>> c.STORE_FAST('x')
    >>> c.LOAD_FAST('x')
    >>> c.DELETE_FAST('x')
    >>> c.RETURN_VALUE()
 
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (42)
                  3 STORE_FAST 0 (x)
                  6 LOAD_FAST 0 (x)
                  9 DELETE_FAST 0 (x)
                 12 RETURN_VALUE
 
 
    >>> c.co_varnames
    ['x']
    >>> c.co_varnames.append('y')
 
    >>> c._patch(
    ... {op.LOAD_FAST: op.LOAD_FAST,
    ... op.STORE_FAST: op.STORE_FAST,
    ... op.DELETE_FAST: op.DELETE_FAST},
    ... {0: 1}
    ... )
 
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (42)
                  3 STORE_FAST 1 (y)
                  6 LOAD_FAST 1 (y)
                  9 DELETE_FAST 1 (y)
                 12 RETURN_VALUE
 
    >>> c._patch({op.RETURN_VALUE: op.POP_TOP})
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (42)
                  3 STORE_FAST 1 (y)
                  6 LOAD_FAST 1 (y)
                  9 DELETE_FAST 1 (y)
                 12 POP_TOP
 
Converting locals to free/cell vars::
 
    >>> c = Code()
    >>> c.LOAD_CONST(42)
    >>> c.STORE_FAST('x')
    >>> c.LOAD_FAST('x')
 
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (42)
                  3 STORE_FAST 0 (x)
                  6 LOAD_FAST 0 (x)
 
    >>> c.co_freevars = 'y', 'x'
    >>> c.co_cellvars = 'z',
 
    >>> c._locals_to_cells()
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (42)
                  3 STORE_DEREF 2 (x)
                  6 LOAD_DEREF 2 (x)
 
    >>> c.DELETE_FAST('x')
    >>> c._locals_to_cells()
    Traceback (most recent call last):
      ...
    AssertionError: Can't delete local 'x' used in nested scope
 
    >>> c = Code()
    >>> c.LOAD_CONST(42)
    >>> c.STORE_FAST('x')
    >>> c.LOAD_FAST('x')
 
    >>> c.co_freevars
    ()
    >>> c.makefree(['x'])
    >>> c.co_freevars
    ('x',)
 
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (42)
                  3 STORE_DEREF 0 (x)
                  6 LOAD_DEREF 0 (x)
 
    >>> c = Code()
    >>> c.LOAD_CONST(42)
    >>> c.STORE_FAST('x')
    >>> c.LOAD_FAST('x')
    >>> c.makecells(['x'])
    >>> c.co_freevars
    ()
    >>> c.co_cellvars
    ('x',)
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (42)
                  3 STORE_DEREF 0 (x)
                  6 LOAD_DEREF 0 (x)
    
    >>> c = Code()
    >>> c.LOAD_CONST(42)
    >>> c.STORE_FAST('x')
    >>> c.LOAD_FAST('x')
    >>> c.makefree('x')
    >>> c.makecells(['y'])
    >>> c.co_freevars
    ('x',)
    >>> c.co_cellvars
    ('y',)
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (42)
                  3 STORE_DEREF 1 (x)
                  6 LOAD_DEREF 1 (x)
 
    >>> c = Code()
    >>> c.co_flags &= ~op.CO_OPTIMIZED
    >>> c.makecells(['q'])
    Traceback (most recent call last):
      ...
    AssertionError: Can't use cellvars in unoptimized scope
    
 
 
Auto-free promotion with code parent:
 
    >>> p = Code()
    >>> c = Code()
    >>> c.LOAD_FAST('x')
    >>> dis(c.code(p))
      0 0 LOAD_DEREF 0 (x)
    >>> p.co_cellvars
    ('x',)
 
    >>> p = Code()
    >>> c = Code.from_function(lambda x,y,z=2: None)
    >>> c.LOAD_FAST('x')
    >>> c.LOAD_FAST('y')
    >>> c.LOAD_FAST('z')
    
    >>> dis(c.code(p))
      0 0 LOAD_FAST 0 (x)
                  3 LOAD_FAST 1 (y)
                  6 LOAD_FAST 2 (z)
    >>> p.co_cellvars
    ()
 
    >>> c.LOAD_FAST('q')
    >>> dis(c.code(p))
      0 0 LOAD_FAST 0 (x)
                  3 LOAD_FAST 1 (y)
                  6 LOAD_FAST 2 (z)
                  9 LOAD_DEREF 0 (q)
    >>> p.co_cellvars
    ('q',)
 
    >>> p = Code()
    >>> c = Code.from_function(lambda x,*y,**z: None)
    >>> c.LOAD_FAST('q')
    >>> c.LOAD_FAST('x')
    >>> c.LOAD_FAST('y')
    >>> c.LOAD_FAST('z')
    >>> dis(c.code(p))
      0 0 LOAD_DEREF 0 (q)
                  3 LOAD_FAST 0 (x)
                  6 LOAD_FAST 1 (y)
                  9 LOAD_FAST 2 (z)
    >>> p.co_cellvars
    ('q',)
 
    >>> p = Code()
    >>> c = Code.from_function(lambda x,*y: None)
    >>> c.LOAD_FAST('x')
    >>> c.LOAD_FAST('y')
    >>> c.LOAD_FAST('z')
    >>> dis(c.code(p))
      0 0 LOAD_FAST 0 (x)
                  3 LOAD_FAST 1 (y)
                  6 LOAD_DEREF 0 (z)
    >>> p.co_cellvars
    ('z',)
 
    >>> p = Code()
    >>> c = Code.from_function(lambda x,**y: None)
    >>> c.LOAD_FAST('x')
    >>> c.LOAD_FAST('y')
    >>> c.LOAD_FAST('z')
    >>> dis(c.code(p))
      0 0 LOAD_FAST 0 (x)
                  3 LOAD_FAST 1 (y)
                  6 LOAD_DEREF 0 (z)
    >>> p.co_cellvars
    ('z',)
 
 
Stack tracking on jumps::
 
    >>> c = Code()
    >>> else_ = Label()
    >>> end = Label()
    >>> c(99, else_.JUMP_IF_TRUE, Code.POP_TOP, end.JUMP_FORWARD)
    >>> c(99, else_.JUMP_IF_TRUE_OR_POP, end.JUMP_FORWARD)
    >>> c(else_, Code.POP_TOP, end)
    >>> dis(c.code())
      0 0 LOAD_CONST 1 (99)
                  3 JUMP_IF_TRUE 4 (to 10)
                  6 POP_TOP
                  7 JUMP_FORWARD 1 (to 11)
            >> 10 POP_TOP
    >>> dump(c.code())
                    LOAD_CONST 1 (99)
                    JUMP_IF_TRUE L1
                    POP_TOP
                    JUMP_FORWARD L2
            L1: POP_TOP
 
    >>> c.stack_size
    0
    >>> c.stack_history
    [0, 1, 1, 1, 1, 1, 1, 0, None, None, 1]
    >>> if sys.version>='2.7':
    ... print c.stack_history == [0, 1, 1, 1, 0, 0, 0, None, None, 1]
    ... else:
    ... print c.stack_history == [0, 1, 1, 1, 1, 1, 1, 0, None, None, 1]
    True
    
 
    >>> c = Code()
    >>> fwd = c.JUMP_FORWARD()

      ...
    AssertionError: Stack level mismatch: actual=1 expected=0
 
    >>> from peak.util.assembler import For
    >>> c = Code()
    >>> c(For((), Code.POP_TOP, Pass))
    >>> c.return_()
    >>> dump(c.code())
                    BUILD_TUPLE 0
                    GET_ITER
            L1: FOR_ITER L2
                    POP_TOP
                    JUMP_ABSOLUTE L1
            L2: LOAD_CONST 0 (None)
                    RETURN_VALUE
 
    >>> c.stack_history
    [0, 1, 1, 1, 1, 2, 2, 2, 1, None, None, 0, 1, 1, 1]
 
 
Yield value::
 
    >>> import sys
    >>> from peak.util.assembler import CO_GENERATOR
    >>> c = Code()
    >>> c.co_flags & CO_GENERATOR
    0
    >>> c(42, Code.YIELD_VALUE)
    >>> c.stack_size == int(sys.version>='2.5')
    True
    >>> (c.co_flags & CO_GENERATOR) == CO_GENERATOR
    True
 
    
    
Sequence operators and stack tracking:
 
 

      ...
    AssertionError: Stack underflow
 
    >>> c.LOAD_CONST(1)
    >>> c.LOAD_CONST(2) # simulate being a function
    >>> c.MAKE_CLOSURE(1, 0)
    >>> c = Code()
    >>> c.LOAD_CONST(1) # closure
    >>> if sys.version>='2.5': c.BUILD_TUPLE(1)
    >>> c.LOAD_CONST(2) # default
    >>> c.LOAD_CONST(3) # simulate being a function
    >>> c.MAKE_CLOSURE(1, 1)
    >>> c.stack_size
    1
 
    >>> c = Code()
    >>> c.LOAD_CONST(1)
    >>> c.LOAD_CONST(2)
    >>> if sys.version>='2.5': c.BUILD_TUPLE(2)
    >>> c.LOAD_CONST(3) # simulate being a function
    >>> c.MAKE_CLOSURE(1, 1)
    >>> c.MAKE_CLOSURE(0, 2)
    >>> c.stack_size
    1
 
 
 
Labels and backpatching forward references::
 
    >>> c = Code()
    >>> where = c.here()
    >>> c.LOAD_CONST(1)
    >>> c.JUMP_IF_TRUE(where)
    >>> c.JUMP_FORWARD(where)
    Traceback (most recent call last):
      ...
    AssertionError: Relative jumps can't go backwards

      0 0 LOAD_CONST 1 ({'x': 1})
                  3 RETURN_VALUE
 
Try/Except stack level tracking::
 
    >>> def class_or_type_of(expr):
    ... return Suite([expr, TryExcept(
    ... Suite([Getattr(Code.DUP_TOP, '__class__'), Code.ROT_TWO]),
    ... [(Const(AttributeError), Call(Const(type), (Code.ROT_TWO,)))]
    ... )])
 
    >>> def type_or_class(x): pass
    >>> c = Code.from_function(type_or_class)
    >>> c.return_(class_or_type_of(Local('x')))
    >>> dump(c.code())
                    LOAD_FAST 0 (x)
                    SETUP_EXCEPT L1
                    DUP_TOP
                    LOAD_ATTR 0 (__class__)
                    ROT_TWO
                    POP_BLOCK
                    JUMP_FORWARD L3
            L1: DUP_TOP
                    LOAD_CONST 1 (<...exceptions.AttributeError...>)
                    COMPARE_OP 10 (exception match)
                    JUMP_IF_FALSE L2
                    POP_TOP
                    POP_TOP
                    POP_TOP
                    POP_TOP
                    LOAD_CONST 2 (<type 'type'>)
                    ROT_TWO
                    CALL_FUNCTION 1
                    JUMP_FORWARD L3
            L2: POP_TOP
                    END_FINALLY
            L3: RETURN_VALUE
 
    >>> type_or_class.func_code = c.code()
    >>> type_or_class(23)
    <type 'int'>
    
 
 
 
 
Demo: "Computed Goto"/"Switch Statement"
========================================

 
    >>> from peak.util.assembler import LOAD_CONST, POP_BLOCK
 
    >>> def Pass(code=None):
    ... if code is None:
    ... return Pass
 
    >>> import sys
    >>> WHY_CONTINUE = {'2.3':5, '2.4':32, '2.5':32}[sys.version[:3]]
    >>> WHY_CONTINUE = {'2.3':5}.get(sys.version[:3], 32)
 
    >>> def Switch(expr, cases, default=Pass, code=None):
    ... if code is None:

    >>> f(3)
    27
 
    >>> dis(c.code())
      0 0 SETUP_LOOP 30 (to 33)
                  3 LOAD_CONST 1 (<...method get of dict...>)
                  6 LOAD_FAST 0 (x)
                  9 CALL_FUNCTION 1
                 12 JUMP_IF_FALSE 12 (to 27)
                 15 LOAD_CONST 2 (...)
                 18 END_FINALLY
                 19 LOAD_CONST 3 (42)
                 22 RETURN_VALUE
                 23 LOAD_CONST 4 ('foo')
                 26 RETURN_VALUE
            >> 27 POP_TOP
                 28 LOAD_CONST 5 (27)
                 31 RETURN_VALUE
                 32 POP_BLOCK
            >> 33 LOAD_CONST 0 (None)
                 36 RETURN_VALUE
    >>> dump(c.code())
                    SETUP_LOOP L2
                    LOAD_CONST 1 (<...method get of dict...>)
                    LOAD_FAST 0 (x)
                    CALL_FUNCTION 1
                    JUMP_IF_FALSE L1
                    LOAD_CONST 2 (...)
                    END_FINALLY
                    LOAD_CONST 3 (42)
                    RETURN_VALUE
                    LOAD_CONST 4 ('foo')
                    RETURN_VALUE
            L1: POP_TOP
                    LOAD_CONST 5 (27)
                    RETURN_VALUE
                    POP_BLOCK
            L2: LOAD_CONST 0 (None)
                    RETURN_VALUE
 
 
TODO

 
* Exhaustive tests of all opcodes' stack history effects
 
* YIELD_EXPR should set CO_GENERATOR; stack effects depend on Python version
* Test wide jumps and wide argument generation in general

PythonPowered
ShowText of this page
EditText of this page
FindPage by browsing, title search , text search or an index
Or try one of these actions: AttachFile, DeletePage, LikePages, LocalSiteMap, SpellCheck