Code Objects

To generate bytecode, you create a Code instance and perform operations on it. For example, here we create a Code object representing lines 15 and 16 of some input source:

>>> from peak.util.assembler import Code
>>> c = Code()
>>> c.set_lineno(15)   # set the current line number (optional)
>>> c.LOAD_CONST(42)

>>> c.set_lineno(16)   # set it as many times as you like
>>> c.RETURN_VALUE()

You'll notice that most Code methods are named for a CPython bytecode operation, but there also some other methods like .set_lineno() to let you set the current line number. There's also a .code() method that returns a Python code object, representing the current state of the Code you've generated:

>>> from dis import dis
>>> dis(c.code())
  15          0 LOAD_CONST               1 (42)
  16          3 RETURN_VALUE

As you can see, Code instances automatically generate a line number table that maps each set_lineno() to the corresponding position in the bytecode.

And of course, the resulting code objects can be run with eval() or exec, or used with new.function to create a function:

>>> eval(c.code())
42

>>> exec c.code()   # exec discards the return value, so no output here

>>> import new
>>> f = new.function(c.code(), globals())
>>> f()
42

Opcodes and Arguments

Code objects have methods for all of CPython's symbolic opcodes. Generally speaking, each method accepts either zero or one argument, depending on whether the opcode accepts an argument.

Python bytecode always encodes opcode arguments as 16 or 32-bit integers, but sometimes these numbers are actually offsets into a sequence of names or constants. Code objects take care of maintaining these sequences for you, allowing you to just pass in a name or value directly, instead of needing to keep track of what numbers map to what names or values.

The name or value you pass in to such methods will be looked up in the appropriate table (see Code Attributes below for a list), and if not found, it will be added:

>>> c = Code()
>>> c.co_consts, c.co_varnames, c.co_names
([None], [], [])

>>> c.LOAD_CONST(42)
>>> c.LOAD_FAST('x')
>>> c.LOAD_GLOBAL('y')
>>> c.LOAD_NAME('z')

>>> c.co_consts, c.co_varnames, c.co_names
([None, 42], ['x'], ['y', 'z'])

The one exception to this automatic addition feature is that opcodes referring to "free" or "cell" variables will not automatically add new names, because the names need to be defined first:

>>> c.LOAD_DEREF('q')
Traceback (most recent call last):
  ...
NameError: ('Undefined free or cell var', 'q')

In general, opcode methods take the same arguments as their Python bytecode equivalent. But there are a few special cases.

>>> c = Code()
>>> c.LOAD_CONST(type)
>>> c.LOAD_CONST(27)
>>> c.CALL_FUNCTION(1)      # 1 positional, no keywords
>>> c.RETURN_VALUE()

>>> eval(c.code())          # computes type(27)
<type 'int'>

>>> c = Code()
>>> c.LOAD_CONST(dict)
>>> c.LOAD_CONST('x')
>>> c.LOAD_CONST(42)
>>> c.CALL_FUNCTION(0,1)    # no positional, 1 keyword
>>> c.RETURN_VALUE()

>>> eval(c.code())          # computes dict(x=42)
{'x': 42}

>>> c = Code()
>>> where = c.here()         # get a location at the start of the code

>>> c.LOAD_CONST(42)
>>> c.JUMP_ABSOLUTE(where)   # now jump back to it

>>> dis(c.code())
  0     >>    0 LOAD_CONST               1 (42)
              3 JUMP_ABSOLUTE            0

>>> c = Code()
>>> forward = c.JUMP_ABSOLUTE() # create a jump and a forward reference

>>> c.LOAD_CONST(42)            # this is what we want to skip over

>>> forward()   # calling the reference changes the jump to point here
>>> c.LOAD_CONST(23)
>>> c.RETURN_VALUE()

>>> dis(c.code())
  0           0 JUMP_ABSOLUTE            6
              3 LOAD_CONST               1 (42)
        >>    6 LOAD_CONST               2 (23)
              9 RETURN_VALUE

>>> eval(c.code())
23

>>> def x(a,b):     # a simple closure example
...     def y():
...         return a+b
...     return y

>>> c = Code()
>>> c.co_cellvars = ('a','b')

>>> c.LOAD_CLOSURE('a')
>>> c.LOAD_CLOSURE('b')
>>> c.LOAD_CONST(None)  # in real code, this'd be a Python code constant
>>> c.MAKE_CLOSURE(0,2) # no defaults, 2 free vars in the new function

High-Level Code Generation

Typical real-life code generation use cases call for transforming tree-like data structures into bytecode, rather than linearly outputting instructions. Code objects provide for this using a simple but high-level transformation API.

Code objects may be called, passing in one or more arguments. Each argument will have bytecode generated for it, according to its type:

>>> c = Code()
>>> c(1, 2L, 3.0, 4j+5, "6", u"7", False, None, c.code())
>>> dis(c.code())
  0           0 LOAD_CONST               1 (1)
              3 LOAD_CONST               2 (2L)
              6 LOAD_CONST               3 (3.0)
              9 LOAD_CONST               4 ((5+4j))
             12 LOAD_CONST               5 ('6')
             15 LOAD_CONST               6 (u'7')
             18 LOAD_CONST               7 (False)
             21 LOAD_CONST               0 (None)
             24 LOAD_CONST               8 (<code object <lambda> at ...>)

>>> c = Code()
>>> c({1:(2,"3"), 4:[5,6]})
>>> dis(c.code())
  0           0 BUILD_MAP                0
              3 DUP_TOP
              4 LOAD_CONST               1 (1)
              7 LOAD_CONST               2 (2)
             10 LOAD_CONST               3 ('3')
             13 BUILD_TUPLE              2
             16 ROT_THREE
             17 STORE_SUBSCR
             18 DUP_TOP
             19 LOAD_CONST               4 (4)
             22 LOAD_CONST               5 (5)
             25 LOAD_CONST               6 (6)
             28 BUILD_LIST               2
             31 ROT_THREE
             32 STORE_SUBSCR

>>> from peak.util.assembler import Const

>>> c = Code()
>>> c( Const( (1,2,3) ) )
>>> dis(c.code())
  0           0 LOAD_CONST               1 ((1, 2, 3))

>>> from peak.util.assembler import Global, Local

>>> c = Code()
>>> c( Local('x'), Global('y') )
>>> dis(c.code())
  0           0 LOAD_FAST                0 (x)
              3 LOAD_GLOBAL              0 (y)

>>> c = Code()
>>> c( {Local('a'): (Local('b'), Local('c'))} )
>>> dis(c.code())
  0           0 BUILD_MAP                0
              3 DUP_TOP
              4 LOAD_FAST                0 (a)
              7 LOAD_FAST                1 (b)
             10 LOAD_FAST                2 (c)
             13 BUILD_TUPLE              2
             16 ROT_THREE
             17 STORE_SUBSCR

>>> from peak.util.assembler import CO_OPTIMIZED
>>> c = Code()
>>> c.co_flags &= ~CO_OPTIMIZED
>>> c.co_cellvars = ('y',)
>>> c.co_freevars = ('z',)
>>> c( Local('x'), Local('y'), Local('z') )
>>> dis(c.code())
  0           0 LOAD_NAME                0 (x)
              3 LOAD_DEREF               0 (y)
              6 LOAD_DEREF               1 (z)

>>> c = Code()
>>> c( Call(Global('type'), [Const(27)]) )

>>> dis(c.code())   # type(27)
  0           0 LOAD_GLOBAL              0 (type)
              3 LOAD_CONST               1 (27)
              6 CALL_FUNCTION            1

>>> c = Code()
>>> c(Call(Global('dict'), (), [('x', 42)]))

>>> dis(c.code())   # dict(x=42)
  0           0 LOAD_GLOBAL              0 (dict)
              3 LOAD_CONST               1 ('x')
              6 LOAD_CONST               2 (42)
              9 CALL_FUNCTION            256

>>> c = Code()
>>> c(Call(Global('foo'), (), (), Local('args'), Local('kw')))

>>> dis(c.code())   # foo(*args, **kw)
  0           0 LOAD_GLOBAL              0 (foo)
              3 LOAD_FAST                0 (args)
              6 LOAD_FAST                1 (kw)
              9 CALL_FUNCTION_VAR_KW     0

>>> from peak.util.assembler import Return

>>> c = Code()
>>> c( Return(1) )
>>> dis(c.code())
  0           0 LOAD_CONST               1 (1)
              3 RETURN_VALUE

>>> c = Code()
>>> c.return_((1,2))
>>> dis(c.code())
  0           0 LOAD_CONST               1 (1)
              3 LOAD_CONST               2 (2)
              6 BUILD_TUPLE              2
              9 RETURN_VALUE

>>> c = Code()
>>> c.return_()
>>> c( Return() )
>>> dis(c.code())
  0           0 LOAD_CONST               0 (None)
              3 RETURN_VALUE
              4 LOAD_CONST               0 (None)
              7 RETURN_VALUE

>>> c = Code()
>>> forward = c.JUMP_FORWARD()
>>> c( 1, 2, forward, Return(3) )
>>> dis(c.code())
  0           0 JUMP_FORWARD             6 (to 9)
              3 LOAD_CONST               1 (1)
              6 LOAD_CONST               2 (2)
         >>   9 LOAD_CONST               3 (3)
             12 RETURN_VALUE

>>> from peak.util.assembler import Label
>>> c = Code()
>>> skip = Label()

>>> c(skip.JUMP_FORWARD, 1, 2, skip, Return(3))
>>> dis(c.code())
  0           0 JUMP_FORWARD             6 (to 9)
              3 LOAD_CONST               1 (1)
              6 LOAD_CONST               2 (2)
         >>   9 LOAD_CONST               3 (3)
             12 RETURN_VALUE

>>> c(skip)
Traceback (most recent call last):
  ...
AssertionError: Label previously defined

Constant Detection and Folding

The const_value() function can be used to check if an expression tree has a constant value, and to obtain that value. Simple constants are returned as-is:

>>> from peak.util.assembler import const_value

>>> simple_values = [1, 2L, 3.0, 4j+5, "6", u"7", False, None, c.code()]

>>> map(const_value, simple_values)
[1, 2L, 3.0, (5+4j), '6', u'7', False, None, <code object <lambda> ...>]

Values wrapped in a Const() are also returned as-is:

>>> map(const_value, map(Const, simple_values))
[1, 2L, 3.0, (5+4j), '6', u'7', False, None, <code object <lambda> ...>]

But no other node types produce constant values; instead, NotAConstant is raised:

>>> const_value(Local('x'))
Traceback (most recent call last):
  ...
NotAConstant: <bound method str.Local of 'x'>

Tuples of constants are recursively replaced by constant tuples:

>>> const_value( (1,2) )
(1, 2)

>>> const_value( (1, (2, Const(3))) )
(1, (2, 3))

But any non-constant values anywhere in the structure cause an error:

>>> const_value( (1,Global('y')) )
Traceback (most recent call last):
  ...
NotAConstant: <bound method str.Global of 'y'>

As do any types not previously described here:

>>> const_value([1,2])
Traceback (most recent call last):
  ...
NotAConstant: [1, 2]

Unless of course they're wrapped with Const:

>>> const_value(Const([1,2]))
[1, 2]

The Call wrapper can also do simple constant folding, if all of its input parameters are constants. (Actually, the args and kwargs arguments must be sequences of constants and 2-tuples of constants, respectively.)

If a Call can thus compute its value in advance, it does so, returning a Const node instead of a Call node:

>>> Call( Const(type), [1] )
<bound method type.Const of <type 'int'>>

Thus, you can also take the const_value() of such calls:

>>> const_value( Call( Const(dict), [], [('x',27)] ) )
{'x': 27}

Which means that constant folding can propagate up an AST if the result is passed in to another Call:

>>> Call(Const(type), [Call( Const(dict), [], [('x',27)] )])
<bound method type.Const of <type 'dict'>>

Notice that this folding takes place eagerly, during AST construction. If you want to implement delayed folding after constant propagation or variable substitution, you'll need to recreate the tree, or use your own custom AST types. (See Custom Code Generation, below.)

Note that you can disable folding using the fold=False keyword argument to Call, if you want to ensure that even compile-time constants are computed at runtime. Compare:

>>> c = Code()
>>> c( Call(Const(type), [1]) )
>>> dis(c.code())
  0           0 LOAD_CONST               1 (<type 'int'>)

>>> c = Code()
>>> c( Call(Const(type), [1], fold=False) )
>>> dis(c.code())
  0           0 LOAD_CONST               1 (<type 'type'>)
              3 LOAD_CONST               2 (1)
              6 CALL_FUNCTION            1

Folding is also automatically disabled for calls with no arguments of any kind (such as globals() or locals()), whose values are much more likely to change dynamically at runtime:

>>> c = Code()
>>> c( Call(Const(locals)) )
>>> dis(c.code())
  0           0 LOAD_CONST               1 (<built-in function locals>)
              3 CALL_FUNCTION            0

Note, however, that folding is disabled for any zero-argument call, regardless of the thing being called. It is not specific to locals() and globals(), in other words.

Custom Code Generation

Code generation is extensible: you can use any callable as a code-generation target. It will be called with exactly one argument: the code object. It can then perform whatever operations are desired.

In the most trivial case, you can use any unbound Code method as a code generation target, e.g.:

>>> c = Code()
>>> c.LOAD_GLOBAL('foo')
>>> c(Call(Code.DUP_TOP, ()))
>>> dis(c.code())
  0           0 LOAD_GLOBAL              0 (foo)
              3 DUP_TOP
              4 CALL_FUNCTION            0

As you can see, the Code.DUP_TOP() is called on the code instance, causing a DUP_TOP opcode to be output. This is sometimes a handy trick for accessing values that are already on the stack. More commonly, however, you'll want to implement more sophisticated callables, perhaps something like:

>>> from peak.util.assembler import ast_curry

>>> def TryFinally(block1, block2, code=None):
...     if code is None:
...         return ast_curry(TryFinally, block1, block2)
...     code(
...         Code.SETUP_FINALLY,
...             block1,
...         Code.POP_BLOCK,
...             block2,
...         Code.END_FINALLY
...     )

>>> def ExprStmt(value, code=None):
...     if code is None:
...         return ast_curry(ExprStmt, value)
...     code( value, Code.POP_TOP )

>>> c = Code()
>>> c( TryFinally(ExprStmt(1), ExprStmt(2)) )
>>> dis(c.code())
  0           0 SETUP_FINALLY            8 (to 11)
              3 LOAD_CONST               1 (1)
              6 POP_TOP
              7 POP_BLOCK
              8 LOAD_CONST               0 (None)
        >>   11 LOAD_CONST               2 (2)
             14 POP_TOP
             15 END_FINALLY

The ast_curry() utility function returns an instancemethod chain that binds the given arguments to the given function, creating a hashable and comparable data structure -- a trivial sort of "AST node". Just follow the code pattern above, using a code=None final argument, and returning a curried version of the function if code is None. Otherwise, your function should simply do whatever is needed to "generate" the arguments.

(This is exactly the same pattern that peak.util.assembler uses internally to implement Const, Call, Local, and other wrapper functions.)

The ast_curry() utility function isn't quite perfect; due to a quirk of the instancemethod type, it can't save arguments whose value is None: if you pass a None argument to ast_curry(), it will be replaced with a special nil object that tests as false, and generates a None constant when code is generated for it. If your function accepts any arguments that might have a value of None, you must correctly handle the cases where you receive a value of nil (found in peak.util.assembler) instead of None.

However, if you can use ast_curry() to generate your AST nodes, you will have objects that are hashable and comparable by default, as long as none of your child nodes are unhashable or incomparable. This can be useful for algorithms that require comparing AST subtrees, such as common subexpression elimination.

If you want to incorporate constant-folding into your AST nodes, you can do so by checking for constant values and folding them at either construction or code generation time. For example, this And node type folds constants during code generation, by not generating unnecessary branches when it can prove which way a branch will go:

>>> from peak.util.assembler import NotAConstant

>>> def And(values, code=None):
...     if code is None:
...         return ast_curry(And, tuple(values))
...     end = Label()
...     for value in values[:-1]:
...         try:
...             if const_value(value):
...                 continue    # true constants can be skipped
...             else:           # and false ones end the chain right away
...                 return code(value, end)
...         except NotAConstant:    # but non-constants require code
...             code(value, end.JUMP_IF_FALSE, Code.POP_TOP)
...     code(values[-1], end)

>>> c = Code()
>>> c.return_( And([1, 2]) )
>>> dis(c.code())
  0           0 LOAD_CONST               1 (2)
              3 RETURN_VALUE

>>> c = Code()
>>> c.return_( And([1, 2, Local('x')]) )
>>> dis(c.code())
  0           0 LOAD_FAST                0 (x)
              3 RETURN_VALUE

>>> c = Code()
>>> c.return_( And([Local('x'), False, 27]) )
>>> dis(c.code())
  0           0 LOAD_FAST                0 (x)
              3 JUMP_IF_FALSE            4 (to 10)
              6 POP_TOP
              7 LOAD_CONST               1 (False)
        >>   10 RETURN_VALUE

Setting the Code's Calling Signature

The simplest way to set up the calling signature for a Code instance is to clone an existing function or code object's signature, using the Code.from_function() or Code.from_code() classmethods. These methods create a new Code instance whose calling signature (number and names of arguments) matches that of the original function or code objects:

>>> def f1(a,b,*c,**d):
...     pass

>>> c = Code.from_function(f1)
>>> f2 = new.function(c.code(), globals())

>>> import inspect

>>> inspect.getargspec(f1)
(['a', 'b'], 'c', 'd', None)

>>> inspect.getargspec(f2)
(['a', 'b'], 'c', 'd', None)

Note that these constructors do not copy any actual code from the code or function objects. They simply copy the signature, and, if you set the copy_lineno keyword argument to a true value, they will also set the created code object's co_firstlineno to match that of the original code or function object:

>>> c1 = Code.from_function(f1, copy_lineno=True)
>>> c1.co_firstlineno
1

If you create a Code instance from a function that has nested positional arguments, the returned code object will include a prologue to unpack the arguments properly:

>>> def f3(a, (b,c), (d,(e,f))):
...     pass

>>> f4 = new.function(Code.from_function(f3).code(), globals())
>>> dis(f4)
  0           0 LOAD_FAST                1 (.1)
              3 UNPACK_SEQUENCE          2
              6 STORE_FAST               3 (b)
              9 STORE_FAST               4 (c)
             12 LOAD_FAST                2 (.2)
             15 UNPACK_SEQUENCE          2
             18 STORE_FAST               5 (d)
             21 UNPACK_SEQUENCE          2
             24 STORE_FAST               6 (e)
             27 STORE_FAST               7 (f)

This is roughly the same code that Python would generate to do the same unpacking process, and is designed so that the inspect module will recognize it as an argument unpacking prologue:

>>> inspect.getargspec(f3)
(['a', ['b', 'c'], ['d', ['e', 'f']]], None, None, None)

>>> inspect.getargspec(f4)
(['a', ['b', 'c'], ['d', ['e', 'f']]], None, None, None)

Code Attributes

Code instances have a variety of attributes corresponding to either the attributes of the Python code objects they generate, or to the current state of code generation.

For example, the co_argcount and co_varnames attributes correspond to those used in creating the code for a Python function. If you want your code to be a function, you can set them as follows:

>>> c = Code()
>>> c.co_argcount = 3
>>> c.co_varnames = ['a','b','c']

>>> c.LOAD_CONST(42)
>>> c.RETURN_VALUE()

>>> f = new.function(c.code(), globals())
>>> f(1,2,3)
42

>>> import inspect
>>> inspect.getargspec(f)
(['a', 'b', 'c'], None, None, None)

Although Python code objects want co_varnames to be a tuple, Code instances use a list, so that names can be added during code generation. The .code() method automatically creates tuples where necessary.

Here are all of the Code attributes you may want to read or write:

co_filename

A string representing the source filename for this code. If it's an actual filename, then tracebacks that pass through the generated code will display lines from the file. The default value is '<generated code>'.

co_name

The name of the function, class, or other block that this code represents. The default value is '<lambda>'.

co_argcount

Number of positional arguments a function accepts; defaults to 0

co_varnames

A list of strings naming the code's local variables, beginning with its positional argument names, followed by its * and ** argument names, if applicable, followed by any other local variable names. These names are used by the LOAD_FAST and STORE_FAST opcodes, and invoking the .LOAD_FAST(name) and .STORE_FAST(name) methods of a code object will automatically add the given name to this list, if it's not already present.

co_flags

The flags for the Python code object. This defaults to CO_OPTIMIZED | CO_NEWLOCALS, which is the correct value for a function using "fast" locals. This value is automatically or-ed with CO_NOFREE when generating a code object, if the co_cellvars and co_freevars attributes are empty. And if you use the LOAD_NAME(), STORE_NAME(), or DELETE_NAME() methods, the CO_OPTIMIZED bit is automatically reset, since these opcodes can only be used when the code is running with a real (i.e. not virtualized) locals() dictionary.

If you need to change any other flag bits besides the above, you'll need to set or clear them manually. For your convenience, the peak.util.assembler module exports all the CO_ constants used by Python. For example, you can use CO_VARARGS and CO_VARKEYWORDS to indicate whether a function accepts * or ** arguments, as long as you extend the co_varnames list accordingly. (Assuming you don't have an existing function or code object with the desired signature, in which case you could just use the from_function() or from_code() classmethods instead of messing with these low-level attributes and flags.)

stack_size

The predicted height of the runtime value stack, as of the current opcode. Its value is automatically updated by most opcodes, but you may want to save and restore it for things like try/finally blocks. If you increase the value of this attribute, you should also update the co_stacksize attribute if it is less than the new stack_size.

co_freevars

A tuple of strings naming a function's "cell" variables. Defaults to an empty tuple. A function's free variables are the variables it "inherits" from its surrounding scope. If you're going to use this, you should set it only once, before generating any code that references any free or cell variables.

co_cellvars

A tuple of strings naming a function's "cell" variables. Defaults to an empty tuple. A function's cell variables are the variables that are "inherited" by one or more of its nested functions. If you're going to use this, you should set it only once, before generating any code that references any free or cell variables.

These other attributes are automatically generated and maintained, so you'll probably never have a reason to change them:

co_consts

A list of constants used by the code; the first (zeroth?) constant is always None. Normally, this is automatically maintained; the .LOAD_CONST(value) method checks to see if the constant is already present in this list, and adds it if it is not there.

co_names

A list of non-optimized or global variable names. It's automatically updated whenever you invoke a method to generate an opcode that uses such names.

co_code

A byte array containing the generated code. Don't mess with this.

co_firstlineno

The first line number of the generated code. It automatically gets set if you call .set_lineno() before generating any code; otherwise it defaults to zero.

co_lnotab

A byte array containing a generated line number table. It's automatically generated, so don't mess with it.

co_stacksize

The maximum amount of stack space the code will require to run. This value is usually updated automatically as you generate code. However, if you manually set a new stack_size that is larger than the current co_stacksize, you should increase the co_stacksize to match, so that co_stacksize is always the largest stack size the code will generate at runtime.

Blocks, Loops, and Exception Handling

The Python SETUP_FINALLY, SETUP_EXCEPT, and SETUP_LOOP opcodes all create "blocks" that go on the frame's "block stack" at runtime. Each of these opcodes must be matched with exactly one POP_BLOCK opcode -- no more, and no less. Code objects enforce this using an internal block stack that matches each setup with its corresponding POP_BLOCK. Trying to pop a nonexistent block, or trying to generate code when unclosed blocks exist is an error:

>>> c = Code()
>>> c.POP_BLOCK()
Traceback (most recent call last):
  ...
AssertionError: Not currently in a block

>>> c.SETUP_FINALLY()
>>> c.code()
Traceback (most recent call last):
  ...
AssertionError: 1 unclosed block(s)

>>> c.POP_BLOCK()
>>> c.code()
<code object <lambda> ...>

Code objects also check that the stack level as of a POP_BLOCK is the same as it was when the block was set up:

>>> c = Code()
>>> c.SETUP_LOOP()
>>> c.LOAD_CONST(23)
>>> c.POP_BLOCK()
Traceback (most recent call last):
  ...
AssertionError: Stack level mismatch: actual=1 expected=0

>>> c = Code()
>>> c.SETUP_FINALLY()
>>> c.stack_size, c.co_stacksize
(0, 0)
>>> c.POP_BLOCK()
>>> c.END_FINALLY()
>>> c.stack_size, c.co_stacksize
(0, 3)

>>> c = Code()
>>> c(1,2,3,4, *[Code.POP_TOP]*4)   # push 4 things, then pop 'em
>>> c.SETUP_FINALLY()
>>> c.POP_BLOCK()
>>> c.END_FINALLY()
>>> c.stack_size, c.co_stacksize
(0, 4)

>>> c = Code()
>>> c.SETUP_LOOP()
>>> break_to = c.POP_BLOCK()
>>> c.stack_size, c.co_stacksize
(0, 0)

>>> c = Code()
>>> c.SETUP_EXCEPT()
>>> else_ = c.POP_BLOCK()
>>> c.stack_size, c.co_stacksize
(3, 3)

>>> c.POP_TOP()     # get rid of exception info
>>> c.POP_TOP()
>>> c.POP_TOP()
>>> else_()
>>> c.return_()
>>> dis(c.code())
  0           0 SETUP_EXCEPT             4 (to 7)
              3 POP_BLOCK
              4 JUMP_FORWARD             3 (to 10)
        >>    7 POP_TOP
              8 POP_TOP
              9 POP_TOP
        >>   10 LOAD_CONST               0 (None)
             13 RETURN_VALUE

>>> c = Code()
>>> done = Label()
>>> c(
...     done.SETUP_EXCEPT,
...     done.POP_BLOCK,
...         Code.POP_TOP, Code.POP_TOP, Code.POP_TOP,
...     done,
...     Return()
... )

>>> dis(c.code())
  0           0 SETUP_EXCEPT             4 (to 7)
              3 POP_BLOCK
              4 JUMP_FORWARD             3 (to 10)
        >>    7 POP_TOP
              8 POP_TOP
              9 POP_TOP
        >>   10 LOAD_CONST               0 (None)
             13 RETURN_VALUE

>>> c = Code()
>>> c.SETUP_FINALLY()
>>> # "try" suite goes here
>>> c.POP_BLOCK()
>>> # "finally" suite goes here
>>> c.END_FINALLY()

>>> dis(c.code())
  0           0 SETUP_FINALLY            4 (to 7)
              3 POP_BLOCK
              4 LOAD_CONST               0 (None)
        >>    7 END_FINALLY

>>> c = Code()
>>> block = Label()
>>> loop = Label()
>>> else_ = Label()
>>> c(
...     block.SETUP_LOOP,
...         5,      # initial setup - this could be a GET_ITER instead
...     loop,
...         else_.JUMP_IF_FALSE,        # while x:
...         1, Code.BINARY_SUBTRACT,    #     x -= 1
...         loop.CONTINUE_LOOP,
...     else_,                          # else:
...         Code.POP_TOP,
...     block.POP_BLOCK,
...         Return(42),                 #     return 42
...     block,
...     Return()
... )

>>> dis(c.code())
  0           0 SETUP_LOOP              19 (to 22)
              3 LOAD_CONST               1 (5)
        >>    6 JUMP_IF_FALSE            7 (to 16)
              9 LOAD_CONST               2 (1)
             12 BINARY_SUBTRACT
             13 JUMP_ABSOLUTE            6
        >>   16 POP_TOP
             17 POP_BLOCK
             18 LOAD_CONST               3 (42)
             21 RETURN_VALUE
        >>   22 LOAD_CONST               0 (None)
             25 RETURN_VALUE

>>> eval(c.code())
42

>>> c = Code()
>>> c.BREAK_LOOP()
Traceback (most recent call last):
  ...
AssertionError: Not inside a loop

>>> c.CONTINUE_LOOP(c.here())
Traceback (most recent call last):
  ...
AssertionError: Not inside a loop

>>> c.SETUP_LOOP()
>>> c.CONTINUE_LOOP(c.here())
>>> c.BREAK_LOOP()
>>> c.POP_BLOCK()()
>>> dis(c.code())
  0           0 SETUP_LOOP               5 (to 8)
        >>    3 JUMP_ABSOLUTE            3
              6 BREAK_LOOP
              7 POP_BLOCK

>>> c = Code()
>>> c.SETUP_LOOP()
>>> loop = c.here()
>>> c.SETUP_FINALLY()
>>> c.CONTINUE_LOOP(loop)
>>> c.POP_BLOCK()
>>> c.END_FINALLY()
>>> c.POP_BLOCK()()
>>> dis(c.code())
  0           0 SETUP_LOOP              12 (to 15)
        >>    3 SETUP_FINALLY            7 (to 13)
              6 CONTINUE_LOOP            3
              9 POP_BLOCK
             10 LOAD_CONST               0 (None)
        >>   13 END_FINALLY
             14 POP_BLOCK

Demo: "Computed Goto"/"Switch Statement"

Finally, to give an example of a creative way to abuse Python bytecode, here is an implementation of a simple "switch/case/else" structure:

>>> from peak.util.assembler import LOAD_CONST, POP_BLOCK

>>> def Pass(code=None):
...     if code is None:
...         return Pass

>>> def NewConst(value, code=None):
...     if code is None:
...         return ast_curry(NewConst, value)
...     code.emit_arg(LOAD_CONST, len(code.co_consts))
...     code.co_consts.append(value)
...     code.stackchange((0,1))

>>> import sys
>>> WHY_CONTINUE = {'2.3':5, '2.4':32, '2.5':32}[sys.version[:3]]

>>> def Switch(expr, cases, default=Pass, code=None):
...     if code is None:
...         return ast_curry(Switch, expr, tuple(cases), default)
...
...     d = {}
...     else_block  = Label()
...     cleanup     = Label()
...     end_switch  = Label()
...
...     code(
...         end_switch.SETUP_LOOP,
...             Call(NewConst(d.get), [expr]),
...         else_block.JUMP_IF_FALSE,
...             WHY_CONTINUE, Code.END_FINALLY
...     )
...
...     cursize = code.stack_size
...     for key, value in cases:
...         d[const_value(key)] = code.here()
...         code(value, cleanup.JUMP_FORWARD)
...
...     code(
...         else_block,
...             Code.POP_TOP, default,
...         cleanup,
...             Code.POP_BLOCK,
...         end_switch
...     )

>>> c = Code()
>>> c.co_argcount=1
>>> c(Switch(Local('x'), [(1,Return(42)),(2,Return("foo"))], Return(27)))
>>> c.return_()

>>> f = new.function(c.code(), globals())
>>> f(1)
42
>>> f(2)
'foo'
>>> f(3)
27

>>> dis(c.code())
  0           0 SETUP_LOOP              36 (to 39)
              3 LOAD_CONST               1 (<...method get of dict...>)
              6 LOAD_FAST                0 (x)
              9 CALL_FUNCTION            1
             12 JUMP_IF_FALSE           18 (to 33)
             15 LOAD_CONST               2 (...)
             18 END_FINALLY
             19 LOAD_CONST               3 (42)
             22 RETURN_VALUE
             23 JUMP_FORWARD            12 (to 38)
             26 LOAD_CONST               4 ('foo')
             29 RETURN_VALUE
             30 JUMP_FORWARD             5 (to 38)
        >>   33 POP_TOP
             34 LOAD_CONST               5 (27)
             37 RETURN_VALUE
        >>   38 POP_BLOCK
        >>   39 LOAD_CONST               0 (None)
             42 RETURN_VALUE

TODO

• AST introspection

  • ast_type(node): called function, Const, or node.__class__ * tuples are Const if their contents are; no other types are Const
  • ast_children(node): tuple of argument values for curried types, const value, or empty tuple. If node is a tuple, the value must be flattened.
  • is_const(node): ast_type(node) is Const

• Inline builtins (getattr, operator.getitem, etc.) to opcodes

  • Getattr/Op/Unary("symbol", arg1 [, arg2]) node types -> Call() if folding
  • Call() translates functions back to Ops if inlining

• Pretty printing and short-naming of ASTs

• Test NAME vs. FAST operators flag checks/sets

• Test code flags generation/cloning