[PEAK] API for generic functions?

Phillip J. Eby pje at telecommunity.com
Mon Nov 8 21:44:16 EST 2004

Now that I've implemented most of the generic function API from my last 
post (dispatch.SimpleGeneric, dispatch.when, dispatch.as, etc.), I'm 
wondering if maybe it needs to be tweaked a little bit.

Currently, you can use 'dispatch.when()' to either create a new generic 
function or update an existing one.  However, if you wish to create a 
single-dispatch generic function, you must explicitly create it first, 
using SimpleGeneric.

I'm wondering if maybe this should be changed, such that you must always 
explicitly define a generic function before adding methods.  There are 
certain advantages to that, especially if the definition is done with a 
decorator on a function.  For example, you can cause source-parsing 
documentation tools (like HappyDoc) to emit correct documentation for the 
generic function that way.  And, you can specify arguments to the decorator 
to configure the generic function.  For example, you could select the 
method-combination strategy to be used:

     def priority(ctx, job):
         """Determine priority of 'job' by summing applicable scoring rules"""

     def rush_priority(ctx,job):
         """Add 20 to the priority for a rush job"""
         return 20

     def we_like_fred(ctx,job):
         """Add 10 more for people we like"""
         return 10

Now, there are three functions in the above code, and 'priority' would 
invoke the other two as applicable, adding the returned scores 
together.  Or, we could change 'reduce' to 'max' so the function returns 
the highest priority assigned by any rule.  Meanwhile, the rules 
'rush_priority' and 'we_like_fred' are available to be reused in other 
generic functions, or just as standalone functions.

Meanwhile, single-dispatch generic functions might be defined thus:

     def getStreamSource(source,ctx):
         """Return a 'naming.IStreamSource' for 'source' and 'ctx'"""

     def already_a_source(source,ctx):
         return source

     def lookup_stream_URL(source,ctx):
         # etc...

The idea here is that you would name the argument that dispatch will occur 
on; a feature we don't actually have yet, but which would be nice to have, 
since it would allow single-dispatch generics to be used as methods, not 
just multi-dispatch ones.  (Of course, neither kind of generic function can 
be used as a method yet, anyway.)

Method definition with the approach above could get a little awkward, though:

     class NormalRules:

         def priority(self, job):
             """Determine priority of 'job' by summing applicable scoring 

         def rush_priority(self,job):
             """Add 20 to the priority for a rush job"""
             return 20

     class Favoritism(NormalRules):

         priority = NormalRules.priority

         def we_like_fred(self,job):
            """Add 10 for people we like"""
            return 10

I guess it's not too bad.  However, what about a use case where we're not 
in the class, but want to add another case to Favoritism directly?

     def we_really_like_bob(self,job):
         return 100

Looks okay, but it won't work right without a lot of behind-the-scenes 
implementation muck, that will also slow down access to generic functions 
used as methods, unless I write that part with Pyrex.

At this point, I suppose we could get rid of 'dispatch.when()' 
altogether.  That seems to work alright, so let's start looking at the 
advanced API for defining fancy method combinations.

CLOS "short-form" method combinations could look something like this:

     def add():
         """Summarizes results by adding them together"""

In addition to taking an operator and a function, there'd be two keyword 
args to 'method_combination', unary_identity=True and 
order=dispatch.MOST_SPECIFIC_FIRST.  (The default 
values.)  'unary_identity' means, if there's only one applicable method, 
just return its value, rather than iterating over applicable 
items.  'order' should let you specify MOST_SPECIFIC_FIRST, 

For more advanced method combinations (like before/after/around, it would 
also be necessary to specify "method groups" that are used to select 
specially-qualified methods.  Method groups would be defined by something like:

     around = dispatch.MethodGroup("around")
     primary = dispatch.MethodGroup("primary", qualifiers=(), required=True)
     after = dispatch.MethodGroup("after", order=dispatch.LEAST_SPECIFIC_FIRST)
     before = dispatch.MethodGroup("before")

And then they would be used something like:

     def standard_combination(primary,before,after,around):
         """CLOS-like method combination"""

         run_primary = dispatch.method_chain(primary)

         if before or after:
             befores     = dispatch.method_list(before)
             afters      = dispatch.method_list(after)

             def inner(*__args,**__kw):
                 if before:
                     for b in befores(*__args,**__kw): pass
                 result = run_primary(*__args,**__kw)
                 if after:
                     for a in afters(*__args,**__kw): pass
                 return result
             inner = run_primary

         return dispatch.method_chain(list(around)+[inner])

Here, we define a method combination that runs any 'around' methods 
"around" the 'inner' method, which runs the 'before' and 'after' methods 
around the chained 'primary' methods.  (In other words, the standard CLOS 
method combination approach.)

Qualifiers defined by method combinations (e.g. 'before', 'after', and 
'around') should effectively add methods to the generic function instance, 
so that 'someGeneric.before("condition")' then works.

CLOS allows multiple qualifiers per method, but I'm not sure if we need 
that.  It could always be added later, if we build the innards right.  CLOS 
also expects methods to be qualified when using "short-form" method 
combinarions, such that if we were doing:

     def priority(ctx, job):
         """Determine priority of 'job' by summing applicable scoring rules"""

it would then expect individual methods to be defined with the equivalent of:

     def rush_priority(ctx,job):
         """Add 20 to the priority for a rush job"""
         return 20

But I don't think I like this.  CLOS presumably does this for redundancy, 
so that you don't forget that the value you return is going to be added (or 
and'ed or or'ed or whatever).  There's something to be said for that.  I 
guess maybe we just need to be able to specify the names nicely, like:

     def rush_priority(ctx,job):
         """Add 20 to the priority for a rush job"""
         return 20

That looks better to me.  Maybe you'd define it with:

     [dispatch.method_combination(operator.add, method="add_when")]
     def add():
         """Summarizes results by adding them together"""

for the short-form qualifier.  A long form qualifier would look something like:

     around = dispatch.MethodGroup("around")
     primary = dispatch.MethodGroup("primary", qualifiers=("add_when",), 

     def add(primary,around):
         """CLOS-like 'add' combination"""

         if len(primary)==1:
             inner = primary[0]
             primaries = dispatch.method_list(primary)
             def inner(*__args,**__kw):
                 return reduce(operator.add,primaries(*__args,**__kw))

         return dispatch.method_chain(list(around)+[inner])

Which shows rather nicely how the short forms will be converted to long 
forms.  Another example:

     around = dispatch.MethodGroup("around")
     primary = dispatch.MethodGroup("primary", qualifiers=("and_when",), 

     def And(primary,around):
         """CLOS-like 'and' combination"""

         if len(primary)==1:
             inner = primary[0]
             primaries = dispatch.method_list(primary)
             def inner(*__args,**__kw):
                 for value in primaries(*__args,**__kw):
                     if not value:
                 return value

         return dispatch.method_chain(list(around)+[inner])

This would be then be used like:

      def allowed(...):

      [allowed.and_when("some condition")]
      def allow_only_if_not_blah(...):
          return False

Whew.  I think I'm about worn out for the moment.  Let me try to make a 
list of what I'd need to implement all this:

* A 'MethodGroup' class that knows how to test whether a (possibly 
qualified) method belongs in that group, and strip off the qualification 
wrapper if any.

* A 'method_combination' decorator that constructs a method-combining 
function and a list of qualifier method names, from the short form or the 
long form invocation.

* 'method_list' and 'method_chain' function constructors, where each take a 
list of functions and return a function that either iterates over the 
functions' return results, or allows each function to call the next with 
'dispatch.next_method()', respectively.  They should also have the option 
of allowing the list/chain to resolve any prioritization ambiguities, 
rather than raising AmbiguousMethod when two applicable methods have 
overlapping rule definitions.

REVERSE_DEFINITION_ORDER functions to handle sorting.

* GenericFunction needs a __getattr__ or other mechanism to access 
"qualifier methods" for the qualifiers that are applicable to it.  This 
mechanism also needs to work with GenericFunctions accessed from a class, 
e.g. 'Favoritism.priority.add_when' should be a different object than 
'NormalRules.priority.add_when', with different behavior.

* Make "qualifier methods" detect when they are used in a class, and if so, 
delay doing the actual registration of the corresponding function until the 
class has been created, at which point the registration should be done with 
a condition requiring the first argument to be an instance of the 
applicable class, for that particular function to be applicable.

* Tweak protocols.advice() functions to accept frames as well as depth, to 
make it easier to mix class and function advice, as some of the above need 
to do.

Too bad most of these things are much easier to describe than to code.  :)

Interesting side note: all of this method combination stuff should also be 
applicable to database-backed generic functions.  That is, if you create a 
database with a bunch of business rules in it (following the San Francisco 
"Key/Keyable" pattern), you could still use these method-combining 
techniques to combine the found, applicable rules, as long as you had a way 
to map the persistent form of a rule into 'dispatch'-compatible predicates 
and callables.

Hmmm...  you know, you could actually map almost all of GenericFunction's 
internal DAG onto relational DB tables, especially since relational 
databases can (at least in theory) do some of the traversal optimization 
themselves that GenericFunction objects do.  That is, right now we build 
indexes that map from various value classifications to what cases are 
applicable.  We then do the equivalent of joining these tables to extract a 
collection of applicable rules.  You could do almost the exact same thing 
in a database.  Verrry interesting, indeed.  Maybe I should patent that 
idea, in the grand tradition of today's wonderful "do X, but on a 
computer/in a database/on the Internet" patents.  ;)

More information about the PEAK mailing list