[PEAK] Circular class adapters..

[Bob Ippolito]

On Feb 19, 2004, at 11:07 PM, Phillip J. Eby wrote:

> At 10:31 PM 2/19/04 -0500, Bob Ippolito wrote:
>> If you want to take a look at what I am using circular class adaption 
>> for, I have made a snapshot of ptypes at 
>> http://undefined.org/python/ptypes-20040219.tgz -- you will need to 
>> install a VERY RECENT (as in hours ago) CVS version of PyProtocols if 
>> you want to play with it.
>>
>> It's basically a toolkit for reading/writing binary formats with an 
>> API sort of like ctypes.  It includes support for an "iterfunc" 
>> reading style (generator abuse), so it can be hooked into Twisted and 
>> read off the wire incrementally without needing Stackless.  The goal 
>> is to be able to define these data structures in such a way that it 
>> does as much of the boilerplate reading and writing as possible.
>
> Interesting.  I'm curious about something, though.  You use 
> BaseObjectsList as a base for a lot of things, but it looks to me as 
> though it'd be simpler to just create an adapter from a 
> sequenceOf(IFunctionWritable) -> IFunctionWritable, and from a 
> sequenceOf(IOutputSize) -> IOutputSize, and so on for any other 
> capabilities that need to be done across a sequence of items.  Then, 
> you could just use sequences of "types" directly.
>
> There seem to be other things that also seem one level removed, like 
> the various 'str' subclasses.  But maybe I'm not following the actual 
> use cases of the library correctly.
>
> It would seem to me that I would define a format as a protocol -- 
> possibly parameterized -- and then use adaptation to define writers 
> directly.  For example, instead of having 'pypackable' create a class, 
> I'd make it a Protocol subclass with parameters, that automatically 
> registered appropriate adapter function(s) for builtin types, or maybe 
> from an IInteger, IString, etc.  And the new protocol object would 
> imply a base protocol for writing or reading.
>
> I think this approach, or something like it, could remove one level of 
> "meta" ness throughout the library, making the code simpler and easier 
> to follow, while possibly expanding the flexibility further.  But, 
> I've only made a very brief pass over the code, so I could be missing 
> your critical use cases.

This is a first pass port from something that was poisoned by 
twisted.python.components which originally didn't use adaptation at 
all... I only spent a couple hours on it over the past two days, and I 
didn't spend a lot of time on either of the previous iterations.  When 
porting it, I did see a lot of opportunities to PyProtocol-ize things, 
but I wanted it to work first :)

The str/int subclasses are to maintain metadata.. you don't know if 
it's supposed to be a pascal string or a c string or a short integer or 
a byte if you don't carry around metadata.

Some code using the original (no adaptation whatsoever) ptypes looked 
like this:

class mach_header(Structure):
     _fields_ = (
         ('magic', p_ulong),
         ('cputype', cpu_type_t),
         ('cpusubtype', cpu_subtype_t),
         ('filetype', p_ulong),
         ('ncmds', p_ulong),
         ('sizeofcmds', p_ulong),
         ('flags', p_ulong),
     )
     def _describe(self):
         bit = 1L
         flags = self.flags
         dflags = []
         while flags and bit < (1<<32L):
             if flags & bit:
                 dflags.append(MH_FLAGS_NAMES.get(bit, str(bit)))
                 flags = flags ^ bit
             bit <<= 1L
         return (
             ('magic', '0x%08X' % self.magic),
             ('cputype', CPU_TYPE_NAMES.get(self.cputype, self.cputype)),
             ('cpusubtype', self.cpusubtype),
             ('filetype', MH_FILETYPE_NAMES.get(self.filetype, 
self.filetype)),
             ('ncmds', self.ncmds),
             ('sizeofcmds', self.sizeofcmds),
             ('flags', dflags),
         )

elsewhere....
         fh = file(filename)
         self.header = mach_header.from_fileobj(fh)
         cmd = self.commands = []

         read_bytes = 0
         low_offset = sys.maxint
         for i in range(header.ncmds):
             # read the load command
             cmd_load = load_command.from_fileobj(fh)

... and it goes on to parse an entire Mach O file into a Python data 
structure, lets you mutate it, and then serialize back to disk.  Mach O 
is the OS X / Darwin equivalent to ELF or PE (I think it was PE on 
win32).  I use it to do dependency walking of python extensions to 
determine what shared libraries they depend on, and then rewrite their 
headers dynamically such that they can live happily inside of a 
standalone application bundle (a use case like py2exe).  It's hard to 
explain why I need to rewrite load commands in the headers because dyld 
is quite different than most linkers, but it's necessary to make things 
work correctly without re-linking everything for standalone 
distribution :)

-bob