[PEAK] imports (Re: Package organization)

[Phillip J. Eby]

At 11:13 AM 12/4/03 +0200, alexander smishlajev wrote:

>i would prefer peak.api (or whatever name is chosen for such module) to 
>export only core - no primitives, no frameworks.
>
>however, this issue does not affect me or my colleagues since the coding 
>policy in our company is to avoid 'from module import *' in favor of 
>'import module' (better) or 'from module import symbol1, symbol2'.  so we 
>never do 'from peak.api import *'.

I'm confused, on two points.  First, why do you care what it exports if 
you're not going to use import *?  Second, if it doesn't export the 
primitives (especiall NOT_GIVEN and NOT_FOUND), where will you get them from?


>by the way, i would like to mention some problems i discovered in 
>peak.util.imports.
>
>first, there is inconsistence between LazyModule and _loadAndRunHooks: if 
>any of the hooks fail, _loadAndRunHooks still believes that the module is 
>imported (by disabling all postLoadHooks for that module), but LazyModule 
>thinks that the module is *not* imported, and tries to _loadAndRunHooks 
>again upon the next attribute access.  this leads to AlreadyRead exception.

True.  And then the module concludes it's *still* not 
imported.  Unfortunately I don't think there's much I can do about this, 
except maybe allow the hooks to be run more than once, which I don't really 
like.  I suppose I could wipe the module's contents.  Or, I suppose I could 
change things so that warnings were issued for errors in the hooks, and 
things just proceeded forward.  Or maybe I could accumulate all the errors 
and issue a single error for them.  Sigh.  None of these options seems 
really appealing.  It would be better if hooks didn't raise errors, and in 
fact the 'whenImported()' docs say:

     "The hook should not raise any exceptions, or it may prevent later 
hooks from running."

Maybe I should just change it to say, "or it may completely hose the module 
and your entire application."   :)


>second, _loadAndRunHooks may be executed before the module code.  if any 
>of the hooks refer to an attribute of the module being imported, it fail:
>
>[snip traceback]
>
>this makes lazy modules incompatible with non-basic import mechanisms, 
>like imputil.  essentially, they do
>
>     module = imp.new_module(fqname)
>     exec code in module.__dict__
>
>this launches _loadAndRunHooks upon access to module.__dict__ (or any 
>special import attribute, like '__importer__' or '__ispkg__'), when the 
>module code has not been executed, and always leads to the problem shown 
>in above example.

I'm a little puzzled as to how they get to that point, unless 
imp.new_module is returning the module from sys.modules.  I suppose I could 
make the __getattribute__ avoid loading on access to attributes that begin 
and end with double underscores.  But, it's possible that for some 
attributes, this would be a bad thing.  For example, a module's __conform__ 
method would be accessed by adapt().  Without a comprehensive list of which 
attributes to ignore, it would be unlikely to be correct in all cases.  I 
could add such a list of attributes known to be read by importers, I suppose.

What I find confusing here, though, is that an importer shouldn't be 
touching such attributes except *during* the reload() call, and during the 
reload() call the __getattribute__ method has been restored to its normal 
value!  So, I don't understand how you're getting this problem, even though 
I see that it's a problem.


>for my application packaging, i found a workaround to overcome peak module 
>lazyness, so these problems do not bite me much yet.

Just for my information, could you tell me what that is?


>i just think: do lazy core modules make sense at all?  i.e. how common is 
>it for core modules to not instantiate at the very start of the application?

The PEAK core itself uses 'from peak.api import *', so the core itself is 
unable to boot without lazy loading.  It could be changed to use direct 
imports from other packages' modules, but it would be a fair amount of 
work.  I wanted to shield packages from having to know each others' 
internal organization as much as possible.  Indeed, the lazy loading was 
something I originally created in order to deal with the circularities of 
booting the core.