[TransWarp] Resource directories, file extensions, etc.

[Phillip J. Eby]

For resource directories to work, there needs to be a way to specify what 
permissions and factories apply to what files, both on an individual and 
wildcard basis.

I'm considering a couple of different possibilities for how to do 
this.  First, we could use .ini files, and Ty and I previously figured on 
using a 'resource.ini'.  The main downside to this approach is that only a 
limited set of characters and syntax are supported by property names, so 
certain perfectly legal filenames such as '.foo' will not be supportable by 
.ini file rules.  We could treat '.foo' as being identical to 'foo', which 
isn't perfect, but I suppose it might do.  Specifically, we could look up 
properties for filenames as follows:

parts = filter(None, filename.split('.'))  # removes empty parts
for i in range(len(parts)):
     yield '.'.join(parts[i:])

IOW, we look at all "suffixes" of a filename, longest to 
shortest.  'foo.bar' will match 'foo.bar' and 'bar', 'foo.tar.gz' will 
match 'foo.tar.gz', 'tar.gz', and 'gz'.

We'd also need to clean this up a bit to translate any characters not valid 
in property names (e.g. ' ') to '_' or something.  All in all, this 
approach would work, but it doesn't seem especially intuitive as a syntax, 
e.g.:

[peak.web.resource.types]
tar.gz = web.Whatever
gif = web.Something
resource.ini = web.somethingElse

[peak.web.resource.permissions]
tar.gz = security.Whatever
gif = security.Something
resource.ini = security.somethingElse

Or maybe I'm being too picky.  The main annoyances are the headers and the 
repetition.  I suppose if the file were limited to peak.web.resource 
properties, it could read:

[types]
tar.gz = web.Whatever
gif = web.Something
resource.ini = web.somethingElse

[permissions]
tar.gz = security.Whatever
gif = security.Something
resource.ini = security.somethingElse

And that doesn't look quite as bad.  More of an annoyance is the need to 
define a line for each file type; if I want a rule to match .py, .pyc, 
.pyo, .pyd, and .so files, I'll need a line for each, in each section.  If 
it's a trivial rule, no big deal, but if it involves a fair amount of 
computation, it's going to be really annoying.

So my next thought was ZConfig.  It would let me do something like:

<Files>
FileTypes py pyc pyo pyd so
Permission security.Nobody
Factory peak.web.resources.ForbiddenResource
</Files>

On the upside, this is a lot easier to read, and a <Files> block could 
allow an unlimited number of match regexes, globs, or individual filenames, 
to which the rules would apply.  On the downside, it would be limited to 
expressing data according to a fixed schema; there would be no way for you 
to redefine arbitrary properties.  Also, questions of precedence ordering 
arise.  Do more specific rules take precedence over more general 
ones?  Rules defined earlier?  Later?  Ugh.

Okay, I think I just convinced myself that ZConfig isn't the way to go for 
this, at least at the present time.  The precedence for ini's is pretty 
clear: most specific to least specific, nearest directory to furthest 
directory.  That is, if a directory declares 'foo.bar' to have a particular 
permission, it's shared by all its subdirectories that don't override 
'foo.bar'.  Defining extension 'bar' in the subdirectory doesn't override 
the parent's foo.bar.

Argh.  While that's *clear*, it's not really what we *want*.  To get what 
we want, we'd have to *reverse* the filenames, e.g., to refer to 'spam.gif' 
we'd need to say:

gif.* = rule for gif's in general
gif.spam = rule for spam.gif

Blech.  This does what we want, but it's nonintuitive and ugly.  (On the 
bright side, it requires only one property lookup: just reverse the 
filename and off you go!)

It almost seems like we want a kind of .ini section where you can say:

[files: *.pyc *.pyo *.py]
peak.web.resource.permissions = [security.Nobody]
peak.web.resource.factories = "peak.web.resources.ForbiddenResource"

and have it translate into the equivalent of:

peak.web.resource.permissions.pyc.* = [security.Nobody]
peak.web.resource.permissions.pyo.* = [security.Nobody]
peak.web.resource.permissions.py.* = [security.Nobody]
peak.web.resource.factories.pyc.* = "peak.web.resources.ForbiddenResource"
peak.web.resource.factories.pyo.* = "peak.web.resources.ForbiddenResource"
peak.web.resource.factories.py.* = "peak.web.resources.ForbiddenResource"

Currently, the .ini file parser has fixed, built-in mechanisms for parsing 
special section headers (e.g. "Load Settings From" and "Provide 
Utilities").  If this were extensible, it'd be easy to add our specialized 
rule mechanism for resources.  In fact, we could generalize this to make 
the existing "load settings from", "load on demand", etc. sections work the 
same way.

I think what I'll do is:

* if a [] section title has a space in it, lowercase it and convert it to a 
property name, converting ' ' to '.'

* Look up the resulting name in 'peak.config.iniFile.sectionParsers', and 
use the found object to parse the section.

* and to support both this and conversion of filenames to property names, 
I'll either add a 'force' keyword to the PropertyName constructor, or add a 
class method that automatically converts invalid characters to '_' and the 
like.  This should also further fix the SQL typeMap issue with Oracle that 
Ty ran into the other day.

The net result would then be that we could define filename rules using a 
sensible syntax, producing intuitive results (i.e. *.gif in subdirectory 
overrides foo.gif in parent directory).  The syntax would not allow 
trailing or embedded wildcards (i.e. no foo.*), but these shouldn't really 
be necessary.  And, the syntax extension will only be usable in resource 
directory .ini files, not just any .ini file.

Okay, I think I'll try it.  :)

There was another issue about file extensions I wanted to bring up, but 
it's completely unrelated to any of the above, except that it has to do 
with file extensions.  :)  But I'll save that for a later post.