[PEAK] protocols advice

[Phillip J. Eby]

At 10:05 PM 8/16/04 -0400, Duncan McGreggor wrote:
>I am working on a project that stores RRD files in a directory hierarchy 
>(UNIX). We are doing a small implementation for right now, but may be 
>requested to do a larger, more robust implementation -- at which point, we 
>will most likely move to some sort of RDBMS storage to avoid local 
>file-locking issues and allow for a distributed rollout. We also have a 
>little nevow appserver that serves RRD files and RRD graphs. For this 
>reason (and others), we need to have pythonic access to the data.
>
>My thoughts are these:
>* Write an interface that defines how we get at our data
>* Write an implementation that actually pulls the data out of 
>paths/filenames/file contents
>* Write an adaptor for other storage systems as needed
>
>The reason I am asking for help, is that this involves a possible mapping 
>between a hierarchy database (filesystem -- similar in concept to LDAP? to 
>ZODB?) and a relational one... via an adaptor, am I correct? These seems 
>to approach many of the conversations on the list about mapping various 
>relational systems...
>
>First, though, we would have to be able to use our interface and its 
>implementation. I don't quite know where to start with this, and would 
>love some advice:
>* Can storage already treat the filesystem as a database (paths, file 
>names, and file contents all treated as data)?

peak.storage.files offers a two-tiered transactional interface for 
files.  The first tier provides the ability to perform file operations 
atomically, by directing all writes to an alternative file name, and then 
moving/renaming that file to the original name upon commit.  The second 
tier, for files small enough to fit into memory, adds to the first tier by 
offering a simpler API based on just accessing a "text" attribute of the 
file object (that represents the file's current contents, or 'None' for a 
nonexistent file).

There is not currently a mechanism that deals with directories or filenames 
etc.


>* If so, can someone give a quick example or point me to the right resources?

The 'peak.tools.version' package uses peak.storage.files to read and write 
its 'version.ini' file.  See the 'peak.tools.version.config.VersionStore' 
class for an example of that usage (subclassing EditableFile to add 
application-specific features) or 'peak.tools.version.config.Editor' for an 
example of using the standard EditableFile.  (The 'Editor' class applies a 
series of "version number" edits to files.)

In order to use transactional files, you have to have an active PEAK 
transaction; that's basically done with 
'storage.beginTransaction(component)' and friends.  See the 
'peak.storage.interfaces' and 'peak.storage.transaction' modules for more 
details.

(Also, most of these matters are explained and illustrated at length in the 
IntroToPeak tutorial on the DevCenter Wiki.)



>* If not, is the LDAP implementation mature? Would that be a good place to 
>look for inspiration, or am I barking up the wrong tree?

LDAP isn't transactional, and peak.storage's modules for LDAP and SQL don't 
provide object mapping in any case.  They're convenience APIs that are 
integrated into PEAK's component and naming systems, so that LDAP and SQL 
URL's can be mapped to ManagedConnection objects.  (A "managed connection" 
is one that's not opened until it's used, and can attempt to reconnect 
itself if used after being disconnected.)


>* If I need to start from the ground up, and we need to write our own 
>interface/implementation, what would be the best way to start? Are there 
>any examples or sandboxes where something like this/analogous to this has 
>been done? I learn best by example ;-)

If your data access needs are roughly "get some object via a known key", 
then the current "Data Manager" framework is likely to suffice.  Take a 
look at the Wiki's "IntroToPeak" tutorial, which has very good coverage of 
the system's current capabilities.  Lesson 1 covers basics of PEAK 
applications, lesson 2 covers read-only data retrieval from the file 
system, lesson 3 shows how to write data to the file system and also how to 
use peak.storage.files.EditableFile to make the writes 
transactional.  Lesson 4 moves the example to SQL-based storage, with a 
more sophisticated domain model.


>I am not sure what other information to provide, nor how clearly I have 
>stated this information.

Probably the interface you want to use would be helpful, and perhaps an 
explanation of what "RRD files" are.  The term seems vaguely familiar, but 
what it means is not coming immediately to mind.