[PEAK] protocols advice
Phillip J. Eby
pje at telecommunity.com
Mon Aug 16 22:53:06 EDT 2004
At 10:05 PM 8/16/04 -0400, Duncan McGreggor wrote:
>I am working on a project that stores RRD files in a directory hierarchy
>(UNIX). We are doing a small implementation for right now, but may be
>requested to do a larger, more robust implementation -- at which point, we
>will most likely move to some sort of RDBMS storage to avoid local
>file-locking issues and allow for a distributed rollout. We also have a
>little nevow appserver that serves RRD files and RRD graphs. For this
>reason (and others), we need to have pythonic access to the data.
>My thoughts are these:
>* Write an interface that defines how we get at our data
>* Write an implementation that actually pulls the data out of
>* Write an adaptor for other storage systems as needed
>The reason I am asking for help, is that this involves a possible mapping
>between a hierarchy database (filesystem -- similar in concept to LDAP? to
>ZODB?) and a relational one... via an adaptor, am I correct? These seems
>to approach many of the conversations on the list about mapping various
>First, though, we would have to be able to use our interface and its
>implementation. I don't quite know where to start with this, and would
>love some advice:
>* Can storage already treat the filesystem as a database (paths, file
>names, and file contents all treated as data)?
peak.storage.files offers a two-tiered transactional interface for
files. The first tier provides the ability to perform file operations
atomically, by directing all writes to an alternative file name, and then
moving/renaming that file to the original name upon commit. The second
tier, for files small enough to fit into memory, adds to the first tier by
offering a simpler API based on just accessing a "text" attribute of the
file object (that represents the file's current contents, or 'None' for a
There is not currently a mechanism that deals with directories or filenames
>* If so, can someone give a quick example or point me to the right resources?
The 'peak.tools.version' package uses peak.storage.files to read and write
its 'version.ini' file. See the 'peak.tools.version.config.VersionStore'
class for an example of that usage (subclassing EditableFile to add
application-specific features) or 'peak.tools.version.config.Editor' for an
example of using the standard EditableFile. (The 'Editor' class applies a
series of "version number" edits to files.)
In order to use transactional files, you have to have an active PEAK
transaction; that's basically done with
'storage.beginTransaction(component)' and friends. See the
'peak.storage.interfaces' and 'peak.storage.transaction' modules for more
(Also, most of these matters are explained and illustrated at length in the
IntroToPeak tutorial on the DevCenter Wiki.)
>* If not, is the LDAP implementation mature? Would that be a good place to
>look for inspiration, or am I barking up the wrong tree?
LDAP isn't transactional, and peak.storage's modules for LDAP and SQL don't
provide object mapping in any case. They're convenience APIs that are
integrated into PEAK's component and naming systems, so that LDAP and SQL
URL's can be mapped to ManagedConnection objects. (A "managed connection"
is one that's not opened until it's used, and can attempt to reconnect
itself if used after being disconnected.)
>* If I need to start from the ground up, and we need to write our own
>interface/implementation, what would be the best way to start? Are there
>any examples or sandboxes where something like this/analogous to this has
>been done? I learn best by example ;-)
If your data access needs are roughly "get some object via a known key",
then the current "Data Manager" framework is likely to suffice. Take a
look at the Wiki's "IntroToPeak" tutorial, which has very good coverage of
the system's current capabilities. Lesson 1 covers basics of PEAK
applications, lesson 2 covers read-only data retrieval from the file
system, lesson 3 shows how to write data to the file system and also how to
use peak.storage.files.EditableFile to make the writes
transactional. Lesson 4 moves the example to SQL-based storage, with a
more sophisticated domain model.
>I am not sure what other information to provide, nor how clearly I have
>stated this information.
Probably the interface you want to use would be helpful, and perhaps an
explanation of what "RRD files" are. The term seems vaguely familiar, but
what it means is not coming immediately to mind.
More information about the PEAK