Type Checking and Conversion

In the simplest case, a recordtype function need not contain any code or return any value. In such a case, the argument names -- and default values, if any -- are sufficient to describe how the resulting record type should behave. However, if you wish to provide type checking or conversion of arguments, you will need to write a bit more code in a record type. For example, here's a new version of the example above, that does a bit more work to ensure it is used correctly:

@sharing.recordtype("http://schemas.osafoundation.org/pim/contentitem")
def itemrecord(itsUUID, title, body, createdOn, description, lastModifiedBy):
    """Record type for content items"""
    if isinstance(lastModifiedBy, schema.Item):
         lastModifiedBy = lastModifiedBy.itsUUID
    if not isinstance(lastModifiedBy, UUID):
        raise TypeError("lastModifiedBy must be an item or a UUID")
    return itsUUID, title, body, createdOn, description, lastModifiedBy

Note, however, that although a recordtype function can accept items as input, it cannot return items as output. They must be converted to UUIDs, strings, numbers, or other elementary values. The EIM API is record-oriented, not object-oriented.

Inter-Record Dependencies

Just as in a relational database, records may contain references to other records. For example, let's suppose that we want to have a record type to record "tags" associated with a content item. And, we want tags to be a kind of content item themselves. Here's what we would do:

@sharing.recordtype("http://schemas.osafoundation.org/pim/tag",
    itsUUID = itemrecord.itsUUID
)
def tag(itsUUID, tagname):
    """Record type for a tag; "inherits" from itemrecord"""

@sharing.recordtype("http://schemas.osafoundation.org/pim/contentitem/tags",
    item = itemrecord.itsUUID, tag = tag.itsUUID
)
def tagging(item, tag):
    """Record type linking tags to items"""
    if isinstance(item, schema.Item):
        item = item.itsUUID
    if isinstance(tag, schema.Item):
        tag = tag.itsUUID
    if not isinstance(item, UUID):
        raise TypeError("must be an item or a UUID", item)
    if not isinstance(tag, UUID):
        raise TypeError("must be an item or a UUID", tag)
    return item, tag

Keyword arguments passed to the recordtype() decorator allow you to define relationships between the fields in the record type being defined, and the fields of existing record types. As you can see above, we use itemrecord.itsUUID and tag.itsUUID to refer to the itsUUID fields of the itemrecord and tag record types. This creates a dependency between the record types, and affects the order in which records will be imported or exported.

In the examples above, the order of record processing will always begin with itemrecord, followed by tag and tagging records. More specifically, before a tagging record is processed, any tag and itemrecord records that have matching itsUUID fields will be processed. And before a tag record is processed, any itemrecord with the same itsUUID will be processed first.

Recordtype Evolution

As an application's schema changes, it may be necessary to add new fields to existing record types. This can be done, as long as:

1. New fields are added to the end of the existing fields in the record type function.
2. New fields must have a default value defined, and import code for the record type must be able to handle a value of sharing.NoChange. (This allows two Chandlers with different versions of a parcel to interoperate, even if one supports fields that the other does not.)
3. The record type's URI must not change, and all existing fields's names must remain the same and in the same order.

In other words, if you want to change the name, meaning, or position of an existing field (or remove fields), you must create a new recordtype with a new URI to replace the old one. Such replacement also means that you must create a new sharing.Schema in order to retain backward compatibility with older sharing clients. (This topic will be covered in more detail below, since we haven't talked about Schema classes yet.)

Export Methods

In order to function, a sharing.Schema subclass must define "exporter" and "importer" methods. Continuing our simple item/tags example, let's look at some exporters:

class ContentSchema(sharing.Schema):
    uri = "http://schemas.osafoundation.org/pim"
    version = 1
    description = _("Core content items")

    @sharing.exporter(pim.ContentItem)
    def export_contentitem(self, item):
        yield itemrecord(
            item.itsUUID, item.title, item.body, item.createdOn,
            item.description, item.lastModifiedBy
        )
        for t in item.tags:
            yield tagging(item, t)

    @sharing.exporter(pim.Tag)
    def export_tag(self, item):
        yield tag(item.itsUUID, item.tagname)

An exporter method is declared using @sharing.exporter(cls, ...), to indicate what class or classes of items are handled by that method. Methods may be generators that yield records, or they can just return a list or other iterable object that yields records.

More than one exporter can be called for the same item. In the example above, assuming that pim.Tag is a subclass of pim.ContentItem, then the export_contentitem() method will be called before export_tag() for each pim.Tag item being exported. The same principle applies for export methods that apply to annotation classes; the export method for each applicable annotation class will be called. All of the records supplied by the various export methods are then output.

Notice that this means that export methods must be written in such a way that they do not produce duplicate records. Each export method should therefore confine itself to writing records specific to the class(es) it is registered for, and allowing the base class export methods to handle the base classes' data.

If you subclass your sharing.Schema, the subclass inherits all of the export methods defined by the base class. If you wish to redefine the export handling for some particular item or annotation class, you must do so by explicitly using a new @sharing.exporter() decoration; it is not sufficient to just override a method with the same name. (This is because for performance reasons, the lookup mechanism is not based on method names.)

Finally, you can declare more than one exporter for the same type in the same sharing.Schema class; both will be called for items they apply to.

Importer Methods

Each sharing.Schema must declare "importer" methods to handle each record type that it outputs. Here are some importers for the record types we defined previously:

class ContentSchema(sharing.Schema):

    # ...

    @itemrecord.importer
    def import_contentitem(self, record):
        self.loadItemByUUID(
            record.itsUUID, pim.ContentItem,
            title = record.title,
            body = record.body,
            createdOn = record.createdOn,
            description = record.description,
            lastModifiedBy = self.loadItemByUUID(record.lastModifiedBy)
        )

    @tag.importer
    def import_tag(self, record):
        self.loadItemByUUID(record.itsUUID, pim.Tag, tagname=record.tagname)

    @tagging.importer
    def import_tagging(self, record):
        the_item = self.loadItemByUUID(record.item)
        the_tag = self.loadItemByUUID(record.tag)
        the_item.tags.add(the_tag)

Notice that importer methods do not need to return a value; their sole purpose is to do whatever processing is required for the received records.

Only one importer can be registered for a given record type in a particular Schema subclass. Importers registered by base classes are inherited in subclasses, unless overridden using the appropriate decorator in the subclass. If you don't want to inherit or override support for a particular record type, the record type can be listed in the do_not_import attribute of the class, e.g.:

do_not_import = sometype, othertype, ...

Utility Methods

The loadItemByUUID() method shown in the importer examples above is a utility method provided by the sharing.Schema base class. It takes a UUID, an optional item or annotation class, and keyword arguments for attributes to set. The return value is an item of the specified class, or a plain schema.Item if no class was specified and the item didn't already exist.

If an item with the given UUID already exists, it's returned. If a class was specified, the item's kind is upgraded if necessary. For example, the importer for the tag recordtype above invokes it like this:

self.loadItemByUUID(record.itsUUID, pim.Tag, tagname=record.tagname)

If a pim.ContentItem of the right UUID exists, its kind is upgraded to pim.Tag. If it does not exist, it is created as a pim.Tag. If an item exists, and it has a kind that is a subclass of pim.Tag, its kind will not be changed. This algorithm allows items' types to be upgraded "just in time" as information becomes available.

If any of the attribute values supplied to loadItemByUUID() are sharing.NoChange, no change is made to the attribute. Similarly, if the UUID supplied to loadItemByUUID() is sharing.NoChange, sharing.NoChange is returned instead of an item.

Over time, there will be additional utility methods added to sharing.Schema as common usage patterns are identified, to help reduce the amount of boilerplate code that needs to be written.

The Sharing Interface

For each import or export operation to be performed, the sharing framework will create instances of the appropriate sharing.Schema subclasses, passing in a repository view. So in our running example, the sharing framework would invoke ContentSchema(rv) to get a ContentSchema instance with an itsView of rv.

Then, depending on the operation(s) to be performed, the sharing framework will call some of the following methods, which all have reasonable default implementations provided by sharing.Schema:

startExport()

Called before an export process begins, to allow the Schema instance to do any pre-export setup operations. The default implementation does nothing, but can be overridden to initialize any data structures that might be needed during the export operation.

exportItem(item)

Called to export an individual item, it should return a sequence or be a generator yielding the relevant records for the supplied item. The default implementation automatically looks up the registered export methods and calls them, combining their results for the return value. This method can be overridden if you have a sufficiently complex special case to need it, or if you want to create a different way of registering exporters. Note also that it's okay for this method to return an empty sequence.

(Note: the sharing framework must not make any assumptions about a relationship between the records returned, and the item passed in, since some of the records may be for related items. Also, a schema can choose not to export records for individual items, but instead just track which items are to be exported and then provide all of the records when finishExport() is called.)

finishExport()

Called after an export operation is completed, this method should return a sequence or be a generator yielding records. These records will be exported along with any that were yielded by calls to exportItem(). The default implementation of this method just returns an empty list, but can be overridden to return or yield records, and perhaps to tear down any temporary data structures created by beginExport() or exportItem().

startImport()

Called before an import operation begins. The default implementation does nothing, but can be overridden to initialize any data structures that might be needed during the import operation.

importRecord(record)

Called for each record to be imported, in an order determined automatically by the declared inter-recordtype dependencies. (That is, this method will not be passed a record until all the records it depends on have been imported first.) The default implementation of this method simply looks up and calls the relevant importer method.

finishImport()

Called after an import operation is completed. The default implementation does nothing, but can be overridden to do any necessary cleanup or finish-out of the import process.

Notice that for both importRecord() and exportItem(), there is no requirement that all processing for the given item or record take place immediately. Some complex schema changes (or complex schemas) may need or want to simply keep track of what items are being exported or what records are being imported, and then do the actual importing or exporting in finishImport() or finishExport().

Thus, the sharing framework must not assume that it has seen all records until all finishExport() methods (for each schema being exported) have been called. Similarly, it cannot assume that items in the repository are in their finished state until all of the active schemas' finishImport() methods have been called.

Processing "Diffs"

Most of the API and examples above are written in terms that assume a more-or-less "complete" and "additive" transfer of records, rather than being difference-oriented.

It is assumed that sharing.NoChange will be used in record fields to indicate that the field's value has not changed, and that the sharing framework will be responsible for replacing records appropriately. Record objects will probably support subtraction to produce diffs, e.g. diffRecord = newRecord - oldRecord. It's possible that the sharing API will do this by exporting both old and new versions of the same collection, and then differencing the records that are in common, and perhaps creating some kind of "deletion" record for records found in the old, but not the new.

At present, however, the API as designed has no support for deletion as such. For well-defined collections (such as the .tags attribute in the examples), this could be handled by clearing the collection when the first record is received, at the cost of re-transmitting all members of the collection. The alternative possibility is to never delete items from collections, only add. (Which is what the above examples do; i.e., tags are always added, and items are always created or updated, but nothing is ever deleted.)

Key Management

The proposed API doesn't have a way to specify what fields of a record are "keys" or are expected to be unique, except indirectly. Inter-record dependencies define some keys by implication, in that the depended-on field must be unique in order for a dependency to have meaning.

However, producing diffs for a record requires that the record know of one or more fields that produce a "primary key" in database terminology, because a difference record must always contain enough information for the receiver to identify what the difference is to be applied to!

At this point, it's not clear to me if we will need some special way to designate a primary key. One obvious way to do it would be to assume that the first field is always the primary key, except that this doesn't work for records like the tagging example, which effectively have all their fields as part of the primary key.

Type Information

Currently, there is no way to define or look up what types are used in what fields, nor is there any formal definition of what types are acceptable. This is a big gaping hole in the current proposal that must be remedied before we can expect any sort of dependable interoperability (e.g. w/Cosmo). For now, we are punting on this until we get a better idea of what's actually needed.

This gap in the proposal also means that we aren't in a position to e.g. define a bunch of record types to describe other record types. This kind of meta-description is important for being able to define an extensible/discoverable sharing format between Chandler and Cosmo.

Multiple Inheritance

There are a few quirks regarding multiple inheritance. First, I think that we're going to have to prohibit a sharing.Schema class from inheriting from more than one other sharing.Schema class, in order to avoid possible ambiguities as to what inherited importers or exporters should be invoked when both base classes have different ones defined, and the subclass doesn't override them.

Second, there is a peculiar corner case that can arise when sharing data between two machines, when multiple parcels and multiple inheritance are involved. Suppose that there are two parcels "a" and "b" containing classes "A" and "B" respectively, both of which are subclasses of pim.Item. And then there is a parcel "c", containing class "C", which inherits from both "A" and "B".

Let us further say that machine 1 has all three parcels installed, but machine 2 has only parcels "a" and "b". As long as these two machines are only sharing instances of "A" and "B", everything will be fine, but if machine 1 transmits a "C" instance to machine 2 there will be a problem.

When machine 2 tries to process the records related to pim.Item or to "A" instances, everything will work correctly. However, the "C" instance will have created both "A" and "B" records, making it impossible for loadItemByUUID() to find a suitable kind. Morgen and I discussed the possibility of having it simply synthesize one, but this could produce some problems of its own, in that the Chandler UI might not know how to correctly display this peculiar A/B hybrid, without additional information that can only be found in parcel "c" -- which machine 2 does not have.

For the first version, we will probably have to have some kind of kludge to detect this situation and handle it -- but precisely how we will handle it is still open to investigation. We may have to create the problem first in order to get a better handle on it.

Schema Registry and Selection

sharing.Schema classes and @sharing.recordtype objects will have to be part of a parcel's persistent data, stored in the repository at the same time that the parcel's kinds and annotations and so forth are initialized. The sharing parcel will probably have some kind of persistent object(s) stored in the repository that reference schemas and index them by their supported record types and kinds, so that the sharing framework can look them up.

The exact nature of these data structures is currently undefined. The data structures needed are dependent on how schemas will need to be selected by the sharing framework, so it's likely that a first cut implementation of the API won't actually create any, and rely on the sharing framework to just explicitly select what schema(s) to use for a particular share.

The selection strategy is further complicated by the possibility that more than one schema might be offering to produce or consume records of the same record type.

And last, but not least, due to the persistent nature of schema classes and recordtype objects, it's likely that the Chandler application will need to either set aside another parcel to contain the core types' sharing schema, or else define that schema within the sharing parcel itself. (Otherwise, we would be introducing circular parcel dependencies between the core types and the sharing parcel.)