(Sent to the Distutils-SIG)

Many applications need or want to be able to support extension via dynamic installation of "plugin" code, such as Zope Products, Chandler Parcels, and WSGI servers' "application" objects. Frequently, these plugins may require access to other plugins or to libraries of other Python code. Currently, application platforms must create their own binary formats to address their specific requirements, but these formats are of course specific to the platform and not portable, so it's not possible to package a third-party module just once, and deploy it in any Python application platform.

Although each platform may have its own additional requirements for the contents of such a "plugin", the minimum basis for such plugins is that they include Python modules and other files, and import and export selected modules. Although the standard distutils pattern is to use a platform's own packaging system, this really only makes sense if you are dealing with Python as a language, and not as an application platform. Platform packaging doesn't make sense for applications that are end-user programmable, because even if the core application can be installed in one location, each instance of use of that application (e.g. per user on a multi-user system) may have its own plugins installed.

Therefore, I would like to propose the creation of:

• a binary package format optimized for "plugin" use (probably just a specific .zip layout)
• a distutils "bdist" command to package a set of modules in this format
• A PEP 302 importer that supports running of code directly from plugin distributions, and which can be configured to install or extract C extension modules. (This will serve as a base for platform developers to create their platform-specific plugin registries, installation mechanisms, etc., although there will be opportunity for standardization/code sharing at this level, too.)
• additions to setup() metadata to support declaration of:
  • modules required/provided, with version information
  • platform metadata for C extensions contained in the plugin distribution
  • ability to specify metadata files to be included in the distribution, that are specific to the target application platform (e.g. Zope config files, Chandler parcel schemas, WSGI deployment configuration, etc.)

(This is actually only "level 1" of the standardization that I'd like to do; levels 2 and 3 would address runtime issues like startup/shutdown of plugins, automatic dependency resolution, isolation between plugins, and mediated service discovery and service management across plugins. However, these other levels are distinct deliverables that don't necessarily relate to the distutils, except insofar as those levels may influence requirements for the first level. Also, it's important to note that even without those higher levels of standardization, the availability of a "plug and play" distribution format should be beneficial to the Python community, in making it easier for applications to bundle arbitrary libraries. Indeed, tools like py2exe and py2app might grow the ability to automatically "link" bundles needed by an application, and there will likely be many other spin-off uses of this technology once it's available.)

My main idea is to expand the existing PKG-INFO format, adding some formally-defined fields for system processing, that are supplied by the setup script or in additional files. The additional metadata should be syntactically validated (and semantically, to the extent possible), and the distutils should not knowingly produce a plugin package with invalid execution metadata. The specific kinds of metadata needed (that I know of currently) are:

• Static imports (module names and specification of compatible versions)
• Dynamic imports (wildcard masks)
• Exports (module names and versions)
• Platform requirements for included extension modules
• Other platform requirements
• Entry point for dynamic integration (technically a level 2 feature, but it'd be nice to reserve the header and define its syntax)
• Update URL (for obtaining the "latest" version of the plugin)

There are several issues that would have to be hammered out, here. Versioning, for example, both in the sense of formats and qualifiers, and in the sense of versioning a module/package versus versioning a distribution. Gathering information about imports is also tricky. Tools like py2exe and py2app try to gather some of this information automatically, but that info doesn't include version requirements. (It may be that we can get by without version requirements, taking the default state to mean, "any version will do.")

Platform specs are another dicey issue, since we're talking about trying to define binary compatibility here. This includes the issue that it might be necessary to include shared libraries other than the C extensions themselves. (For example, like the wxWidgets libraries that ship with wxPython.)

While researching a deployment strategy for WSGI, I discovered the OSGi specifications for Java, which address all of these issues and more. Where OSGi's solutions are directly usable, I'd like to apply them, rather than re-inventing wheels... not to mention axles, brakes, and transmissions! However, their syntax for these things is often weird and verbose, and could probably use some "Pythonizing".

Anyway, I would see the deliverables here as being a PEP documenting the format, a prototype implementation in setuptools (for current Python versions), migrating to an official implementation in the distutils for Python 2.5. I'd like to find out "who's with me" at this stage in having an interest in any aspect of this project, especially if you have requirements or issues I haven't thought of. Thanks!