The PEAK Developers' Center   setuptools UserPreferences
 
HelpContents Search Diffs Info Edit Subscribe XML Print View
Version as of 2005-07-08 11:15:43

Clear message


Building and Distributing Packages with setuptools

setuptools is a collection of enhancements to the Python distutils (for Python 2.3 and up) that allow you to more easily build and distribute Python packages, especially ones that have dependencies on other packages.

Packages built and distributed using setuptools look to the user like ordinary Python packages based on the distutils. Your users don't need to install or even know about setuptools in order to use them, and you don't have to include the entire setuptools package in your distributions. By including just a single bootstrap module (a 5K .py file), your package will automatically download and install setuptools if the user is building your package from source and doesn't have a suitable version already installed.

Feature Highlights:

Table of Contents

Developer's Guide

Installing setuptools

Windows users can just download and run the setuptools binary installer for Windows. All others should just download ez_setup.py and run it; this will download and install the appropriate egg for your Python version.

You may receive a message telling you about an obsolete version of setuptools being present; if so, you must be sure to delete it entirely, along with the old pkg_resources module if it's present on sys.path.

Basic Use

For basic use of setuptools, just import things from setuptools instead of the distutils. Here's a minimal setup script using setuptools:

from setuptools import setup, find_packages
setup(
    name = "HelloWorld",
    version = "0.1",
    packages = find_packages(),
)

As you can see, it doesn't take much to use setuptools in a project. Just by doing the above, this project will be able to produce eggs, upload to PyPI, and automatically include all packages in the directory where the setup.py lives. See the sections below on Enhanced Commands and New Commands to see what commands you can give to this setup.py.

Of course, before you release your project to PyPI, you'll want to add a bit more information to your setup script to help people find or learn about your project. And maybe your project will have grown by then to include a few dependencies, and perhaps some data files and scripts:

from setuptools import setup, find_packages
setup(
    name = "HelloWorld",
    version = "0.1",
    packages = find_packages(),
    scripts = ['say_hello.py'],

    # Project uses reStructuredText, so ensure that the docutils get
    # installed or upgraded on the target machine
    install_requires = ['docutils>=0.3'],

    package_data = {
        # If any package contains *.txt or *.rst files, include them:
        '': ['*.txt', '*.rst'],
        # And include any *.msg files found in the 'hello' package, too:
        'hello': ['*.msg'],
    }

    # metadata for upload to PyPI
    author = "Me",
    author_email = "me@example.com",
    description = "This is an Example Package",
    license = "PSF",
    keywords = "hello world example examples",
    url = "http://example.com/HelloWorld/",   # project home page, if any

    # could also include long_description, download_url, classifiers, etc.
)

In the following sections, we'll explain what most of these

Declaring Dependencies

setuptools supports automatically installing dependencies when a package is installed, and including information about dependencies in Python Eggs (so that package management tools like EasyInstall can use the information).

setuptools and pkg_resources use a common syntax for specifying a project's required dependencies. This syntax consists of a project's PyPI name, optionally followed by a comma-separated list of "extras" in square brackets, optionally followed by a comma-separated list of version specifiers. A version specifier is one of the operators <, >, <=, >=, == or !=, followed by a version identifier. Tokens may be separated by whitespace, but any whitespace or nonstandard characters within a project name or version identifier must be replaced with -.

Here are some example requirement specifiers:

docutils >= 0.3

# comment lines and \ continuations are allowed in requirement strings
BazSpam ==1.1, ==1.2, ==1.3, ==1.4, ==1.5, \
    ==1.6, ==1.7  # and so are line-end comments

PEAK[FastCGI, reST]>=0.5a4

setuptools==0.5a7

The simplest way to include requirement specifiers is to use the install_requires argument to setup(). It takes a string or list of strings containing requirement specifiers. If you include more than one requirement in a string, each requirement must begin on a new line.

This has three effects:

  1. When your project is installed, either by using EasyInstall, setup.py install, or setup.py develop, all of the dependencies not already installed will be located (via PyPI), downloaded, built (if necessary), and installed.
  2. Any scripts in your project will be installed with wrappers that verify the availability of the specified dependencies at runtime, and ensure that the correct versions are added to sys.path (e.g. if multiple versions have been installed).
  3. Python Egg distributions will include a metadata file listing the dependencies.

Note, by the way, that if you declare your dependencies in setup.py, you do not need to use the require() function in your scripts or modules, as long as you either install the project or use setup.py develop to do development work on it. (See "Development Mode" below for more details on using setup.py develop.)

Declaring "Extras" (optional features with their own dependencies)

Sometimes a project has "recommended" dependencies, that are not required for all uses of the project. For example, a project might offer optional PDF output if ReportLab is installed, and reStructuredText support if docutils is installed. These optional features are called "extras", and setuptools allows you to define their requirements as well. In this way, other projects that require these optional features can force the additional requirements to be installed, by naming the desired extras in their install_requires.

For example, let's say that Project A offers optional PDF and reST support:

setup(
    name="Project-A",
    ...
    extras_require = {
        'PDF':  ["ReportLab>=1.2", "RXP"],
        'reST': ["docutils>=0.3"],
    }
)

And that project B needs project A, with PDF support:

setup(
    name="Project-B",
    install_requires = ["Project-A[PDF]"],
    ...
)

This will cause ReportLab to be installed along with project A, if project B is installed -- even if project A was already installed. In this way, a project can encapsulate groups of optional "downstream dependencies" under a feature name, so that packages that depend on it don't have to know what the downstream dependencies are. If a later version of Project A builds in PDF support and no longer needs ReportLab, or if it ends up needing other dependencies besides ReportLab in order to provide PDF support, Project B's setup information does not need to change, but the right packages will still be installed if needed.

As you can see, the extras_require argument takes a dictionary mapping names of "extra" features, to strings or lists of strings describing those features' requirements. These requirements will not be automatically installed unless another package depends on them (directly or indirectly) by including the desired "extras" in square brackets after the associated project name. (Or if the extras were listed in a requirement spec on the EasyInstall command line.)

Note, by the way, that if a project ends up not needing any other packages to support a feature, it should keep an empty requirements list for that feature in its extras_require argument, so that packages depending on that feature don't break (due to an invalid feature name). For example, if Project A above builds in PDF support and no longer needs ReportLab, it should change its setup to this:

setup(
    name="Project-A",
    ...
    extras_require = {
        'PDF':  [],
        'reST': ["docutils>=0.3"],
    }
)

so that Package B doesn't have to remove the [PDF] from its requirement specifier.

Distributing a setuptools-based package

Your users might not have setuptools installed on their machines, or even if they do, it might not be the right version. Fixing this is easy; just download ez_setup.py, and put it in the same directory as your setup.py script. (Be sure to add it to your revision control system, too.) Then add these two lines to the very top of your setup script, before the script imports anything from setuptools:

import ez_setup
ez_setup.use_setuptools()

That's it. The ez_setup module will automatically download a matching version of setuptools from PyPI, if it isn't present on the target system. Whenever you install an updated version of setuptools, you should also update your projects' ez_setup.py files, so that a matching version gets installed on the target machine(s). If you're using Subversion, you may wish to use "externals" to make all your projects share a single ez_setup.py, so that you only have to update it in one place to update all your projects.

By the way, setuptools supports the new PyPI "upload" command, so you can use setup.py sdist upload or setup.py bdist_egg upload to upload your source or egg distributions respectively. Your project's current version must be registered with PyPI first, of course; you can use setup.py register to do that. Or you can do it all in one step, e.g. setup.py register sdist bdist_egg upload will register the package, build source and egg distributions, and then upload them both to PyPI, where they'll be easily found by other projects that depend on them.

Including Data Files

The distutils have traditionally allowed installation of "data files", which are placed in a platform-specific location. However, the most common use case for data files distributed with a package is for use by the package, usually by including the data files in the package directory. Setuptools supports this by allowing a package_data argument to setup(), e.g.:

from setuptools import setup, find_packages
setup(
    ...
    package_data = {
        # If any package contains *.txt or *.rst files, include them:
        '': ['*.txt', '*.rst'],
        # And include any *.msg files found in the 'hello' package, too:
        'hello': ['*.msg'],
    }
)

The package_data argument is a dictionary that maps from package names to lists of glob patterns. The globs may include subdirectory names, if the data files are contained in a subdirectory of the package. For example, if the package tree looks like this:

setup.py
src/
    mypkg/
        __init__.py
        mypkg.txt
        data/
            somefile.dat
            otherdata.dat

The setuptools setup file might look like this:

from setuptools import setup, find_packages
setup(
    ...
    packages = find_packages('src'),  # include all packages under src
    package_dir = {'':'src'},   # tell distutils packages are under src
    
    package_data = {
        # If any package contains *.txt files, include them:
        '': ['*.txt'],
        # And include any *.dat files found in the 'data' subdirectory
        # of the 'mypkg' package, also:
        'mypkg': ['data/*.dat'],
    }
)

Notice that if you list patterns in package_data under the empty string, these patterns are used to find files in every package, even ones that also have their own patterns listed. Thus, in the above example, the mypkg.txt file gets included even though it's not listed in the patterns for mypkg.

Also notice that if you use paths, you must use a forward slash (/) as the path separator, even if you are on Windows. Setuptools automatically converts slashes to appropriate platform-specific separators at build time.

(Note: although the package_data argument was previously only available in setuptools, it was also added to the Python distutils package as of Python 2.4; there is some documentation for the feature available on the python.org website.)

Accessing Data Files at Runtime

Typically, existing programs manipulate a package's __file__ attribute in order to find the location of data files. However, this manipulation isn't compatible with PEP 302-based import hooks, including importing from zip files and Python Eggs. It is strongly recommended that, if you are using data files, you should use the Resource Management API of pkg_resources to access them. The pkg_resources module is distributed as part of setuptools, so if you're using setuptools to distribute your package, there is no reason not to use its resource management API. See also Accessing Package Resources for a quick example of converting code that uses __file__ to use pkg_resources instead.

"Development Mode"

Sorry, this section isn't written yet, and neither is anything below this point except the changelog. You might want to subscribe to changes in this page to see when the documentation is added or updated.

Enhanced Commands

New Commands

Extending and Reusing setuptools

Release Notes/Change History

0.5a8
  • The "egg_info" command now always sets the distribution metadata to "safe" forms of the distribution name and version, so that distribution files will be generated with parseable names (i.e., ones that don't include '-' in the name or version). Also, this means that if you use the various --tag options of "egg_info", any distributions generated will use the tags in the version, not just egg distributions.
  • Added support for defining command aliases in distutils configuration files, under the "[aliases]" section. To prevent recursion and to allow aliases to call the command of the same name, a given alias can be expanded only once per command-line invocation. You can define new aliases with the "alias" command, either for the local, global, or per-user configuration.
  • Added "rotate" command to delete old distribution files, given a set of patterns to match and the number of files to keep. (Keeps the most recently-modified distribution files matching each pattern.)
  • Added "saveopts" command that saves all command-line options for the current invocation to the local, global, or per-user configuration file. Useful for setting defaults without having to hand-edit a configuration file.
  • Added a "setopt" command that sets a single option in a specified distutils configuration file.
0.5a7
  • Added "upload" support for egg and source distributions, including a bug fix for "upload" and a temporary workaround for lack of .egg support in PyPI.
0.5a6
  • Beefed up the "sdist" command so that if you don't have a MANIFEST.in, it will include all files under revision control (CVS or Subversion) in the current directory, and it will regenerate the list every time you create a source distribution, not just when you tell it to. This should make the default "do what you mean" more often than the distutils' default behavior did, while still retaining the old behavior in the presence of MANIFEST.in.
  • Fixed the "develop" command always updating .pth files, even if you specified -n or --dry-run.
  • Slightly changed the format of the generated version when you use --tag-build on the "egg_info" command, so that you can make tagged revisions compare lower than the version specified in setup.py (e.g. by using --tag-build=dev).
0.5a5
  • Added develop command to setuptools-based packages. This command installs an .egg-link pointing to the package's source directory, and script wrappers that execfile() the source versions of the package's scripts. This lets you put your development checkout(s) on sys.path without having to actually install them. (To uninstall the link, use use setup.py develop --uninstall.)
  • Added egg_info command to setuptools-based packages. This command just creates or updates the "projectname.egg-info" directory, without building an egg. (It's used by the bdist_egg, test, and develop commands.)
  • Enhanced the test command so that it doesn't install the package, but instead builds any C extensions in-place, updates the .egg-info metadata, adds the source directory to sys.path, and runs the tests directly on the source. This avoids an "unmanaged" installation of the package to site-packages or elsewhere.
  • Made easy_install a standard setuptools command, moving it from the easy_install module to setuptools.command.easy_install. Note that if you were importing or extending it, you must now change your imports accordingly. easy_install.py is still installed as a script, but not as a module.
0.5a4
  • Setup scripts using setuptools can now list their dependencies directly in the setup.py file, without having to manually create a depends.txt file. The install_requires and extras_require arguments to setup() are used to create a dependencies file automatically. If you are manually creating depends.txt right now, please switch to using these setup arguments as soon as practical, because depends.txt support will be removed in the 0.6 release cycle. For documentation on the new arguments, see the setuptools.dist.Distribution class.
  • Setup scripts using setuptools now always install using easy_install internally, for ease of uninstallation and upgrading. Note: you must remove any extra_path argument from your setup script, as it conflicts with the proper functioning of the easy_install command.
  • pkg_resources.AvailableDistributions.resolve() and related methods now accept an installer argument: a callable taking one argument, a Requirement instance. The callable must return a Distribution object, or None if no distribution is found. This feature is used by EasyInstall to resolve dependencies by recursively invoking itself.
0.5a1
  • Added support for "self-installation" bootstrapping. Packages can now include ez_setup.py in their source distribution, and add the following to their setup.py, in order to automatically bootstrap installation of setuptools as part of their setup process:

    from ez_setup import use_setuptools
    use_setuptools()
    
    from setuptools import setup
    # etc...
    
0.4a4
  • Fix problems with resource_listdir(), resource_isdir() and resource directory extraction for zipped eggs.
0.4a3
  • Fixed scripts not being able to see a __file__ variable in __main__
  • Fixed a problem with resource_isdir() implementation that was introduced in 0.4a2.
0.4a2
  • Added ez_setup.py installer/bootstrap script to make initial setuptools installation easier, and to allow distributions using setuptools to avoid having to include setuptools in their source distribution.
  • All downloads are now managed by the PackageIndex class (which is now subclassable and replaceable), so that embedders can more easily override download logic, give download progress reports, etc. The class has also been moved to the new setuptools.package_index module.
  • The Installer class no longer handles downloading, manages a temporary directory, or tracks the zip_ok option. Downloading is now handled by PackageIndex, and Installer has become an easy_install command class based on setuptools.Command.
  • There is a new setuptools.sandbox.run_setup() API to invoke a setup script in a directory sandbox, and a new setuptools.archive_util module with an unpack_archive() API. These were split out of EasyInstall to allow reuse by other tools and applications.
  • setuptools.Command now supports reinitializing commands using keyword arguments to set/reset options. Also, Command subclasses can now set their command_consumes_arguments attribute to True in order to receive an args option containing the rest of the command line.
0.4a1
  • Fixed a bug in requirements processing for exact versions (i.e. == and !=) when only one condition was included.
  • Added safe_name() and safe_version() APIs to clean up handling of arbitrary distribution names and versions found on PyPI.
0.3a4
  • pkg_resources now supports resource directories, not just the resources in them. In particular, there are resource_listdir() and resource_isdir() APIs.
  • pkg_resources now supports "egg baskets" -- .egg zipfiles which contain multiple distributions in subdirectories whose names end with .egg. Having such a "basket" in a directory on sys.path is equivalent to having the individual eggs in that directory, but the contained eggs can be individually added (or not) to sys.path. Currently, however, there is no automated way to create baskets.
  • Namespace package manipulation is now protected by the Python import lock.
0.3a2
  • Added new options to bdist_egg to allow tagging the egg's version number with a subversion revision number, the current date, or an explicit tag value. Run setup.py bdist_egg --help to get more information.
  • Misc. bug fixes
0.3a1
Initial release.

PythonPowered
EditText of this page (last modified 2005-07-08 11:15:43)
FindPage by browsing, title search , text search or an index
Or try one of these actions: AttachFile, DeletePage, LikePages, LocalSiteMap, SpellCheck