Distutils - Usage Instructions ============================== Currently, there are two group of users handled by the Distutils: module developers and module installers (sysadmins or end-users). Generally, writing and edting the setup.py script is the domain of the developer, while running it -- and using its extensive command-line options -- is for end users. Of course, developers will need to run the script too. And in this early release of the Distutils, there are probably situations where the user will have to edit the setup script. (This is something to be avoided, though, and as the capabilities of the Distutils expand, it will become Very Bad Form to require your users to edit *anything* in order to build and install your module distribution!) Thus, this document first covers using the Distutils for installation via the setup script, since everyone has to do that. It then covers the contents of the setup script, for developers and adventurous/unlucky installers. Installer instructions ---------------------- [Eventually, this text should transmute into a short "Installing Python Modules" document -- perhaps part of the standard documentation, or perhaps a "how-to" document.] [Note that in this early development release, not all of the following examples have been tested: this is how things are *supposed* to work, not necessarily how they *do* work. Any deviations of reality from documentation are bugs in one or the other; please report them. Patches are welcome.] Basic syntax and global options ............................... The standard incantation for installing a Distutils-based module distribution (including the Distutils themselves) is python setup.py install More generally, the syntax of the setup script is setup.py [global-options] cmd [cmd-options ...] [cmd [cmd-options] ...] You can always get help with one of the following: ./setup.py --help ./setup.py cmd --help The former lists all global options, the latter lists options for a particular command in addition to the global options. Currently, there are four global options, two of which (-v and -q) are just the opposite of each other: --verbose (-v) run verbosely (default) --quiet (-q) run quietly (turns verbosity off) --dry-run (-n) don't actually do anything --force (-f) skip dependency checking between files In verbose mode (the default), the Distutils will print a line to stdout for every action that affects the filesystem, such as copying a file, compiling a file, creating a directory, etc. The --dry-run (-n) flag is just what it sounds like: "Don't do anything, but tell me what you would have done." In dry-run mode, the meaning of the verbose flag changes subtly: the Distutils will print a line for every action that *would* affect the filesystem if you weren't in dry-run mode. Running quietly in dry-run mode (-n -q) is pointless, as it means "Don't do anything, and don't tell me what you would have done either." Also, some Distutils commands base their action on the state of the filesystem, which is often determined by the action of earlier commands. For example, if you run setup.py -nv build install on a fresh module distribution, you'll get a detailed report of what would happen in the 'build' phase -- but since the effects of the 'build' phase control the 'install' phase, you won't see anything relating to installation, because it doesn't see any files to install. (This might be considered a bug, or perhaps a design flaw.) The Distutils commands ...................... After the global options come a series of one or more commands, each with optional command-specific options. The commands included with this release are: build - build everything build_py - "build" pure Python modules build_ext - build C/C++ extension modules (C++ support untested) install - install everything install_py - install pure Python modules install_ext - install extension modules dist - create source distribution Note that commands may run other commands: the 'build' command runs 'build_py' and then 'build_ext', because that is what "everything" means currently; the 'install' command first runs 'build' (because you can't install what you haven't built), and then 'install_py' and 'install_ext'. Distutils is smart enough not to run the same command twice, and the commands are generally smart enough not to do redundant work in their run. This illustrates why, if all you want to do is install a module distribution, all you have to do is python setup.py install -- the 'install' command runs 'build', which runs 'build_py' and 'build_ext' to build everything, and then installs everything with 'install_py' and 'install_ext'. (Actually, both 'build' and 'install' are smart enough that they only run 'build_py' or 'install_py' if there are pure Python modules in the distribution, and likewise with 'build_ext' and 'install_ext'. That's a minor optimization, though, as it mainly saves the unnecessary import of all the code for that operation -- which, in the case of building extensions, is a fair chunk.) In the absence of command options (described momentarily), the following are all nearly equivalent: setup.py install setup.py build install setup.py build_py build_ext install setup.py build_py build_ext install_py install_ext (The main difference is that when you explicitly specify a command, the module that implements it will be imported -- which can cause a noticeable delay in the case of 'build_ext'.) Command options ............... The actions of every command are controlled by a set of options. Some command options can be specified on the command-line of the setup script; others may only be supplied in the setup script itself. This section only covers the command options that may be supplied by the installer, i.e. on the command-line -- the remaining command options are generally the domain of the module developer, so will be covered below. The vast majority of user-specifiable command options deal with where to put various files. For instance, the 'build*' commands let you specify where to build to, and the 'install*' commands let you specify where to install to. Command option names are only unique within their command, so multiple commands may have (e.g.) a 'build_base' option. All command options have a long name (e.g. 'build_base' in the setup script or '--build-base' on the command line), and many have a one letter form that only appears on the command line. There's no guarantee that the same option name has the same one-letter form (or even the same meaning) across different commands, although this is certainly a desirable goal for command implementors. Before launching into a reference of all options for all commands, we'll see a few examples. First, the 'build_base' option: by default, the Distutils 'build*' commands put ready-to-install modules and extensions into the "build/" subdirectory of the distribution root. (The distribution root is the directory where setup.py exists and is run from, and most files and directories used by the Distutils are relative to this directory. Thus, you can generally read "distribution root" as "current directory".) If you want these files put somewhere else, use the 'build_base' option to 'build': setup.py build --build-base=/tmp/pybuild In this case, pure Python modules will be put in /tmp/pybuild/lib, and extension modules will go in /tmp/pybuild/platlib. If you want exact control over these two directories, you can specify them independently: setup.py build --build-lib=/tmp/pybuild.shared \ --build-platlib=/tmp/pybuild.plat In this case, you don't need to specify 'build_base', since it is only used to generate 'build_lib' and 'build_plat'. Of course, if you then attempt to install the module distribution with "setup.py install", it won't work as intended: 'install' looks in the default build directory ("./build"), but the ready-to-install built files aren't there. So your files will be re-built to "./build" instead of fetched from "/tmp/pybuild" (or wherever). If you run 'install' separately from 'build', you have to tell it the build directories. For instance, the 'install' command to go with the first 'build' command above would be setup.py install --build-base=/tmp/pybuild and to go with the second 'build' command: setup.py install --build-lib=/tmp/pybuild.shared \ --build-platlib=/tmp/pybuild.plat (Note that the great similarity of these commands is due not to some conspiracy within the Distutils, but to the deliberate choice of the same option names for the same purposes across the two commands. Nothing enforces this design principle except common sense!) Obviously, it's preferable to supply the build directory only once, as in: setup.py build --build-base=/tmp/pybuild install or, since 'install' implies 'build', setup.py install --build-base=/tmp/pybuild but note that this: setup.py build install --build-base=/tmp/pybuild DOES NOT WORK -- when you explicitly specify the 'build' command in this way, its options are decided *before* the options for the 'install' command are parsed. (This may be another bug/design flaw that would be nice, but tricky, to fix.) Of course, you may be perfectly happy with the default build directories (heck, I think they're fine), but want to install elsewhere -- e.g. if you don't have superuser privileges on a Unix system, you'll probably want to install to your home directory. Naturally, this is an option -- well, several options really -- to the 'install' command: setup.py install --prefix=/home/greg --exec-prefix=/home/greg will install both pure Python modules and extension modules under /home/greg/lib/python1.5 (assuming you're using Python 1.5, of course). If you want more precise control, you can specify the 'install_site_lib' and 'install_site_platlib' options directly: setup.py install --install-site-lib=/home/greg/lib/python \ --install-site-platlib=/home/greg/lib/python.plat Note that the "lib" and "platlib" directories are completely independent; if you want to control both of them, you must specify both (or both 'prefix' and 'exec_prefix'). I used to think this was a feature, but now it really looks like a bug that will be fixed in Distutils 0.2. The command options that may be supplied by the installer as arguments to the setup script for each command are: build --build-base (-b): base directory to build to [default: "build"] --build-lib (-l): directory to build pure Python modules to [default: 'build_base' + "lib", eg. "build/lib"] --build-platlib (-p): directory to build extension modules to [default: 'build_base' + "platlib", eg. "build/platlib"] build_py --build-dir (-d): directory to build pure Python modules to [default: 'build_lib' option of 'build' command] build_ext --build-dir (-d): directory to build extension modules to [default: 'build_platlib' option of 'build' command] [warning: the following options are not terribly well thought-out or implemented, completely untested, and probably won't work in the current release; patches are welcome] --include-dirs (-I): list of directories to search for header files when compiling C/C++ code; list is delimited by the local path delimiter (colon on Unix, semicolon on DOS/Windows, ...?) --define (-D): C pre-processor macro to define, as "var=value" --undef (-U): C pre-process macros to undefine --libs (-l): external C libraries to link with --library-dirs (-L): directories to search at link-time for external C libraries --rpath (-R): directories to search at run-time for external shared C libraries --link-objects (-O): extra explicit link objects to include in the link (eg. object files, static or shared libraries, compiled resource files, etc.) install --prefix: installation prefix for platform-neutral files [default: Python's sys.prefix] --exec-prefix: installation prefix for platform-specific files [default: Python's sys.exec_prefix] --build-base: base build directory -- where to find all files to install --build-lib: pure Python module build directory -- where to find pure Python modules to install --build-platlib: extension module build directory -- where to find extension modules to install --install-lib: system installation directory for pure Python modules --install-platlib: system installation directory for extension modules --install-site-lib: site-specific installation directory for pure Python modules --install-site-platlib: site-specific installation directory for extension modules --install-scheme: "system" or "site" -- which pair of installation directories to actually use [not currently implemented] --install-path: extra intervening directory (and .pth file) to put between install-site-lib/install-site-platlib and the actual installation --doc-format --install-man --install-html --install-info: [all currently unimplemented and unused, as there is not yet a standard format for Python module documentation] --compile-py --optimize-py: compile .py to .pyc files and/or .pyo files [not really working -- currently always compiles to whatever format the current interpreter outputs] install_py --build-dir: build directory -- where to find pure Python modules to install --install-dir: installation directory -- where to install pure Python modules --compile --optimize: [see --compile-py and --optimize-py under 'install'] install_ext --build-dir: build directory -- where to find extension modules to install --install-dir: installation directory -- where to install extension modules dist [the 'dist' command is really meant for use by developers, but I'm documenting its command-line options here for consistency] --formats: comma-separated list of source distribution formats to generate [currently allowed values: "gztar", "zip"; "tar", "ztar" are planned to be added] [default: "gztar" on Unix, "zip" on DOS/Windows] --manifest (-m): name of manifest file [default: "MANIFEST"] Developer instructions ---------------------- [Eventually, this text should transmute into a "Guidelines for Distributing Python Modules" document.] If you want to learn how to write your own setup script, or what's going on in the examples I've provided, you've come to the right place. First, in order to understand writing setup scripts, you have to understand running them -- so if you skipped straight down here from the top, you should skip right back up again and read the "Installer instructions" section. Right, now that you know about commands and command options, we can reveal the full truth, which is that command options can actually have three sources -- two of them in the setup file. (This ignores where developers extend the Distutils by subclassing command classes, which is not documented yet. It also ignores configuration files, which have been speculated about but remain blue-sky-ware.) Option theory ............. The first option source is, as described above, the command line to the setup script. Whatever happens, this option source is the last word; options supplied here should override all other sources (so that the installer, or developer/distributor running the setup script, has the final word). However, not all options can appear on the command line: of course we allow the installer to specify the build and installation directories, but they don't get to specify the extension modules to build or what package they belong to. That sort of information -- what's included in the module distribution and where it belongs in the space of Python modules -- goes in the setup script. However, I said there are *two* sources of options in the setup script, and to understand why you need to understand the difference between *distribution options* and *command options*. Consider the following setup script fragment: setup (name = "FooBar" version = "1.1", description = "Modules to foo bars and bar foos", py_modules = ['foobar.foo', 'foobar.bar'], options = { 'build': { 'build_base': 'blib' } } ) This illustrates a microcosm of the Distutils option universe. First, (almost) all keyword arguments to 'setup()' are "distribution options": things that wind up as instance attributes of the Distribution object that underlies everything. (Whoops! that's an implementation detail that you don't have to know about. But I thought you might be curious.) A few special arguments to 'setup()' are, well, special and are not treated as distribution options: 'distclass' and 'options'. ('distclass' is intended to allow developers to extend the Distutils by subclassing Distribution and providing the subclass; we'll get to 'options' in a moment.) The other keyword arguments here -- 'name', 'version', 'description', and 'py_modules' -- are all distribution options. 'py_modules' is not a *pure* distribution option, though, as it is hijacked by a particular command ('build_py') to guide its actions in the guise of a command option. 'py_modules' is thus called an *alias option*, because it is a stand-ins for a command options, 'build_py.modules'. (Alias options don't have to have the same name in the distribution as they do in whatever command(s) hijack them; in fact, the various commands that use them don't even have to call them by the same name.) Alias options are needed for convenience (so there's an easy and obvious place to list your packages or pure Pythom modules) and authoritativeness (so that all the commands that care about extension modules have one place to go for that information). Finally, the 'options' argument lets you supply any arbitrary command option. The example shown here makes things look more familiar to Perl refugees (Perl's MakeMaker, the main inspiration for the Distutils, uses "blib" as the name of the "build library" directory). Note that this sort of mischief is anti-social and officially Frowned Upon, but it is possible. Setting options in the 'options' dictionary is deliberately awkward: it should not be needed often; if a particular command option frequently needs to be set for many module distributions, an alias option should be created for it. Also, note that options from the 'options' dictionary generally override alias options (depending on how the particular command is implemented), so before you go setting the list of extension modules or packages this way, you had better know exactly what you are doing and what ramifications doing so will have. Options in practice ................... In case it wasn't clear yet, let me state it explicitly: using the Distutils, whether as a developer, distributor, or installer, boils down to specifying the commands to run and the options to guide them. (At least until we figure out how to extend the Distutils by subclassing Distribution or creating new commands. That has been explicitly allowed for in the design, but never actually tested in practice, so I'll shut up about it now.) Specifying the commands to run is entirely up to the person (or program) that runs the setup script. Since we're concentrating on *writing* the setup script here, let's stick to supplying options -- in particular, options that are relevant in the setup script. The place to start is probably by documenting all pure distribution options: [Miscellaneous options] verbose dry_run: see the descriptions of the '--verbose' and '--dry-run' command-line options above. Yes, this means you could set these in as keyword arguments to 'setup()', but this too is anti-social and Frowned Upon. cmdclass: a dictionary mapping command names to command classes. This is part of the infrastructure for making the Distutils extensible, but has not been tried in practice. [Distribution meta-data options] name version: the name and version number of the module distribution. The name should be brief, descriptive, and unique across all Python module distributions. It is used to build filenames and directory names (and, someday, database records, web pages, etc.), so should not contain any punctuation except *possibly* underscores. The version number should be a reasonably normal-looking version number; there are no strict requirements for what a version number should be, but there will someday be guidelines: see distutils/version.py for details. Both of these are required by the 'dist' command. author author_email identifies the author of the module distribution; if 'author' is given, 'author_email' really should be as well. maintainer maintainer_email identifies the current maintainer of the distribution. If 'author'/'author_email' not given, these two must be. url where to find out about the distribution: generally, this should be a web page with (at the very least) pointers to download the latest version. licence a brief description of the licence under which the distribution is distributed (sorry). Examples: "GPL", "LGPL", "BSD". description a *brief* description of the module distribution itself. If you can't describe your distribution in 40 characters, perhaps it needs to be refactored into more focussed sub-distributions. The alias options will be documented along with the commands that use them. Note that it is possible for a distribution option to be aliased to more than one command, but I don't think this actually happens anywhere in the current code. [Notation note: in the documentation for individual commands, 'bar' refers to option 'bar' in the present command, and 'foo.bar' refers to option 'bar' in command 'foo'.] The `build' command ................... Creates and fills in the "build directories" with ready-to-install files. All the real work is done by other 'build_*' commands, currently 'build_py' and 'build_ext' to build, respectively, pure Python modules and extension modules. Options: build_base: base build directory; it has a subdirectory for every family of related files to install, which are generally "built" by a dedicated 'build_*' command. [default: "build"] build_lib: build directory for pure Python modules [default: build_base + "lib"] Build_platlib: build directory for extension modules [default: build_base + "platlib"] The `build_py' command ...................... "Build" pure Python modules by copying the source files into a build directory. This may seem pointless, but it makes running test scripts, installation, and generation of build distributions a snap. Options: build_dir: directory to build into. Note that only top-level modules will go straight into 'build_dir'; most will wind up in some subdirectory according to the package they belong to. [default: 'build.build_lib'] modules: list of modules to build, specified in fully-qualified package form: ['pkg.mod1', 'pkg.mod2', 'mod3'] selects two modules in package 'pkg' and one top-level module. Does the sensible thing to find source files: 'pkg.mod1' is expected to be in "pkg/mod1.py"; use the 'package_dir' option to override this. Mutually exclusive with 'packages'. [default: distribution option 'py_modules', which is empty by default] [that is, the distribution option 'py_modules' is an alias for 'build_py.modules'] packages: list of packages to build. This selects all pure Python modules from the listed packages, eg. ['pkg', 'pkg.subpkg'] means "install all pure Python modules in the pkg and pkg.subpkg packages". No search for sub-packages is performed. There's no provision for "install everything *except* ..." because there're so many ways to do it and I didn't want to get sucked into that morass yet. Again, does the sensible thing to find modules: in this case, look for "pkg/*.py" and "pkg/subpkg/*.py"; and again, you can use 'package_dir' to override this. Mutually exclusive with 'modules'. [default: distribution option 'packages', is empty by default] package_dir: dictionary mapping package name to directories where the Python source files comprising the package can be found. Useful when your source files are not laid out according to the plan outlined above. For instance, if your modules are all in the 'foo' package, but the sources are right in the distribution directory, ie. module 'foo.bar' is in "./bar.py", then 'package_dir' would be {'foo': ''}. If those same source files are down one level in "src", eg. "src/bar.py", then supply {'foo': 'src'}. If they're in "src/foo", make it {'foo': 'src/foo'}. Does the right thing with sub-packages, ie. as long as package 'foo.bar' is in a subdirectory of package 'foo', you don't have to specify 'foo.bar' in 'package_dir'. [default: distribution option 'package_dir', which is empty by default] The `build_ext' command ....................... Builds extension modules by compiling the C/C++ source files that comprise them, and linking them together into shared object files (.so on most Unices, .dll on DOS/Windows). (Currently there's no provision for creating a new, statically linked Python binary.) Options: build_dir: directory to build into. Like with the 'build_py' command, only top-level extension modules get dropped right into 'build_dir'; most extensions will wind up in a directory determined by their package. [default: 'build.build_platlib'] include_dirs: list of directories to search for C/C++ header files. [default: distribution option 'include_dirs'] define: list of C preprocessor macros to define, as (name,value) tuples. A 'value' of None means "no specified value", eg. ('FOO',None) is like putting "#define FOO" in your source, or "-DFOO" on the traditional Unix C compiler command line. [default: distribution option 'define'] undef: list of C preprocessor macros to undefine. Eg. including 'FOO' in the list is like putting "#undef FOO" in the source, or "-UFOO" on the command line. libs: list of libraries to link with. These are just bare library names, eg. "foo" stands for "libfoo.a" on Unix and "foo.lib" on DOS/Windows. Currently there's no provision for forcing a library to be found in a particular directory, although there should be to support Distutils for use as a developer-only build system. library_dirs: list of directories to search for libraries at link-time. rpath: list of directories to search for shared libraries at run-time. link_objects: list of extra filenames to include in the link; could include explicitly specified library files ('/funky/devel/version/libfoo.a'), or compiled resource files, or other stuff that doesn't come from a C/C++ source file. The `install' command ..................... Install everything in the build directories to a sensible place. Currently this just means pure Python modules and extension modules, since that's all we know how to build so far. In general, every build directory (ie. every subdirectory of 'build.build_base') has its own installation location, which is specified by its own 'install' option -- and the installation is taken care of by one of the 'install_*' commands. Almost all of the options used by the 'install' command are installation directories of some sort. No one will have to set *all* of these -- the reason there are so many is to give installers flexibility to customize their installation directories at however fine a grain they wish. Note that many of the default values are taken from the 'Makefile.pre.in' installed with Python -- these are what I mean by "Makefile value" in the below text. To illustrate, I will give the default values for many options on a Unix system with prefix "/usr/local" and exec-prefix "/usr/local.plat". Options: build_base: where to find the build directories [default: 'build.build_base'] build_lib: where to find pure Python modules to install [default: 'build.build_lib'] build_platlib: where to find extension modules to install [default: 'build.build_platlib'] prefix exec_prefix: the installation prefix and exec-prefix, i.e. the default starting directory for, respectively, platform-neutral and platform-specific files. Only used to build default values for other directories. [default: Python's 'sys.prefix' and 'sys.exec_prefix'] doc_format: unused; will be a list of documentation formats to install once we have a standard documentation format for Python modules. install_lib: base installation directory for pure Python modules that are part of the standard library [currently unused] [default: 'prefix' + the post-prefix part of Makefile value LIBDEST, eg. "/usr/local/python/lib/python1.5"] install_platlib: base installation directory for extension modules that are part of the standard library [currently unused] [default: 'exec_prefix' + the post-prefix part of Makefile BINLIBDEST, eg. "/usr/local.plat/python/lib/python1.5"] install_site_lib: base installation directory for third-party pure Python modules [Unix default: 'install_lib' + "site-packages", eg. "/usr/local/lib/python1.5/site-packages"] [DOS/Windows default: 'prefix', eg. "C:\Python"] [actually it gets more complicated in the presence of the 'install_path' option; see below] install_site_platlib: base installation directory for third-party extension modules [Unix default: 'install_platlib' + "site-packages", eg. "/usr/local.plat/lib/python1.5/site-packages"] [DOS/Windows default: 'exec_prefix', eg. "C:\Python"] install_path: information about extra intervening directories to put between 'install_lib' and 'install_sitelib', along with a .pth file to ensure that those directories wind up in sys.path. Can be a 1- or 2-tuple, or a comma-delimited string with 1 or 2 parts. The 1-element case is simpler: the .pth file and directory have the same name (except for ".pth"). Eg. if install_path is "foo" or ("foo",), then Distutils sets 'install_site_lib' to 'install_lib' + "site-packages/foo", and puts foo.path in the "site-packages" directory; it contains just "foo". The 2-element case allows the .pth file and intervening directories to be named differently; eg. if 'install_path' is ("foo","foo/bar/baz") (or "foo,foo/bar/baz"), then Distutils will set 'install_site_lib' to 'install_lib' + "site-packages/foo/bar/baz", and put "foo.pth" containing "foo/bar/baz" in the "site-packages" directory. install_man install_html install_info: unused. compile_py optimize_py: sort of used, but not really fully supported. The `install_py' command ........................ Installs pure Python modules by copying the entire 'build_lib' build directory to the appropriate installation destination. Both the build and install directory may be specified as 'install_py' options, but it usually makes more sense to let them take their default values from 'install'. Options: build_dir: where to find pure Python modules to install [default: 'install.build_lib'] install_dir: installation directory; only top-level modules will actually wind up in 'install_dir', since the build process took care of putting packagized modules into their package directories under 'build_dir'. [default: 'install.install_site_lib' if the module distribution consists entirely of pure Python modules, or 'install.install_site_platlib' if there are any extension modules in the distribution] compile_py optimize_py: sort of used, but not really fully supported. [default: same-named options in 'install'] The `install_ext' command ......................... Installs extension modules by copying the entire 'build_platlib' build directory to the appropriate installation destination. Again, both of these can be set as 'install_ext' options, but it's usually better to let them take their defaults here. Options: build_dir: where to find extension modules to install [default: 'install.build_platlib'] install_dir: where to install them; again, thanks to the build process, extensions will actually wind up in a package directory somewhere under 'install_dir' [default: 'install.install_site_platlib'] The `dist' command ................... Creates a source distribution. By default, Distutils includes just enough files for the distribution to be usable: all source files mentioned or implied by the 'packages', 'py_modules', and 'ext_modules' distribution options, plus the setup script "setup.py" and a README file. It also includes anything that looks like a test script ("test/test*.py"), although there's not yet a framework for running test scripts and reporting the results. Issues warnings if any essential pieces are missing, such as distribution meta-data in the setup script (eg. the 'name', 'version', 'url' distribution options), the setup script itself, or the README file. The 'dist' command supports generating multiple archive formats in the same run; currently, only "gztar" and "zip" are supported. The name of the source distribution is built from the 'name' and 'version' distribution options plus the archive format. For example: setup (name = "foobar", version = "1.23") and setup.py dist -f gztar will result in a source distribution called "foobar-1.23.tar.gz". You don't have to specify a format; Distutils will pick a default for your platform (currently gztar on Unix, zip on DOS/Windows). You don't get to pick the name of your source distribution (this is a feature). If the defaults aren't enough for you (are they ever?), you can specify additional files to include in (or exclude from) the source distribution in a manifest file (called "MANIFEST" by default). As each line in the manifest is processed, it adds to or deletes from the list of files that will be distributed. There are three kinds of manifest lines: * simple include pattern a Unix-style wildcard (see the standard "glob" module for details) that specifies a set of files to include in one directory, eg. "examples/*.py" * simple exclude pattern a Unix-style wildcard prefixed with a bang (!) that specifies a set of files to exclude from one directory, eg. "! src/hackedup_*.py" * directory pattern a directory name, followed by a list of include and exclude patterns which apply recursively within that directory Before processing the manifest file, Distutils already has a set of default files to distribute: sources for all your pure Python and extension modules, plus the setup script, README, and any test scripts. Processing the manifest can add to or remove from this list. When a simple include pattern is encountered, Distutils adds all matching files to its file list. When a simple exclude pattern is encountered, Distutils removes all files from the list that match it. This means that an exclude pattern line can only affect what comes before it, namely the default fileset and manifest lines that precede the exclude line. For example, consider the "foobar" distribution, which distributes modules in the "foo" and "foo.bar" packages: setup (name = "foobar", version = "1.23", # ... packages = ['foo', 'foo.bar']) If the source tree looks like this: setup.py README foo/ ------+ mod1.py | mod2.py | hackedup_mod2.py + bar/ ------ mod3.py examples/ -+ README | example1.py + example2.py Then the default fileset will be: foo/mod1.py foo/mod2.py foo/hackedup_mod2.py foo/bar/mod3.py setup.py README You probably want to exclude "hackedup_mod2.py", and add the examples directory. This is easily done with the following manifest: ! foo/hackedup* examples/* Of course, this will gobble up everything in "examples" -- you might want to exclude .pyc files, Emacs turds, etc. You could do this with separate exclude lines: ! examples/*.pyc ! examples/*~ but that could get tedious pretty quickly; it would be especially tiresome if you want to do it on a whole directory tree. (What happens when you have so many examples you want to split the "examples" directory up?) The solution is to use a directory pattern, which in its simplest form is this: examples * meaning "Recursively gather up all files under examples". Again, this will get backups, .pyc files, etc. -- so you probably want either this: examples *.py README or this examples * !*~ !*.pyc depending on whether you subscribe to the Prussian ("forbid everything not explicitly allowed") or American ("allow everything not explicitly forbidden") viewpoint. Your call. Incidentally, a directory name on its own on a manifest line is *not* a simple include pattern, it is a directory pattern -- eg. in this example examples is shorthand for examples * In fact, any directory pattern that doesn't start with an include pattern implicitly starts with "*", because there's not much point excluding files from nothing. That is, examples !*~ !*.pyc is shorthand for examples * !*~ !*.pyc The current implementation of the 'dist' command is very Unix-biased, even Linux-biased (hey, I needed it in a hurry). It uses 'os.link()' to generate the directory tree of files to distribute, and thus will only work on Unix. Furthermore, it assumes that "tar" is GNU tar and supports the "-z" option, which limits its usefulness for non-Linux Unices. To generate ZIP files, it assumes the existence of a "zip" on the path that is the usual Unix "zip" utility -- ie. it doesn't know about PKZIP for DOS/Windows. (Does anyone still use PKZIP?) These shortcomings will be addressed in the next version. Patches are welcome. Options: formats: list of strings, or comma-delimitd string, naming all the formats to generate (currently limited to "gztar" and "zip") [default: "gztar" on Unix, "zip" on DOS/Windows] manifest: name of the manifest file that describes additional files, on top of the defaults, to include in/exclude from the source distribution [default: "MANIFEST"] Well, that's all for now: happy module distributing! $Id$