Buildouts

=========

The word "buildout" refers to a description of a set of parts and the

software to create and assemble them.  It is often used informally to

refer to an installed system based on a buildout definition.  For

example, if we are creating an application named "Foo", then "the Foo

buildout" is the collection of configuration and application-specific

software that allows an instance of the application to be created.  We

may refer to such an instance of the application informally as "a Foo

buildout".

This document describes how to define buildouts using buildout

configuration files and recipes.  There are three ways to set up the

buildout software and create a buildout instance:

1. Install the zc.buildout egg with easy_install and use the buildout

   script installed in a Python scripts area.

2. Use the buildout bootstrap script to create a buildout that

   includes both the setuptools and zc.buildout eggs.  This allows you

   to use the buildout software without modifying a Python install.

   The buildout script is installed into your buildout local scripts

   area.

3. Use a buildout command from an already installed buildout to

   bootstrap a new buildout.  (See the section on bootstraping later

   in this document.)

Often, a software project will be managed in a software repository,

such as a subversion repository, that includes some software source

directories, buildout configuration files, and a copy of the buildout

bootstrap script.  To work on the project, one would check out the

project from the repository and run the bootstrap script which

installs setuptools and zc.buildout into the checkout as well as any

parts defined.

We have a sample buildout that we created using the bootstrap command

of an existing buildout (method 3 above).  It has the absolute minimum

information.  We have bin, develop-eggs, eggs and parts directories,

and a configuration file:

    >>> ls(sample_buildout)

    d  bin

    -  buildout.cfg

    d  develop-eggs

    d  eggs

    d  parts

The bin directory contains scripts.

    >>> ls(sample_buildout, 'bin')

    -  buildout

    >>> ls(sample_buildout, 'eggs')

    -  setuptools-0.6-py2.4.egg

    -  zc.buildout-1.0-py2.4.egg

The develop-eggs directory is initially empty:

    >>> ls(sample_buildout, 'develop-eggs')

The develop-eggs directory holds egg links for software being

developed in the buildout.  We separate develop-eggs and other eggs to

allow eggs directories to be shared across multiple buildouts.  For

example, a common developer technique is to define a common eggs

directory in their home that all non-develop eggs are stored in.  This

allows larger buildouts to be set up much more quickly and saves disk

space.

The parts directory just contains some helpers for the buildout script

itself.

    >>> ls(sample_buildout, 'parts')

    d  buildout

The parts directory provides an area where recipes can install

part data.  For example, if we built a custom Python, we would

install it in the part directory.  Part data is stored in a

sub-directory of the parts directory with the same name as the part.

Buildouts are defined using configuration files.  These are in the

format defined by the Python ConfigParser module, with extensions

that we'll describe later.  By default, when a buildout is run, it

looks for the file buildout.cfg in the directory where the buildout is

run.

The minimal configuration file has a buildout section that defines no

parts:

    >>> cat(sample_buildout, 'buildout.cfg')

    [buildout]

    parts =

A part is simply something to be created by a buildout.  It can be

almost anything, such as a Python package, a program, a directory, or

even a configuration file.

Recipes

-------

A part is created by a recipe.  Recipes are always installed as Python

eggs. They can be downloaded from a package server, such as the

Python Package Index, or they can be developed as part of a project

using a "develop" egg.

A develop egg is a special kind of egg that gets installed as an "egg

link" that contains the name of a source directory.  Develop eggs

don't have to be packaged for distribution to be used and can be

modified in place, which is especially useful while they are being

developed.

Let's create a recipe as part of the sample project.  We'll create a

recipe for creating directories.  First, we'll create a recipes source

directory for our local recipes:

    >>> mkdir(sample_buildout, 'recipes')

and then we'll create a source file for our mkdir recipe:

    >>> write(sample_buildout, 'recipes', 'mkdir.py',

    ... """

    ... import logging, os, zc.buildout

    ...

    ... class Mkdir:

    ...

    ...     def __init__(self, buildout, name, options):

    ...         self.name, self.options = name, options

    ...         options['path'] = os.path.join(

    ...                               buildout['buildout']['directory'],

    ...                               options['path'],

    ...                               )

    ...         if not os.path.isdir(os.path.dirname(options['path'])):

    ...             logging.getLogger(self.name).error(

    ...                 'Cannot create %s. %s is not a directory.',

    ...                 options['path'], os.path.dirname(options['path']))

    ...             raise zc.buildout.UserError('Invalid Path')

    ...

    ...

    ...     def install(self):

    ...         path = self.options['path']

    ...         logging.getLogger(self.name).info(

    ...             'Creating directory %s', os.path.basename(path))

    ...         os.mkdir(path)

    ...         return path

    ...

    ...     def update(self):

    ...         pass

    ... """)

Currently, recipes must define 3 methods [#future_recipe_methods]_:

- a constructor,

- an install method, and

- an update method.

The constructor is responsible for updating a parts options to reflect

data read from other sections.  The buildout system keeps track of

whether a part specification has changed.  A part specification has

changed if it's options, after adjusting for data read from other

sections, has changed, or if the recipe has changed.  Only the options

for the part are considered.  If data are read from other sections,

then that information has to be reflected in the parts options.  In

the Mkdir example, the given path is interpreted relative to the

buildout directory, and data from the buildout directory is read.  The

path option is updated to reflect this.  If the directory option was

changed in the buildout sections, we would know to update parts

created using the mkdir recipe using relative path names.

When buildout is run, it saves configuration data for installed parts

in a file named ".installed.cfg".  In subsequent runs, it compares

part-configuration data stored in the .installed.cfg file and the

part-configuration data loaded from the configuration files as

modified by recipe constructors to decide if the configuration of a

part has changed. If the configuration has changed, or if the recipe

has changed, then the part is uninstalled and reinstalled.  The

buildout only looks at the part's options, so any data used to

configure the part needs to be reflected in the part's options.  It is

the job of a recipe constructor to make sure that the options include

all relevant data.

Of course, parts are also uninstalled if they are no-longer used.

The recipe defines a constructor that takes a buildout object, a part

name, and an options dictionary. It saves them in instance attributes.

If the path is relative, we'll interpret it as relative to the

buildout directory.  The buildout object passed in is a mapping from

section name to a mapping of options for that section. The buildout

directory is available as the directory option of the buildout

section.  We normalize the path and save it back into the options

directory.

The install method is responsible for creating the part.  In this

case, we need the path of the directory to create.  We'll use a path

option from our options dictionary.  The install method logs what it's

doing using the Python logging call.  We return the path that we

installed.  If the part is uninstalled or reinstalled, then the path

returned will be removed by the buildout machinery.  A recipe install

method is expected to return a string, or an iterable of strings

containing paths to be removed if a part is uninstalled.  For most

recipes, this is all of the uninstall support needed. For more complex

uninstallation scenarios use `Uninstall recipes`_.

The update method is responsible for updating an already installed

part.  An empty method is often provided, as in this example, if parts

can't be updated.  An update method can return None, a string, or an

iterable of strings.  If a string or iterable of strings is returned,

then the saved list of paths to be uninstalled is updated with the new

information by adding any new files returned by the update method.

We need to provide packaging information so that our recipe can be

installed as a develop egg. The minimum information we need to specify

[#packaging_info]_ is a name.  For recipes, we also need to define the

names of the recipe classes as entry points.  Packaging information is

provided via a setup.py script:

    >>> write(sample_buildout, 'recipes', 'setup.py',

    ... """

    ... from setuptools import setup

    ...

    ... setup(

    ...     name = "recipes",

    ...     entry_points = {'zc.buildout': ['mkdir = mkdir:Mkdir']},

    ...     )

    ... """)

Our setup script defines an entry point. Entry points provide

a way for an egg to define the services it provides.  Here we've said

that we define a zc.buildout entry point named mkdir.  Recipe

classes must be exposed as entry points in the zc.buildout group.  we

give entry points names within the group.

We also need a README.txt for our recipes to avoid an annoying warning

from distutils, on which setuptools and zc.buildout are based:

    >>> write(sample_buildout, 'recipes', 'README.txt', " ")

Now let's update our buildout.cfg:

    >>> write(sample_buildout, 'buildout.cfg',

    ... """

    ... [buildout]

    ... develop = recipes

    ... parts = data-dir

    ...

    ... [data-dir]

    ... recipe = recipes:mkdir

    ... path = mystuff

    ... """)

Let's go through the changes one by one::

    develop = recipes

This tells the buildout to install a development egg for our recipes.

Any number of paths can be listed.  The paths can be relative or

absolute.  If relative, they are treated as relative to the buildout

directory.  They can be directory or file paths.  If a file path is

given, it should point to a Python setup script.  If a directory path

is given, it should point to a directory containing a setup.py file.

Development eggs are installed before building any parts, as they may

provide locally-defined recipes needed by the parts.

::

    parts = data-dir

Here we've named a part to be "built".  We can use any name we want

except that different part names must be unique and recipes will often

use the part name to decide what to do.

::

    [data-dir]

    recipe = recipes:mkdir

    path = mystuff

When we name a part, we also create a section of the same

name that contains part data.  In this section, we'll define

the recipe to be used to install the part.  In this case, we also

specify the path to be created.

Let's run the buildout.  We do so by running the build script in the

buildout:

    >>> import os

    >>> os.chdir(sample_buildout)

    >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout')

    >>> print system(buildout),

    Develop: '/sample-buildout/recipes'

    Installing data-dir.

    data-dir: Creating directory mystuff

We see that the recipe created the directory, as expected:

    >>> ls(sample_buildout)

    -  .installed.cfg

    d  bin

    -  buildout.cfg

    d  develop-eggs

    d  eggs

    d  mystuff

    d  parts

    d  recipes

In addition, .installed.cfg has been created containing information

about the part we installed:

    >>> cat(sample_buildout, '.installed.cfg')

    [buildout]

    installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link

    parts = data-dir

    <BLANKLINE>

    [data-dir]

    __buildout_installed__ = /sample-buildout/mystuff

    __buildout_signature__ = recipes-c7vHV6ekIDUPy/7fjAaYjg==

    path = /sample-buildout/mystuff

    recipe = recipes:mkdir

Note that the directory we installed is included in .installed.cfg.

In addition, the path option includes the actual destination

directory.

If we change the name of the directory in the configuration file,

we'll see that the directory gets removed and recreated:

    >>> write(sample_buildout, 'buildout.cfg',

    ... """

    ... [buildout]

    ... develop = recipes

    ... parts = data-dir

    ...

    ... [data-dir]

    ... recipe = recipes:mkdir

    ... path = mydata

    ... """)

    >>> print system(buildout),

    Develop: '/sample-buildout/recipes'

    Uninstalling data-dir.

    Installing data-dir.

    data-dir: Creating directory mydata

    >>> ls(sample_buildout)

    -  .installed.cfg

    d  bin

    -  buildout.cfg

    d  develop-eggs

    d  eggs

    d  mydata

    d  parts

    d  recipes

If any of the files or directories created by a recipe are removed,

the part will be reinstalled:

    >>> rmdir(sample_buildout, 'mydata')

    >>> print system(buildout),

    Develop: '/sample-buildout/recipes'

    Uninstalling data-dir.

    Installing data-dir.

    data-dir: Creating directory mydata

Error reporting

---------------

If a user makes an error, an error needs to be printed and work needs

to stop.  This is accomplished by logging a detailed error message and

then raising a (or an instance of a subclass of a)

zc.buildout.UserError exception.  Raising an error other than a

UserError still displays the error, but labels it as a bug in the

buildout software or recipe. In the sample above, of someone gives a

non-existent directory to create the directory in:

    >>> write(sample_buildout, 'buildout.cfg',

    ... """

    ... [buildout]

    ... develop = recipes

    ... parts = data-dir

    ...

    ... [data-dir]

    ... recipe = recipes:mkdir

    ... path = /xxx/mydata

    ... """)

We'll get a user error, not a traceback.

    >>> print system(buildout),

    Develop: '/sample-buildout/recipes'

    data-dir: Cannot create /xxx/mydata. /xxx is not a directory.

    While:

      Installing.

      Getting section data-dir.

      Initializing part data-dir.

    Error: Invalid Path

Recipe Error Handling

---------------------

If an error occurs during installation, it is up to the recipe to

clean up any system side effects, such as files created.  Let's update

the mkdir recipe to support multiple paths:

    >>> write(sample_buildout, 'recipes', 'mkdir.py',

    ... """

    ... import logging, os, zc.buildout

    ...

    ... class Mkdir:

    ...

    ...     def __init__(self, buildout, name, options):

    ...         self.name, self.options = name, options

    ...

    ...         # Normalize paths and check that their parent

    ...         # directories exist:

    ...         paths = []

    ...         for path in options['path'].split():

    ...             path = os.path.join(buildout['buildout']['directory'], path)

    ...             if not os.path.isdir(os.path.dirname(path)):

    ...                 logging.getLogger(self.name).error(

    ...                     'Cannot create %s. %s is not a directory.',

    ...                     options['path'], os.path.dirname(options['path']))

    ...                 raise zc.buildout.UserError('Invalid Path')

    ...             paths.append(path)

    ...         options['path'] = ' '.join(paths)

    ...

    ...     def install(self):

    ...         paths = self.options['path'].split()

    ...         for path in paths:

    ...             logging.getLogger(self.name).info(

    ...                 'Creating directory %s', os.path.basename(path))

    ...             os.mkdir(path)

    ...         return paths

    ...

    ...     def update(self):

    ...         pass

    ... """)

If there is an error creating a path, the install method will exit and

leave previously created paths in place:

    >>> write(sample_buildout, 'buildout.cfg',

    ... """

    ... [buildout]

    ... develop = recipes

    ... parts = data-dir

    ...

    ... [data-dir]

    ... recipe = recipes:mkdir

    ... path = foo bin

    ... """)

    >>> print system(buildout), # doctest: +ELLIPSIS

    Develop: '/sample-buildout/recipes'

    Uninstalling data-dir.

    Installing data-dir.

    data-dir: Creating directory foo

    data-dir: Creating directory bin

    While:

      Installing data-dir.

    <BLANKLINE>

    An internal error occurred due to a bug in either zc.buildout or in a

    recipe being used:

    Traceback (most recent call last):

      ...

    OSError: [Errno 17] File exists: '/sample-buildout/bin'

We meant to create a directory bins, but typed bin.  Now foo was

left behind.

    >>> os.path.exists('foo')

    True

If we fix the typo:

    >>> write(sample_buildout, 'buildout.cfg',

    ... """

    ... [buildout]

    ... develop = recipes

    ... parts = data-dir

    ...

    ... [data-dir]

    ... recipe = recipes:mkdir

    ... path = foo bins

    ... """)

    >>> print system(buildout), # doctest: +ELLIPSIS

    Develop: '/sample-buildout/recipes'

    Installing data-dir.

    data-dir: Creating directory foo

    While:

      Installing data-dir.

    <BLANKLINE>

    An internal error occurred due to a bug in either zc.buildout or in a

    recipe being used:

    Traceback (most recent call last):

    ...

    OSError: [Errno 17] File exists: '/sample-buildout/foo'

Now they fail because foo exists, because it was left behind.

    >>> remove('foo')

Let's fix the recipe:

    >>> write(sample_buildout, 'recipes', 'mkdir.py',

    ... """

    ... import logging, os, zc.buildout

    ...

    ... class Mkdir:

    ...

    ...     def __init__(self, buildout, name, options):

    ...         self.name, self.options = name, options

    ...

    ...         # Normalize paths and check that their parent

    ...         # directories exist:

    ...         paths = []

    ...         for path in options['path'].split():

    ...             path = os.path.join(buildout['buildout']['directory'], path)

    ...             if not os.path.isdir(os.path.dirname(path)):

    ...                 logging.getLogger(self.name).error(

    ...                     'Cannot create %s. %s is not a directory.',

    ...                     options['path'], os.path.dirname(options['path']))

    ...                 raise zc.buildout.UserError('Invalid Path')

    ...             paths.append(path)