1 Metadata-Version: 1.0

     2 Name: zc.buildout

     3 Version: 1.5.2

     4 Summary: System for managing development buildouts

     5 Home-page: http://pypi.python.org/pypi/zc.buildout

     6 Author: Jim Fulton

     7 Author-email: jim@zope.com

     8 License: ZPL 2.1

     9 Description: ********

    10         Buildout

    11         ********

    12         

    13         .. contents::

    14         

    15         The Buildout project provides support for creating applications,

    16         especially Python applications.  It provides tools for assembling

    17         applications from multiple parts, Python or otherwise.  An application

    18         may actually contain multiple programs, processes, and configuration

    19         settings.

    20         

    21         The word "buildout" refers to a description of a set of parts and the

    22         software to create and assemble them.  It is often used informally to

    23         refer to an installed system based on a buildout definition.  For

    24         example, if we are creating an application named "Foo", then "the Foo

    25         buildout" is the collection of configuration and application-specific

    26         software that allows an instance of the application to be created.  We

    27         may refer to such an instance of the application informally as "a Foo

    28         buildout".

    29         

    30         To get a feel for some of the things you might use buildouts for, see

    31         the `Buildout examples`_.

    32         

    33         To lean more about using buildouts, see `Detailed Documentation`_.

    34         

    35         To see screencasts, talks, useful links and more documentation, visit

    36         the `Buildout website <http://www.buildout.org>`_.

    37         

    38         Recipes

    39         *******

    40         

    41         Existing recipes include:

    42         

    43         `zc.recipe.egg <http://pypi.python.org/pypi/zc.recipe.egg>`_

    44            The egg recipe installes one or more eggs, with their

    45            dependencies.  It installs their console-script entry points with

    46            the needed eggs included in their paths.  It is suitable for use with

    47            a "clean" Python: one without packages installed in site-packages.

    48         

    49         `z3c.recipe.scripts <http://pypi.python.org/pypi/z3c.recipe.scripts>`_

    50           Like zc.recipe.egg, this recipe builds interpreter scripts and entry

    51           point scripts based on eggs.  It can be used with a Python that has

    52           packages installed in site-packages, such as a system Python.  The

    53           interpreter also has more features than the one offered by

    54           zc.recipe.egg.

    55         

    56         `zc.recipe.testrunner <http://pypi.python.org/pypi/zc.recipe.testrunner>`_

    57            The testrunner egg creates a test runner script for one or

    58            more eggs.

    59         

    60         `zc.recipe.zope3checkout <http://pypi.python.org/pypi/zc.recipe.zope3checkout>`_

    61            The zope3checkout recipe installs a Zope 3 checkout into a

    62            buildout.

    63         

    64         `zc.recipe.zope3instance <http://pypi.python.org/pypi/zc.recipe.zope3instance>`_

    65            The zope3instance recipe sets up a Zope 3 instance.

    66         

    67         `zc.recipe.filestorage <http://pypi.python.org/pypi/zc.recipe.filestorage>`_

    68            The filestorage recipe sets up a ZODB file storage for use in a

    69            Zope 3 instance created by the zope3instance recipe.

    70         

    71         Buildout examples

    72         *****************

    73         

    74         Here are a few examples of what you can do with buildouts.  We'll

    75         present these as a set of use cases.

    76         

    77         Try out an egg

    78         ==============

    79         

    80         Sometimes you want to try an egg (or eggs) that someone has released.

    81         You'd like to get a Python interpreter that lets you try things

    82         interactively or run sample scripts without having to do path

    83         manipulations.  If you can and don't mind modifying your Python

    84         installation, you could use easy_install, otherwise, you could create

    85         a directory somewhere and create a buildout.cfg file in that directory

    86         containing::

    87         

    88           [buildout]

    89           parts = mypython

    90         

    91           [mypython]

    92           recipe = zc.recipe.egg

    93           interpreter = mypython

    94           eggs = theegg

    95         

    96         where theegg is the name of the egg you want to try out.

    97         

    98         Run buildout in this directory.  It will create a bin subdirectory

    99         that includes a mypython script.  If you run mypython without any

   100         arguments you'll get an interactive interpreter with the egg in the

   101         path. If you run it with a script and script arguments, the script

   102         will run with the egg in its path.  Of course, you can specify as many

   103         eggs as you want in the eggs option.

   104         

   105         If the egg provides any scripts (console_scripts entry points), those

   106         will be installed in your bin directory too.

   107         

   108         Work on a package

   109         =================

   110         

   111         I often work on packages that are managed separately.  They don't have

   112         scripts to be installed, but I want to be able to run their tests

   113         using the `zope.testing test runner

   114         <http://www.python.org/pypi/zope.testing>`_.  In this kind of

   115         application, the program to be installed is the test runner.  A good

   116         example of this is `zc.ngi <http://svn.zope.org/zc.ngi/trunk/>`_.

   117         

   118         Here I have a subversion project for the zc.ngi package.  The software

   119         is in the src directory.  The configuration file is very simple::

   120         

   121           [buildout]

   122           develop = .

   123           parts = test

   124         

   125           [test]

   126           recipe = zc.recipe.testrunner

   127           eggs = zc.ngi

   128         

   129         I use the develop option to create a develop egg based on the current

   130         directory.  I request a test script named "test" using the

   131         zc.recipe.testrunner recipe.  In the section for the test script, I

   132         specify that I want to run the tests in the zc.ngi package.

   133         

   134         When I check out this project into a new sandbox, I run bootstrap.py

   135         to get setuptools and zc.buildout and to create bin/buildout.  I run

   136         bin/buildout, which installs the test script, bin/test, which I can

   137         then use to run the tests.

   138         

   139         This is probably the most common type of buildout.

   140         

   141         If I need to run a previous version of zc.buildout, I use the

   142         `--version` option of the bootstrap.py script::

   143         

   144             $ python bootstrap.py --version 1.1.3

   145         

   146         The `zc.buildout project <http://svn.zope.org/zc.buildout/trunk>`_

   147         is a slightly more complex example of this type of buildout.

   148         

   149         Install egg-based scripts

   150         =========================

   151         

   152         A variation of the `Try out an egg`_ use case is to install scripts

   153         into your ~/bin directory (on Unix, of course).  My ~/bin directory is

   154         a buildout with a configuration file that looks like::

   155         

   156         

   157           [buildout]

   158           parts = foo bar

   159           bin-directory = .

   160         

   161           [foo]

   162           ...

   163         

   164         where foo and bar are packages with scripts that I want available.  As

   165         I need new scripts, I can add additional sections.  The bin-directory

   166         option specified that scripts should be installed into the current

   167         directory.

   168         

   169         Multi-program multi-machine systems

   170         ===================================

   171         

   172         Using an older prototype version of the buildout, we've build a number

   173         of systems involving multiple programs, databases, and machines.  One

   174         typical example consists of:

   175         

   176         - Multiple Zope instances

   177         

   178         - Multiple ZEO servers

   179         

   180         - An LDAP server

   181         

   182         - Cache-invalidation and Mail delivery servers

   183         

   184         - Dozens of add-on packages

   185         

   186         - Multiple test runners

   187         

   188         - Multiple deployment modes, including dev, stage, and prod,

   189           with prod deployment over multiple servers

   190         

   191         Parts installed include:

   192         

   193         - Application software installs, including Zope, ZEO and LDAP

   194           software

   195         

   196         - Add-on packages

   197         

   198         - Bundles of configuration that define Zope, ZEO and LDAP instances

   199         

   200         - Utility scripts such as test runners, server-control

   201           scripts, cron jobs.

   202         

   203         Questions and Bug Reporting

   204         ***************************

   205         

   206         Please send questions and comments to the

   207         `distutils SIG mailing list <mailto://distutils-sig@python.org>`_.

   208         

   209         Report bugs using the `zc.buildout Launchpad Bug Tracker

   210         <https://launchpad.net/zc.buildout/+bugs>`_.

   211         

   212         System Python and zc.buildout 1.5

   213         *********************************

   214         

   215         The 1.5 line of zc.buildout introduced a number of changes.

   216         

   217         Problems

   218         ========

   219         

   220         As usual, please send questions and comments to the `distutils SIG

   221         mailing list <mailto://distutils-sig@python.org>`_. Report bugs using

   222         the `zc.buildout Launchpad Bug Tracker

   223         <https://launchpad.net/zc.buildout/+bugs>`_.

   224         

   225         If problems are keeping you from your work, here's an easy way to

   226         revert to the old code temporarily: switch to a custom "emergency"

   227         bootstrap script, available from

   228         http://svn.zope.org/repos/main/zc.buildout/branches/1.4/bootstrap/bootstrap.py .

   229         

   230         This customized script will select zc.buildout 1.4.4 by default.

   231         zc.buildout 1.4.4 will not upgrade itself unless you explicitly specify

   232         a new version.  It will also prefer older versions of zc.recipe.egg and

   233         some other common recipes.  If you have trouble with other recipes,

   234         consider using a standard buildout "versions" section to specify older

   235         versions of these, as described in the Buildout documentation

   236         (http://pypi.python.org/pypi/zc.buildout#repeatable-buildouts-controlling-eggs-used).

   237         

   238         Working with a System Python

   239         ============================

   240         

   241         While there are a number of new features available in zc.buildout 1.5,

   242         the biggest is that Buildout itself supports usage with a system Python.

   243         This can work if you follow a couple of simple rules.

   244         

   245         1. Use the new bootstrap.py (available from

   246            svn://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap/bootstrap.py).

   247         

   248         2. Use buildout recipes that have been upgraded to work with zc.buildout 1.5

   249            and higher.  Specifically, they should use

   250            ``zc.buildout.easy_install.sitepackage_safe_scripts`` to generate

   251            their scripts, if any, rather than ``zc.buildout.easy_install.scripts``.

   252            See the `Recipes That Support a System Python`_ section below for more

   253            details on recipes that are available as of this writing, and

   254            `Updating Recipes to Support a System Python`_ for instructions on

   255            how to update a recipe.  Note that you should generally only need to

   256            update recipes that generate scripts.

   257         

   258         You can then use ``include-site-packages = false`` and

   259         ``exec-sitecustomize = false`` buildout options to eliminate access to

   260         your Python's site packages and not execute its sitecustomize file, if

   261         it exists, respectively.

   262         

   263         Alternately, you can use the ``allowed-eggs-from-site-packages`` buildout

   264         option as a glob-aware whitelist of eggs that may come from site-packages.

   265         This value defaults to "*", accepting all eggs.

   266         

   267         It's important to note that recipes not upgraded for zc.buildout 1.5.0

   268         should continue to work--just without internal support for a system Python.

   269         

   270         Using a system Python is inherently fragile.  Using a clean,

   271         freshly-installed Python without customization in site-packages is more

   272         robust and repeatable.  See some of the regression tests added to the

   273         1.5.0 line for the kinds of issues that you can encounter with a system

   274         Python, and see

   275         http://pypi.python.org/pypi/z3c.recipe.scripts#including-site-packages-and-sitecustomize

   276         for more discussion.

   277         

   278         However, using a system Python can be very convenient, and the

   279         zc.buildout code for this feature has been tested by many users already.

   280         Moreover, it has automated tests to exercise the problems that have been

   281         encountered and fixed.  Many people rely on it.

   282         

   283         Recipes That Support a System Python

   284         ====================================

   285         

   286         zc.recipe.egg continues to generate old-style scripts that are not safe

   287         for use with a system Python.  This was done for backwards

   288         compatibility, because it is integral to so many buildouts and used as a

   289         dependency of so many other recipes.

   290         

   291         If you want to generate new-style scripts that do support system Python

   292         usage, use z3c.recipe.scripts instead

   293         (http://pypi.python.org/pypi/z3c.recipe.scripts). z3c.recipe.scripts has

   294         the same script and interpreter generation options as zc.recipe.egg,

   295         plus a few more for the new features mentioned above.  In the simplest

   296         case, you should be able to simply change ``recipe = zc.recipe.egg`` to

   297         ``recipe = z3c.recipe.scripts`` in the pertinent sections of your

   298         buildout configuration and your generated scripts will work with a system

   299         Python.

   300         

   301         Other updated recipes include zc.recipe.testrunner 1.4.0 and

   302         z3c.recipe.tag 0.4.0.  Others should be updated soon: see their change

   303         documents for details, or see `Updating Recipes to Support a System

   304         Python`_ for instructions on how to update recipes yourself.

   305         

   306         Templates for creating Python scripts with the z3c.recipe.filetemplate

   307         recipe can be easily changed to support a system Python.

   308         

   309         - If you don't care about supporting relative paths, simply using a

   310           generated interpreter with the eggs you want should be sufficient, as

   311           it was before. For instance, if the interpreter is named "py", use

   312           ``#!${buildout:bin-directory/py}`` or ``#!/usr/bin/env

   313           ${buildout:bin-directory/py}``).

   314         

   315         - If you do care about relative paths,  (``relative-paths = true`` in

   316           your buildout configuration), then z3c.recipe.scripts does require a

   317           bit more changes, as is usual for the relative path support in that

   318           package.  First, use z3c.recipe.scripts to generate a script or

   319           interpreter with the dependencies you want.  This will create a

   320           directory in ``parts`` that has a site.py and sitecustomize.py.  Then,

   321           begin your script as in the snippet below.  The example assumes that

   322           the z3c.recipe.scripts generated were from a Buildout configuration

   323           section labeled "scripts": adjust accordingly.

   324         

   325           ::

   326         

   327             #!${buildout:executable} -S

   328             ${python-relative-path-setup}

   329             import sys

   330             sys.path.insert(0, ${scripts:parts-directory|path-repr})

   331             import site

   332         

   333         Updating Recipes to Support a System Python

   334         ===========================================

   335         

   336         You should generally only need to update recipes that generate scripts.

   337         These recipes need to change from using ``zc.buildout.easy_install.scripts``

   338         to be using ``zc.buildout.easy_install.sitepackage_safe_scripts``.

   339         The signatures of the two functions are different.  Please compare::

   340         

   341             def scripts(

   342                 reqs, working_set, executable, dest,

   343                 scripts=None,

   344                 extra_paths=(),

   345                 arguments='',

   346                 interpreter=None,

   347                 initialization='',

   348                 relative_paths=False,

   349                 ):

   350         

   351             def sitepackage_safe_scripts(

   352                 dest, working_set, executable, site_py_dest,

   353                 reqs=(),

   354                 scripts=None,

   355                 interpreter=None,

   356                 extra_paths=(),

   357                 initialization='',

   358                 include_site_packages=False,

   359                 exec_sitecustomize=False,

   360                 relative_paths=False,

   361                 script_arguments='',

   362                 script_initialization='',

   363                 ):

   364         

   365         In most cases, the arguments are merely reordered.  The ``reqs``

   366         argument is no longer required in order to make it easier to generate an

   367         interpreter alone.  The ``arguments`` argument was renamed to

   368         ``script_arguments`` to clarify that it did not affect interpreter

   369         generation.

   370         

   371         The only new required argument is ``site_py_dest``.  It must be the path

   372         to a directory in which the customized site.py and sitecustomize.py

   373         files will be written.  A typical generation in a recipe will look like

   374         this.

   375         

   376         (In the recipe's __init__ method...)

   377         

   378         ::

   379         

   380                 self.options = options

   381                 b_options = buildout['buildout']

   382                 options['parts-directory'] = os.path.join(

   383                     b_options['parts-directory'], self.name)

   384         

   385         (In the recipe's install method...)

   386         

   387         ::

   388         

   389                 options = self.options

   390                 generated = []

   391                 if not os.path.exists(options['parts-directory']):

   392                     os.mkdir(options['parts-directory'])

   393                     generated.append(options['parts-directory'])

   394         

   395         Then ``options['parts-directory']`` can be used for the ``site_py_dest``

   396         value.

   397         

   398         If you want to support the other arguments (``include_site_packages``,

   399         ``exec_sitecustomize``, ``script_initialization``, as well as the

   400         ``allowed-eggs-from-site-packages`` option),  you might want to look at

   401         some of the code in

   402         svn://svn.zope.org/repos/main/zc.buildout/trunk/z3c.recipe.scripts\_/src/z3c/recipe/scripts/scripts.py .

   403         You might even be able to adopt some of it by subclassing or delegating.

   404         The Scripts class in that file is the closest to what you might be used

   405         to from zc.recipe.egg.

   406         

   407         Important note for recipe authors: As of buildout 1.5.2, the code in

   408         recipes is *always run with the access to the site-packages as

   409         configured in the buildout section*.

   410         

   411         virtualenv

   412         ==========

   413         

   414         Using virtualenv (http://pypi.python.org/pypi/virtualenv) with the

   415         --no-site-packages option already provided a simple way of using a

   416         system Python.  This is intended to continue to work, and some automated

   417         tests exist to demonstrate this.

   418         

   419         However, it is only supported to the degree that people have found it to

   420         work in the past.  The existing Buildout tests for virtualenv are only

   421         for problems encountered previously.  They are very far from

   422         comprehensive.

   423         

   424         Using Buildout with a system python has at least three advantages over

   425         using Buildout in conjunction with virtualenv.  They may or may not be

   426         pertinent to your desired usage.

   427         

   428         - Unlike ``virtualenv --no-site-packages``, Buildout's support allows you

   429           to choose to let packages from your system Python be available to your

   430           software (see ``include-site-packages`` in

   431           http://pypi.python.org/pypi/z3c.recipe.scripts).

   432         

   433           You can even specify which eggs installed in your system Python can be

   434           allowed to fulfill some of your packages' dependencies (see

   435           ``allowed-eggs-from-site-packages`` in

   436           http://pypi.python.org/pypi/z3c.recipe.scripts).

   437         

   438           At the expense of some repeatability and platform dependency, this

   439           flexibility means that, for instance, you can rely on

   440           difficult-to-build eggs like lxml coming from your system Python.

   441         

   442         - Buildout's implementation has a full set of automated tests.

   443         

   444         - An integral Buildout implementation means fewer steps and fewer dependencies

   445           to work with a system Python.

   446         

   447         Detailed Documentation

   448         **********************

   449         

   450         Buildouts

   451         =========

   452         

   453         The word "buildout" refers to a description of a set of parts and the

   454         software to create and assemble them.  It is often used informally to

   455         refer to an installed system based on a buildout definition.  For

   456         example, if we are creating an application named "Foo", then "the Foo

   457         buildout" is the collection of configuration and application-specific

   458         software that allows an instance of the application to be created.  We

   459         may refer to such an instance of the application informally as "a Foo

   460         buildout".

   461         

   462         This document describes how to define buildouts using buildout

   463         configuration files and recipes.  There are three ways to set up the

   464         buildout software and create a buildout instance:

   465         

   466         1. Install the zc.buildout egg with easy_install and use the buildout

   467            script installed in a Python scripts area.

   468         

   469         2. Use the buildout bootstrap script to create a buildout that

   470            includes both the setuptools and zc.buildout eggs.  This allows you

   471            to use the buildout software without modifying a Python install.

   472            The buildout script is installed into your buildout local scripts

   473            area.

   474         

   475         3. Use a buildout command from an already installed buildout to

   476            bootstrap a new buildout.  (See the section on bootstraping later

   477            in this document.)

   478         

   479         Often, a software project will be managed in a software repository,

   480         such as a subversion repository, that includes some software source

   481         directories, buildout configuration files, and a copy of the buildout

   482         bootstrap script.  To work on the project, one would check out the

   483         project from the repository and run the bootstrap script which

   484         installs setuptools and zc.buildout into the checkout as well as any

   485         parts defined.

   486         

   487         We have a sample buildout that we created using the bootstrap command

   488         of an existing buildout (method 3 above).  It has the absolute minimum

   489         information.  We have bin, develop-eggs, eggs and parts directories,

   490         and a configuration file:

   491         

   492             >>> ls(sample_buildout)

   493             d  bin

   494             -  buildout.cfg

   495             d  develop-eggs

   496             d  eggs

   497             d  parts

   498         

   499         The bin directory contains scripts.

   500         

   501             >>> ls(sample_buildout, 'bin')

   502             -  buildout

   503         

   504             >>> ls(sample_buildout, 'eggs')

   505             -  setuptools-0.6-py2.4.egg

   506             -  zc.buildout-1.0-py2.4.egg

   507         

   508         The develop-eggs directory is initially empty:

   509         

   510             >>> ls(sample_buildout, 'develop-eggs')

   511         

   512         The develop-eggs directory holds egg links for software being

   513         developed in the buildout.  We separate develop-eggs and other eggs to

   514         allow eggs directories to be shared across multiple buildouts.  For

   515         example, a common developer technique is to define a common eggs

   516         directory in their home that all non-develop eggs are stored in.  This

   517         allows larger buildouts to be set up much more quickly and saves disk

   518         space.

   519         

   520         The parts directory just contains some helpers for the buildout script

   521         itself.

   522         

   523             >>> ls(sample_buildout, 'parts')

   524             d  buildout

   525         

   526         The parts directory provides an area where recipes can install

   527         part data.  For example, if we built a custom Python, we would

   528         install it in the part directory.  Part data is stored in a

   529         sub-directory of the parts directory with the same name as the part.

   530         

   531         Buildouts are defined using configuration files.  These are in the

   532         format defined by the Python ConfigParser module, with extensions

   533         that we'll describe later.  By default, when a buildout is run, it

   534         looks for the file buildout.cfg in the directory where the buildout is

   535         run.

   536         

   537         The minimal configuration file has a buildout section that defines no

   538         parts:

   539         

   540             >>> cat(sample_buildout, 'buildout.cfg')

   541             [buildout]

   542             parts =

   543         

   544         A part is simply something to be created by a buildout.  It can be

   545         almost anything, such as a Python package, a program, a directory, or

   546         even a configuration file.

   547         

   548         Recipes

   549         -------

   550         

   551         A part is created by a recipe.  Recipes are always installed as Python

   552         eggs. They can be downloaded from a package server, such as the

   553         Python Package Index, or they can be developed as part of a project

   554         using a "develop" egg.

   555         

   556         A develop egg is a special kind of egg that gets installed as an "egg

   557         link" that contains the name of a source directory.  Develop eggs

   558         don't have to be packaged for distribution to be used and can be

   559         modified in place, which is especially useful while they are being

   560         developed.

   561         

   562         Let's create a recipe as part of the sample project.  We'll create a

   563         recipe for creating directories.  First, we'll create a recipes source

   564         directory for our local recipes:

   565         

   566             >>> mkdir(sample_buildout, 'recipes')

   567         

   568         and then we'll create a source file for our mkdir recipe:

   569         

   570             >>> write(sample_buildout, 'recipes', 'mkdir.py',

   571             ... """

   572             ... import logging, os, zc.buildout

   573             ...

   574             ... class Mkdir:

   575             ...

   576             ...     def __init__(self, buildout, name, options):

   577             ...         self.name, self.options = name, options

   578             ...         options['path'] = os.path.join(

   579             ...                               buildout['buildout']['directory'],

   580             ...                               options['path'],

   581             ...                               )

   582             ...         if not os.path.isdir(os.path.dirname(options['path'])):

   583             ...             logging.getLogger(self.name).error(

   584             ...                 'Cannot create %s. %s is not a directory.',

   585             ...                 options['path'], os.path.dirname(options['path']))

   586             ...             raise zc.buildout.UserError('Invalid Path')

   587             ...

   588             ...

   589             ...     def install(self):

   590             ...         path = self.options['path']

   591             ...         logging.getLogger(self.name).info(

   592             ...             'Creating directory %s', os.path.basename(path))

   593             ...         os.mkdir(path)

   594             ...         return path

   595             ...

   596             ...     def update(self):

   597             ...         pass

   598             ... """)

   599         

   600         Currently, recipes must define 3 methods [#future_recipe_methods]_:

   601         

   602         - a constructor,

   603         

   604         - an install method, and

   605         

   606         - an update method.

   607         

   608         The constructor is responsible for updating a parts options to reflect

   609         data read from other sections.  The buildout system keeps track of

   610         whether a part specification has changed.  A part specification has

   611         changed if it's options, after adjusting for data read from other

   612         sections, has changed, or if the recipe has changed.  Only the options

   613         for the part are considered.  If data are read from other sections,

   614         then that information has to be reflected in the parts options.  In

   615         the Mkdir example, the given path is interpreted relative to the

   616         buildout directory, and data from the buildout directory is read.  The

   617         path option is updated to reflect this.  If the directory option was

   618         changed in the buildout sections, we would know to update parts

   619         created using the mkdir recipe using relative path names.

   620         

   621         When buildout is run, it saves configuration data for installed parts

   622         in a file named ".installed.cfg".  In subsequent runs, it compares

   623         part-configuration data stored in the .installed.cfg file and the

   624         part-configuration data loaded from the configuration files as

   625         modified by recipe constructors to decide if the configuration of a

   626         part has changed. If the configuration has changed, or if the recipe

   627         has changed, then the part is uninstalled and reinstalled.  The

   628         buildout only looks at the part's options, so any data used to

   629         configure the part needs to be reflected in the part's options.  It is

   630         the job of a recipe constructor to make sure that the options include

   631         all relevant data.

   632         

   633         Of course, parts are also uninstalled if they are no-longer used.

   634         

   635         The recipe defines a constructor that takes a buildout object, a part

   636         name, and an options dictionary. It saves them in instance attributes.

   637         If the path is relative, we'll interpret it as relative to the

   638         buildout directory.  The buildout object passed in is a mapping from

   639         section name to a mapping of options for that section. The buildout

   640         directory is available as the directory option of the buildout

   641         section.  We normalize the path and save it back into the options

   642         directory.

   643         

   644         The install method is responsible for creating the part.  In this

   645         case, we need the path of the directory to create.  We'll use a path

   646         option from our options dictionary.  The install method logs what it's

   647         doing using the Python logging call.  We return the path that we

   648         installed.  If the part is uninstalled or reinstalled, then the path

   649         returned will be removed by the buildout machinery.  A recipe install

   650         method is expected to return a string, or an iterable of strings

   651         containing paths to be removed if a part is uninstalled.  For most

   652         recipes, this is all of the uninstall support needed. For more complex

   653         uninstallation scenarios use `Uninstall recipes`_.

   654         

   655         The update method is responsible for updating an already installed

   656         part.  An empty method is often provided, as in this example, if parts

   657         can't be updated.  An update method can return None, a string, or an

   658         iterable of strings.  If a string or iterable of strings is returned,

   659         then the saved list of paths to be uninstalled is updated with the new

   660         information by adding any new files returned by the update method.

   661         

   662         We need to provide packaging information so that our recipe can be

   663         installed as a develop egg. The minimum information we need to specify

   664         [#packaging_info]_ is a name.  For recipes, we also need to define the

   665         names of the recipe classes as entry points.  Packaging information is

   666         provided via a setup.py script:

   667         

   668             >>> write(sample_buildout, 'recipes', 'setup.py',

   669             ... """

   670             ... from setuptools import setup

   671             ...

   672             ... setup(

   673             ...     name = "recipes",

   674             ...     entry_points = {'zc.buildout': ['mkdir = mkdir:Mkdir']},

   675             ...     )

   676             ... """)

   677         

   678         Our setup script defines an entry point. Entry points provide

   679         a way for an egg to define the services it provides.  Here we've said

   680         that we define a zc.buildout entry point named mkdir.  Recipe

   681         classes must be exposed as entry points in the zc.buildout group.  we

   682         give entry points names within the group.

   683         

   684         We also need a README.txt for our recipes to avoid an annoying warning

   685         from distutils, on which setuptools and zc.buildout are based:

   686         

   687             >>> write(sample_buildout, 'recipes', 'README.txt', " ")

   688         

   689         Now let's update our buildout.cfg:

   690         

   691             >>> write(sample_buildout, 'buildout.cfg',

   692             ... """

   693             ... [buildout]

   694             ... develop = recipes

   695             ... parts = data-dir

   696             ...

   697             ... [data-dir]

   698             ... recipe = recipes:mkdir

   699             ... path = mystuff

   700             ... """)

   701         

   702         Let's go through the changes one by one::

   703         

   704             develop = recipes

   705         

   706         This tells the buildout to install a development egg for our recipes.

   707         Any number of paths can be listed.  The paths can be relative or

   708         absolute.  If relative, they are treated as relative to the buildout

   709         directory.  They can be directory or file paths.  If a file path is

   710         given, it should point to a Python setup script.  If a directory path

   711         is given, it should point to a directory containing a setup.py file.

   712         Development eggs are installed before building any parts, as they may

   713         provide locally-defined recipes needed by the parts.

   714         

   715         ::

   716         

   717             parts = data-dir

   718         

   719         Here we've named a part to be "built".  We can use any name we want

   720         except that different part names must be unique and recipes will often

   721         use the part name to decide what to do.

   722         

   723         ::

   724         

   725             [data-dir]

   726             recipe = recipes:mkdir

   727             path = mystuff

   728         

   729         

   730         When we name a part, we also create a section of the same

   731         name that contains part data.  In this section, we'll define

   732         the recipe to be used to install the part.  In this case, we also

   733         specify the path to be created.

   734         

   735         Let's run the buildout.  We do so by running the build script in the

   736         buildout:

   737         

   738             >>> import os

   739             >>> os.chdir(sample_buildout)

   740             >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout')

   741             >>> print system(buildout),

   742             Develop: '/sample-buildout/recipes'

   743             Installing data-dir.

   744             data-dir: Creating directory mystuff

   745         

   746         We see that the recipe created the directory, as expected:

   747         

   748             >>> ls(sample_buildout)

   749             -  .installed.cfg

   750             d  bin

   751             -  buildout.cfg

   752             d  develop-eggs

   753             d  eggs

   754             d  mystuff

   755             d  parts

   756             d  recipes

   757         

   758         In addition, .installed.cfg has been created containing information

   759         about the part we installed:

   760         

   761             >>> cat(sample_buildout, '.installed.cfg')

   762             [buildout]

   763             installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link

   764             parts = data-dir

   765             <BLANKLINE>

   766             [data-dir]

   767             __buildout_installed__ = /sample-buildout/mystuff

   768             __buildout_signature__ = recipes-c7vHV6ekIDUPy/7fjAaYjg==

   769             path = /sample-buildout/mystuff

   770             recipe = recipes:mkdir

   771         

   772         Note that the directory we installed is included in .installed.cfg.

   773         In addition, the path option includes the actual destination

   774         directory.

   775         

   776         If we change the name of the directory in the configuration file,

   777         we'll see that the directory gets removed and recreated:

   778         

   779             >>> write(sample_buildout, 'buildout.cfg',

   780             ... """

   781             ... [buildout]

   782             ... develop = recipes

   783             ... parts = data-dir

   784             ...

   785             ... [data-dir]

   786             ... recipe = recipes:mkdir

   787             ... path = mydata

   788             ... """)

   789         

   790             >>> print system(buildout),

   791             Develop: '/sample-buildout/recipes'

   792             Uninstalling data-dir.

   793             Installing data-dir.

   794             data-dir: Creating directory mydata

   795         

   796             >>> ls(sample_buildout)

   797             -  .installed.cfg

   798             d  bin

   799             -  buildout.cfg

   800             d  develop-eggs

   801             d  eggs

   802             d  mydata

   803             d  parts

   804             d  recipes

   805         

   806         If any of the files or directories created by a recipe are removed,

   807         the part will be reinstalled:

   808         

   809             >>> rmdir(sample_buildout, 'mydata')

   810             >>> print system(buildout),

   811             Develop: '/sample-buildout/recipes'

   812             Uninstalling data-dir.

   813             Installing data-dir.

   814             data-dir: Creating directory mydata

   815         

   816         Error reporting

   817         ---------------

   818         

   819         If a user makes an error, an error needs to be printed and work needs

   820         to stop.  This is accomplished by logging a detailed error message and

   821         then raising a (or an instance of a subclass of a)

   822         zc.buildout.UserError exception.  Raising an error other than a

   823         UserError still displays the error, but labels it as a bug in the

   824         buildout software or recipe. In the sample above, of someone gives a

   825         non-existent directory to create the directory in:

   826         

   827         

   828             >>> write(sample_buildout, 'buildout.cfg',

   829             ... """

   830             ... [buildout]

   831             ... develop = recipes

   832             ... parts = data-dir

   833             ...

   834             ... [data-dir]

   835             ... recipe = recipes:mkdir

   836             ... path = /xxx/mydata

   837             ... """)

   838         

   839         We'll get a user error, not a traceback.

   840         

   841             >>> print system(buildout),

   842             Develop: '/sample-buildout/recipes'

   843             data-dir: Cannot create /xxx/mydata. /xxx is not a directory.

   844             While:

   845               Installing.

   846               Getting section data-dir.

   847               Initializing part data-dir.

   848             Error: Invalid Path

   849         

   850         

   851         Recipe Error Handling

   852         ---------------------

   853         

   854         If an error occurs during installation, it is up to the recipe to

   855         clean up any system side effects, such as files created.  Let's update

   856         the mkdir recipe to support multiple paths:

   857         

   858             >>> write(sample_buildout, 'recipes', 'mkdir.py',

   859             ... """

   860             ... import logging, os, zc.buildout

   861             ...

   862             ... class Mkdir:

   863             ...

   864             ...     def __init__(self, buildout, name, options):

   865             ...         self.name, self.options = name, options

   866             ...

   867             ...         # Normalize paths and check that their parent

   868             ...         # directories exist:

   869             ...         paths = []

   870             ...         for path in options['path'].split():

   871             ...             path = os.path.join(buildout['buildout']['directory'], path)

   872             ...             if not os.path.isdir(os.path.dirname(path)):

   873             ...                 logging.getLogger(self.name).error(

   874             ...                     'Cannot create %s. %s is not a directory.',

   875             ...                     options['path'], os.path.dirname(options['path']))

   876             ...                 raise zc.buildout.UserError('Invalid Path')

   877             ...             paths.append(path)

   878             ...         options['path'] = ' '.join(paths)

   879             ...

   880             ...     def install(self):

   881             ...         paths = self.options['path'].split()

   882             ...         for path in paths:

   883             ...             logging.getLogger(self.name).info(

   884             ...                 'Creating directory %s', os.path.basename(path))

   885             ...             os.mkdir(path)

   886             ...         return paths

   887             ...

   888             ...     def update(self):

   889             ...         pass

   890             ... """)

   891         

   892         If there is an error creating a path, the install method will exit and

   893         leave previously created paths in place:

   894         

   895             >>> write(sample_buildout, 'buildout.cfg',

   896             ... """

   897             ... [buildout]

   898             ... develop = recipes

   899             ... parts = data-dir

   900             ...

   901             ... [data-dir]

   902             ... recipe = recipes:mkdir

   903             ... path = foo bin

   904             ... """)

   905         

   906             >>> print system(buildout), # doctest: +ELLIPSIS

   907             Develop: '/sample-buildout/recipes'

   908             Uninstalling data-dir.

   909             Installing data-dir.

   910             data-dir: Creating directory foo

   911             data-dir: Creating directory bin

   912             While:

   913               Installing data-dir.

   914             <BLANKLINE>

   915             An internal error occurred due to a bug in either zc.buildout or in a

   916             recipe being used:

   917             Traceback (most recent call last):

   918               ...

   919             OSError: [Errno 17] File exists: '/sample-buildout/bin'

   920         

   921         We meant to create a directory bins, but typed bin.  Now foo was

   922         left behind.

   923         

   924             >>> os.path.exists('foo')

   925             True

   926         

   927         If we fix the typo:

   928         

   929             >>> write(sample_buildout, 'buildout.cfg',

   930             ... """

   931             ... [buildout]

   932             ... develop = recipes

   933             ... parts = data-dir

   934             ...

   935             ... [data-dir]

   936             ... recipe = recipes:mkdir

   937             ... path = foo bins

   938             ... """)

   939         

   940             >>> print system(buildout), # doctest: +ELLIPSIS

   941             Develop: '/sample-buildout/recipes'

   942             Installing data-dir.

   943             data-dir: Creating directory foo

   944             While:

   945               Installing data-dir.

   946             <BLANKLINE>

   947             An internal error occurred due to a bug in either zc.buildout or in a

   948             recipe being used:

   949             Traceback (most recent call last):

   950             ...

   951             OSError: [Errno 17] File exists: '/sample-buildout/foo'

   952         

   953         Now they fail because foo exists, because it was left behind.

   954         

   955             >>> remove('foo')

   956         

   957         Let's fix the recipe:

   958         

   959             >>> write(sample_buildout, 'recipes', 'mkdir.py',

   960             ... """

   961             ... import logging, os, zc.buildout

   962             ...

   963             ... class Mkdir:

   964             ...

   965             ...     def __init__(self, buildout, name, options):

   966             ...         self.name, self.options = name, options

   967             ...

   968             ...         # Normalize paths and check that their parent

   969             ...         # directories exist:

   970             ...         paths = []

   971             ...         for path in options['path'].split():

   972             ...             path = os.path.join(buildout['buildout']['directory'], path)

   973             ...             if not os.path.isdir(os.path.dirname(path)):

   974             ...                 logging.getLogger(self.name).error(

   975             ...                     'Cannot create %s. %s is not a directory.',

   976             ...                     options['path'], os.path.dirname(options['path']))

   977             ...                 raise zc.buildout.UserError('Invalid Path')

   978             ...             paths.append(path)

   979             ...         options['path'] = ' '.join(paths)

   980             ...

   981             ...     def install(self):

   982             ...         paths = self.options['path'].split()

   983             ...         created = []

   984             ...         try:

   985             ...             for path in paths:

   986             ...                 logging.getLogger(self.name).info(

   987             ...                     'Creating directory %s', os.path.basename(path))

   988             ...                 os.mkdir(path)

   989             ...                 created.append(path)

   990             ...         except:

   991             ...             for d in created:

   992             ...                 os.rmdir(d)

   993             ...             raise

   994             ...

   995             ...         return paths

   996             ...

   997             ...     def update(self):

   998             ...         pass

   999             ... """)

  1000         

  1001         And put back the typo:

  1002         

  1003             >>> write(sample_buildout, 'buildout.cfg',

  1004             ... """

  1005             ... [buildout]

  1006             ... develop = recipes

  1007             ... parts = data-dir

  1008             ...

  1009             ... [data-dir]

  1010             ... recipe = recipes:mkdir

  1011             ... path = foo bin

  1012             ... """)

  1013         

  1014         When we rerun the buildout:

  1015         

  1016             >>> print system(buildout), # doctest: +ELLIPSIS

  1017             Develop: '/sample-buildout/recipes'

  1018             Installing data-dir.

  1019             data-dir: Creating directory foo

  1020             data-dir: Creating directory bin

  1021             While:

  1022               Installing data-dir.

  1023             <BLANKLINE>

  1024             An internal error occurred due to a bug in either zc.buildout or in a

  1025             recipe being used:

  1026             Traceback (most recent call last):

  1027             ...

  1028             OSError: [Errno 17] File exists: '/sample-buildout/bin'

  1029         

  1030         .. Wait for the file to really disappear. My linux is weird.

  1031         

  1032             >>> wait_until("foo goes away", lambda : not os.path.exists('foo'),

  1033             ...            timeout=200)

  1034         

  1035         we get the same error, but we don't get the directory left behind:

  1036         

  1037             >>> os.path.exists('foo')

  1038             False

  1039         

  1040         It's critical that recipes clean up partial effects when errors

  1041         occur.  Because recipes most commonly create files and directories,

  1042         buildout provides a helper API for removing created files when an

  1043         error occurs.  Option objects have a created method that can be called

  1044         to record files as they are created.  If the install or update method

  1045         returns with an error, then any registered paths are removed

  1046         automatically.  The method returns the files registered and can be

  1047         used to return the files created.  Let's use this API to simplify the

  1048         recipe:

  1049         

  1050             >>> write(sample_buildout, 'recipes', 'mkdir.py',

  1051             ... """

  1052             ... import logging, os, zc.buildout

  1053             ...

  1054             ... class Mkdir:

  1055             ...

  1056             ...     def __init__(self, buildout, name, options):

  1057             ...         self.name, self.options = name, options

  1058             ...

  1059             ...         # Normalize paths and check that their parent

  1060             ...         # directories exist:

  1061             ...         paths = []

  1062             ...         for path in options['path'].split():

  1063             ...             path = os.path.join(buildout['buildout']['directory'], path)

  1064             ...             if not os.path.isdir(os.path.dirname(path)):

  1065             ...                 logging.getLogger(self.name).error(

  1066             ...                     'Cannot create %s. %s is not a directory.',

  1067             ...                     options['path'], os.path.dirname(options['path']))

  1068             ...                 raise zc.buildout.UserError('Invalid Path')

  1069             ...             paths.append(path)

  1070             ...         options['path'] = ' '.join(paths)

  1071             ...

  1072             ...     def install(self):

  1073             ...         paths = self.options['path'].split()

  1074             ...         for path in paths:

  1075             ...             logging.getLogger(self.name).info(

  1076             ...                 'Creating directory %s', os.path.basename(path))

  1077             ...             os.mkdir(path)

  1078             ...             self.options.created(path)

  1079             ...

  1080             ...         return self.options.created()

  1081             ...

  1082             ...     def update(self):

  1083             ...         pass

  1084             ... """)

  1085         

  1086         ..

  1087         

  1088             >>> remove(sample_buildout, 'recipes', 'mkdir.pyc')

  1089         

  1090         We returned by calling created, taking advantage of the fact that it

  1091         returns the registered paths.  We did this for illustrative purposes.

  1092         It would be simpler just to return the paths as before.

  1093         

  1094         If we rerun the buildout, again, we'll get the error and no

  1095         directories will be created:

  1096         

  1097             >>> print system(buildout), # doctest: +ELLIPSIS

  1098             Develop: '/sample-buildout/recipes'

  1099             Installing data-dir.

  1100             data-dir: Creating directory foo

  1101             data-dir: Creating directory bin

  1102             While:

  1103               Installing data-dir.

  1104             <BLANKLINE>

  1105             An internal error occurred due to a bug in either zc.buildout or in a

  1106             recipe being used:

  1107             Traceback (most recent call last):

  1108             ...

  1109             OSError: [Errno 17] File exists: '/sample-buildout/bin'

  1110         

  1111             >>> os.path.exists('foo')

  1112             False

  1113         

  1114         Now, we'll fix the typo again and we'll get the directories we expect:

  1115         

  1116             >>> write(sample_buildout, 'buildout.cfg',

  1117             ... """

  1118             ... [buildout]

  1119             ... develop = recipes

  1120             ... parts = data-dir

  1121             ...

  1122             ... [data-dir]

  1123             ... recipe = recipes:mkdir

  1124             ... path = foo bins

  1125             ... """)

  1126         

  1127             >>> print system(buildout),

  1128             Develop: '/sample-buildout/recipes'

  1129             Installing data-dir.

  1130             data-dir: Creating directory foo

  1131             data-dir: Creating directory bins

  1132         

  1133             >>> os.path.exists('foo')

  1134             True

  1135             >>> os.path.exists('bins')

  1136             True