Creating eggs with extensions needing custom build settings

=============================================================

Sometimes, It's necessary to provide extra control over how an egg is

created.  This is commonly true for eggs with extension modules that

need to access libraries or include files.

The zc.recipe.egg:custom recipe can be used to define an egg with

custom build parameters.  The currently defined parameters are:

include-dirs

   A new-line separated list of directories to search for include

   files.

library-dirs

   A new-line separated list of directories to search for libraries

   to link with.

rpath

   A new-line separated list of directories to search for dynamic libraries

   at run time.

define

   A comma-separated list of names of C preprocessor variables to

   define.

undef

   A comman separated list of names of C preprocessor variables to

   undefine.

libraries

   The name of an additional library to link with.  Due to limitations

   in distutils and desprite the option name, only a single library

   can be specified.

link-objects

   The name of an link object to link against.  Due to limitations

   in distutils and desprite the option name, only a single link object

   can be specified.

debug

   Compile/link with debugging information

force

   Forcibly build everything (ignore file timestamps)

compiler

   Specify the compiler type

swig

   The path to the swig executable

swig-cpp

   Make SWIG create C++ files (default is C)

swig-opts

   List of SWIG command line options

In addition, the following options can be used to specify the egg:

egg

    An specification for the egg to be created, to install given as a

    setuptools requirement string.  This defaults to the part name.

find-links

   A list of URLs, files, or directories to search for distributions.

index

   The URL of an index server, or almost any other valid URL. :)

   If not specified, the Python Package Index,

   http://cheeseshop.python.org/pypi, is used.  You can specify an

   alternate index with this option.  If you use the links option and

   if the links point to the needed distributions, then the index can

   be anything and will be largely ignored.  In the examples, here,

   we'll just point to an empty directory on our link server.  This

   will make our examples run a little bit faster.

python

   The name of a section to get the Python executable from.

   If not specified, then the buildout python option is used.  The

   Python executable is found in the executable option of the named

   section.

environment

   The name of a section with additional environment variables. The

   environment variables are set before the egg is built.

To illustrate this, we'll define a buildout that builds an egg for a

package that has a simple extension module::

  #include <Python.h>

  #include <extdemo.h>

  static PyMethodDef methods[] = {};

  PyMODINIT_FUNC

  initextdemo(void)

  {

      PyObject *m;

      m = Py_InitModule3("extdemo", methods, "");

  #ifdef TWO

      PyModule_AddObject(m, "val", PyInt_FromLong(2));

  #else

      PyModule_AddObject(m, "val", PyInt_FromLong(EXTDEMO));

  #endif

  }

The extension depends on a system-dependent include file, extdemo.h,

that defines a constant, EXTDEMO, that is exposed by the extension.

The extension module is available as a source distribution,

extdemo-1.4.tar.gz, on a distribution server.

We have a sample buildout that we'll add an include directory to with

the necessary include file:

    >>> mkdir('include')

    >>> write('include', 'extdemo.h',

    ... """

    ... #define EXTDEMO 42

    ... """)

We'll also update the buildout configuration file to define a part for

the egg:

    >>> write(sample_buildout, 'buildout.cfg',

    ... """

    ... [buildout]

    ... parts = extdemo

    ...

    ... [extdemo]

    ... recipe = zc.recipe.egg:custom

    ... find-links = %(server)s

    ... index = %(server)s/index

    ... include-dirs = include

    ...

    ... """ % dict(server=link_server))

    >>> print system(buildout),

    Installing extdemo.

    zip_safe flag not set; analyzing archive contents...

We got the zip_safe warning because the source distribution we used

wasn't setuptools based and thus didn't set the option.

The egg is created in the develop-eggs directory *not* the eggs

directory because it depends on buildout-specific parameters and the

eggs directory can be shared across multiple buildouts.

    >>> ls(sample_buildout, 'develop-eggs')

    d  extdemo-1.4-py2.4-unix-i686.egg

    -  z3c.recipe.scripts.egg-link

    -  zc.recipe.egg.egg-link

Note that no scripts or dependencies are installed.  To install

dependencies or scripts for a custom egg, define another part and use

the zc.recipe.egg recipe, listing the custom egg as one of the eggs to

be installed.  The zc.recipe.egg recipe will use the installed egg.

Let's define a script that uses out ext demo:

    >>> mkdir('demo')

    >>> write('demo', 'demo.py',

    ... """

    ... import extdemo

    ... def main():

    ...     print extdemo.val

    ... """)

    >>> write('demo', 'setup.py',

    ... """

    ... from setuptools import setup

    ... setup(name='demo')

    ... """)

    >>> write('buildout.cfg',

    ... """

    ... [buildout]

    ... develop = demo

    ... parts = extdemo demo

    ...

    ... [extdemo]

    ... recipe = zc.recipe.egg:custom

    ... find-links = %(server)s

    ... index = %(server)s/index

    ... include-dirs = include

    ...

    ... [demo]

    ... recipe = zc.recipe.egg

    ... eggs = demo

    ...        extdemo

    ... entry-points = demo=demo:main

    ... """ % dict(server=link_server))

    >>> print system(buildout),

    Develop: '/sample-buildout/demo'

    Updating extdemo.

    Installing demo.

    Generated script '/sample-buildout/bin/demo'.

When we run the script, we'll 42 printed:

    >>> print system(join('bin', 'demo')),

    42

Updating

--------

The custom recipe will normally check for new source distributions

that meet the given specification.  This can be suppressed using the

buildout non-newest and offline modes.  We'll generate a new source

distribution for extdemo:

    >>> update_extdemo()

If we run the buildout in non-newest or offline modes:

    >>> print system(buildout+' -N'),

    Develop: '/sample-buildout/demo'

    Updating extdemo.

    Updating demo.

    >>> print system(buildout+' -o'),

    Develop: '/sample-buildout/demo'

    Updating extdemo.

    Updating demo.

We won't get an update.

    >>> ls(sample_buildout, 'develop-eggs')

    -  demo.egg-link

    d  extdemo-1.4-py2.4-unix-i686.egg

    -  z3c.recipe.scripts.egg-link

    -  zc.recipe.egg.egg-link

But if we run the buildout in the default on-line and newest modes, we

will. This time we also get the test-variable message again, because the new

version is imported:

    >>> print system(buildout),

    Develop: '/sample-buildout/demo'

    Updating extdemo.

    zip_safe flag not set; analyzing archive contents...

    Updating demo.

    Generated script '/sample-buildout/bin/demo'.

    >>> ls(sample_buildout, 'develop-eggs')

    -  demo.egg-link

    d  extdemo-1.4-py2.4-linux-i686.egg

    d  extdemo-1.5-py2.4-linux-i686.egg

    -  z3c.recipe.scripts.egg-link

    -  zc.recipe.egg.egg-link

Controlling the version used

----------------------------

We can specify a specific version using the egg option:

    >>> write('buildout.cfg',

    ... """

    ... [buildout]

    ... develop = demo

    ... parts = extdemo demo

    ...

    ... [extdemo]

    ... recipe = zc.recipe.egg:custom

    ... egg = extdemo ==1.4

    ... find-links = %(server)s

    ... index = %(server)s/index

    ... include-dirs = include

    ...

    ... [demo]

    ... recipe = zc.recipe.egg

    ... eggs = demo

    ...        extdemo ==1.4

    ... entry-points = demo=demo:main

    ... """ % dict(server=link_server))

    >>> print system(buildout+' -D'),

    Develop: '/sample-buildout/demo'

    Uninstalling demo.

    Uninstalling extdemo.

    Installing extdemo.

    zip_safe flag not set; analyzing archive contents...

    Installing demo.

    Generated script '/sample-buildout/bin/demo'.

    >>> ls(sample_buildout, 'develop-eggs')

    -  demo.egg-link

    d  extdemo-1.4-py2.4-linux-i686.egg

    -  z3c.recipe.scripts.egg-link

    -  zc.recipe.egg.egg-link

Controlling environment variables

+++++++++++++++++++++++++++++++++

To set additional environment variables, the `environment` option is used.

Let's create a recipe which prints out environment variables. We need this to

make sure the set envirionment variables are removed after the egg:custom

recipe was run.

    >>> mkdir(sample_buildout, 'recipes')

    >>> write(sample_buildout, 'recipes', 'environ.py',

    ... """

    ... import logging, os, zc.buildout

    ...

    ... class Environ:

    ...

    ...     def __init__(self, buildout, name, options):

    ...         self.name = name

    ...

    ...     def install(self):

    ...         logging.getLogger(self.name).info(

    ...             'test-variable left over: %s' % (

    ...                 'test-variable' in os.environ))

    ...         return []

    ...

    ...     def update(self):

    ...         self.install()

    ... """)

    >>> write(sample_buildout, 'recipes', 'setup.py',

    ... """

    ... from setuptools import setup

    ...

    ... setup(

    ...     name = "recipes",

    ...     entry_points = {'zc.buildout': ['environ = environ:Environ']},

    ...     )

    ... """)

Create our buildout:

    >>> write(sample_buildout, 'buildout.cfg',

    ... """

    ... [buildout]

    ... develop = recipes

    ... parts = extdemo checkenv

    ...

    ... [extdemo-env]

    ... test-variable = foo

    ...

    ... [extdemo]

    ... recipe = zc.recipe.egg:custom

    ... find-links = %(server)s

    ... index = %(server)s/index

    ... include-dirs = include

    ... environment = extdemo-env

    ...

    ... [checkenv]

    ... recipe = recipes:environ

    ...

    ... """ % dict(server=link_server))

    >>> print system(buildout),

    Develop: '/sample-buildout/recipes'

    Uninstalling demo.

    Uninstalling extdemo.

    Installing extdemo.

    Have environment test-variable: foo

    zip_safe flag not set; analyzing archive contents...

    Installing checkenv.

    checkenv: test-variable left over: False

The setup.py also printed out that we have set the environment `test-variable`

to foo. After the buildout the variable is reset to its original value (i.e.

removed).

When an environment variable has a value before zc.recipe.egg:custom is run,

the original value will be restored:

    >>> import os

    >>> os.environ['test-variable'] = 'bar'

    >>> print system(buildout),

    Develop: '/sample-buildout/recipes'

    Updating extdemo.

    Updating checkenv.

    checkenv: test-variable left over: True

    >>> os.environ['test-variable']

    'bar'

Sometimes it is required to prepend or append to an existing environment

variable, for instance for adding something to the PATH. Therefor all variables

are interpolated with os.environ before the're set:

    >>> write(sample_buildout, 'buildout.cfg',

    ... """

    ... [buildout]

    ... develop = recipes

    ... parts = extdemo checkenv

    ...

    ... [extdemo-env]

    ... test-variable = foo:%%(test-variable)s

    ...

    ... [extdemo]

    ... recipe = zc.recipe.egg:custom

    ... find-links = %(server)s

    ... index = %(server)s/index

    ... include-dirs = include

    ... environment = extdemo-env

    ...

    ... [checkenv]

    ... recipe = recipes:environ

    ...

    ... """ % dict(server=link_server))

    >>> print system(buildout),

    Develop: '/sample-buildout/recipes'

    Uninstalling extdemo.

    Installing extdemo.

    Have environment test-variable: foo:bar

    zip_safe flag not set; analyzing archive contents...

    Updating checkenv.

    checkenv: test-variable left over: True

    >>> os.environ['test-variable']

    'bar'

    >>> del os.environ['test-variable']

Create a clean buildout.cfg w/o the checkenv recipe, and delete the recipe:

    >>> write(sample_buildout, 'buildout.cfg',

    ... """

    ... [buildout]

    ... develop = recipes

    ... parts = extdemo

    ...

    ... [extdemo]

    ... recipe = zc.recipe.egg:custom

    ... find-links = %(server)s

    ... index = %(server)s/index

    ... include-dirs = include

    ...

    ... """ % dict(server=link_server))

    >>> print system(buildout),

    Develop: '/sample-buildout/recipes'

    Uninstalling checkenv.

    Uninstalling extdemo.

    Installing extdemo.

    zip_safe flag not set; analyzing archive contents...

    >>> rmdir(sample_buildout, 'recipes')

Controlling develop-egg generation

==================================

If you want to provide custom build options for a develop egg, you can

use the develop recipe.  The recipe has the following options:

path

   The path to a setup script or directory containing a startup

   script. This is required.

include-dirs

   A new-line separated list of directories to search for include

   files.

library-dirs

   A new-line separated list of directories to search for libraries

   to link with.

rpath

   A new-line separated list of directories to search for dynamic libraries

   at run time.

define

   A comma-separated list of names of C preprocessor variables to

   define.

undef

   A comman separated list of names of C preprocessor variables to

   undefine.

libraries

   The name of an additional library to link with.  Due to limitations

   in distutils and desprite the option name, only a single library

   can be specified.

link-objects

   The name of an link object to link against.  Due to limitations

   in distutils and desprite the option name, only a single link object

   can be specified.

debug

   Compile/link with debugging information

force

   Forcibly build everything (ignore file timestamps)

compiler

   Specify the compiler type

swig

   The path to the swig executable

swig-cpp

   Make SWIG create C++ files (default is C)

swig-opts

   List of SWIG command line options

python

   The name of a section to get the Python executable from.

   If not specified, then the buildout python option is used.  The

   Python executable is found in the executable option of the named

   section.

To illustrate this, we'll use a directory containing the extdemo

example from the earlier section:

    >>> ls(extdemo)

    -  MANIFEST

    -  MANIFEST.in

    -  README

    -  extdemo.c

    -  setup.py

    >>> write('buildout.cfg',

    ... """

    ... [buildout]

    ... develop = demo

    ... parts = extdemo demo