Python API for egg and script installation

==========================================

The easy_install module provides some functions to provide support for

egg and script installation.  It provides functionality at the python

level that is similar to easy_install, with a few exceptions:

- By default, we look for new packages *and* the packages that

  they depend on.  This is somewhat like (and uses) the --upgrade

  option of easy_install, except that we also upgrade required

  packages.

- If the highest-revision package satisfying a specification is

  already present, then we don't try to get another one.  This saves a

  lot of search time in the common case that packages are pegged to

  specific versions.

- If there is a develop egg that satisfies a requirement, we don't

  look for additional distributions.  We always give preference to

  develop eggs.

- Distutils options for building extensions can be passed.

Distribution installation

-------------------------

The easy_install module provides a function, install, for installing one

or more packages and their dependencies.  The install function takes 2

positional arguments:

- An iterable of setuptools requirement strings for the distributions

  to be installed, and

- A destination directory to install to and to satisfy requirements

  from.  The destination directory can be None, in which case, no new

  distributions are downloaded and there will be an error if the

  needed distributions can't be found among those already installed.

It supports a number of optional keyword arguments:

links

   A sequence of URLs, file names, or directories to look for

   links to distributions.

index

   The URL of an index server, or almost any other valid URL. :)

   If not specified, the Python Package Index,

   http://pypi.python.org/simple/, is used.  You can specify an

   alternate index with this option.  If you use the links option and

   if the links point to the needed distributions, then the index can

   be anything and will be largely ignored.  In the examples, here,

   we'll just point to an empty directory on our link server.  This

   will make our examples run a little bit faster.

executable

   A path to a Python executable.  Distributions will be installed

   using this executable and will be for the matching Python version.

path

   A list of additional directories to search for locally-installed

   distributions.

always_unzip

   A flag indicating that newly-downloaded distributions should be

   directories even if they could be installed as zip files.

working_set

   An existing working set to be augmented with additional

   distributions, if necessary to satisfy requirements.  This allows

   you to call install multiple times, if necessary, to gather

   multiple sets of requirements.

newest

   A boolean value indicating whether to search for new distributions

   when already-installed distributions meet the requirement.  When

   this is true, the default, and when the destination directory is

   not None, then the install function will search for the newest

   distributions that satisfy the requirements.

versions

   A dictionary mapping project names to version numbers to be used

   when selecting distributions.  This can be used to specify a set of

   distribution versions independent of other requirements.

use_dependency_links

   A flag indicating whether to search for dependencies using the

   setup dependency_links metadata or not. If true, links are searched

   for using dependency_links in preference to other

   locations. Defaults to true.

include_site_packages

    A flag indicating whether Python's non-standard-library packages should

    be available for finding dependencies.  Defaults to true.

    Paths outside of Python's standard library--or more precisely, those that

    are not included when Python is started with the -S argument--are loosely

    referred to as "site-packages" here.

relative_paths

   Adjust egg paths so they are relative to the script path.  This

   allows scripts to work when scripts and eggs are moved, as long as

   they are both moved in the same way.

The install method returns a working set containing the distributions

needed to meet the given requirements.

We have a link server that has a number of eggs:

    >>> print get(link_server),

    <html><body>

    <a href="bigdemo-0.1-py2.4.egg">bigdemo-0.1-py2.4.egg</a><br>

    <a href="demo-0.1-py2.4.egg">demo-0.1-py2.4.egg</a><br>

    <a href="demo-0.2-py2.4.egg">demo-0.2-py2.4.egg</a><br>

    <a href="demo-0.3-py2.4.egg">demo-0.3-py2.4.egg</a><br>

    <a href="demo-0.4c1-py2.4.egg">demo-0.4c1-py2.4.egg</a><br>

    <a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br>

    <a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br>

    <a href="demoneeded-1.2c1.zip">demoneeded-1.2c1.zip</a><br>

    <a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br>

    <a href="index/">index/</a><br>

    <a href="other-1.0-py2.4.egg">other-1.0-py2.4.egg</a><br>

    </body></html>

Let's make a directory and install the demo egg to it, using the demo:

    >>> dest = tmpdir('sample-install')

    >>> import zc.buildout.easy_install

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo==0.2'], dest,

    ...     links=[link_server], index=link_server+'index/')

We requested version 0.2 of the demo distribution to be installed into

the destination server.  We specified that we should search for links

on the link server and that we should use the (empty) link server

index directory as a package index.

The working set contains the distributions we retrieved.

    >>> for dist in ws:

    ...     print dist

    demo 0.2

    demoneeded 1.1

We got demoneeded because it was a dependency of demo.

And the actual eggs were added to the eggs directory.

    >>> ls(dest)

    -  demo-0.2-py2.4.egg

    -  demoneeded-1.1-py2.4.egg

If we remove the version restriction on demo, but specify a false

value for newest, no new distributions will be installed:

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo'], dest, links=[link_server], index=link_server+'index/',

    ...     newest=False)

    >>> ls(dest)

    -  demo-0.2-py2.4.egg

    -  demoneeded-1.1-py2.4.egg

If we leave off the newest option, we'll get an update for demo:

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo'], dest, links=[link_server], index=link_server+'index/')

    >>> ls(dest)

    -  demo-0.2-py2.4.egg

    -  demo-0.3-py2.4.egg

    -  demoneeded-1.1-py2.4.egg

Note that we didn't get the newest versions available.  There were

release candidates for newer versions of both packages. By default,

final releases are preferred.  We can change this behavior using the

prefer_final function:

    >>> zc.buildout.easy_install.prefer_final(False)

    True

The old setting is returned.

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo'], dest, links=[link_server], index=link_server+'index/')

    >>> for dist in ws:

    ...     print dist

    demo 0.4c1

    demoneeded 1.2c1

    >>> ls(dest)

    -  demo-0.2-py2.4.egg

    -  demo-0.3-py2.4.egg

    -  demo-0.4c1-py2.4.egg

    -  demoneeded-1.1-py2.4.egg

    -  demoneeded-1.2c1-py2.4.egg

Let's put the setting back to the default.

    >>> zc.buildout.easy_install.prefer_final(True)

    False

We can supply additional distributions.  We can also supply

specifications for distributions that would normally be found via

dependencies.  We might do this to specify a specific version.

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo', 'other', 'demoneeded==1.0'], dest,

    ...     links=[link_server], index=link_server+'index/')

    >>> for dist in ws:

    ...     print dist

    demo 0.3

    other 1.0

    demoneeded 1.0

    >>> ls(dest)

    -  demo-0.2-py2.4.egg

    -  demo-0.3-py2.4.egg

    -  demo-0.4c1-py2.4.egg

    -  demoneeded-1.0-py2.4.egg

    -  demoneeded-1.1-py2.4.egg

    -  demoneeded-1.2c1-py2.4.egg

    d  other-1.0-py2.4.egg

We can request that eggs be unzipped even if they are zip safe.  This

can be useful when debugging.  (Note that Distribute will unzip eggs by

default, so if you are using Distribute, most or all eggs will already be

unzipped without this flag.)

    >>> rmdir(dest)

    >>> dest = tmpdir('sample-install')

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo'], dest, links=[link_server], index=link_server+'index/',

    ...     always_unzip=True)

    >>> ls(dest)

    d  demo-0.3-py2.4.egg

    d  demoneeded-1.1-py2.4.egg

    >>> rmdir(dest)

    >>> dest = tmpdir('sample-install')

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo'], dest, links=[link_server], index=link_server+'index/',

    ...     always_unzip=False)

    >>> ls(dest)

    -  demo-0.3-py2.4.egg

    -  demoneeded-1.1-py2.4.egg

We can also set a default by calling the always_unzip function:

    >>> zc.buildout.easy_install.always_unzip(True)

    False

The old default is returned:

    >>> rmdir(dest)

    >>> dest = tmpdir('sample-install')

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo'], dest, links=[link_server], index=link_server+'index/')

    >>> ls(dest)

    d  demo-0.3-py2.4.egg

    d  demoneeded-1.1-py2.4.egg

    >>> zc.buildout.easy_install.always_unzip(False)

    True

    >>> rmdir(dest)

    >>> dest = tmpdir('sample-install')

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo'], dest, links=[link_server], index=link_server+'index/')

    >>> ls(dest)

    -  demo-0.3-py2.4.egg

    -  demoneeded-1.1-py2.4.egg

    >>> rmdir(dest)

    >>> dest = tmpdir('sample-install')

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo'], dest, links=[link_server], index=link_server+'index/',

    ...     always_unzip=True)

    >>> ls(dest)

    d  demo-0.3-py2.4.egg

    d  demoneeded-1.1-py2.4.egg

Specifying version information independent of requirements

----------------------------------------------------------

Sometimes it's useful to specify version information independent of

normal requirements specifications.  For example, a buildout may need

to lock down a set of versions, without having to put put version

numbers in setup files or part definitions.  If a dictionary is passed

to the install function, mapping project names to version numbers,

then the versions numbers will be used.

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo'], dest, links=[link_server], index=link_server+'index/',

    ...     versions = dict(demo='0.2', demoneeded='1.0'))

    >>> [d.version for d in ws]

    ['0.2', '1.0']

In this example, we specified a version for demoneeded, even though we

didn't define a requirement for it.  The versions specified apply to

dependencies as well as the specified requirements.

If we specify a version that's incompatible with a requirement, then

we'll get an error:

    >>> from zope.testing.loggingsupport import InstalledHandler

    >>> handler = InstalledHandler('zc.buildout.easy_install')

    >>> import logging

    >>> logging.getLogger('zc.buildout.easy_install').propagate = False

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo >0.2'], dest, links=[link_server],

    ...     index=link_server+'index/',

    ...     versions = dict(demo='0.2', demoneeded='1.0'))

    Traceback (most recent call last):

    ...

    IncompatibleVersionError: Bad version 0.2

    >>> print handler

    zc.buildout.easy_install DEBUG

      Installing 'demo >0.2'.

    zc.buildout.easy_install ERROR

      The version, 0.2, is not consistent with the requirement, 'demo>0.2'.

    >>> handler.clear()

If no versions are specified, a debugging message will be output

reporting that a version was picked automatically:

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo'], dest, links=[link_server], index=link_server+'index/',

    ...     )

    >>> print handler

    zc.buildout.easy_install DEBUG

      Installing 'demo'.

    zc.buildout.easy_install DEBUG

      We have the best distribution that satisfies 'demo'.

    zc.buildout.easy_install DEBUG

      Picked: demo = 0.3

    zc.buildout.easy_install DEBUG

      Getting required 'demoneeded'

    zc.buildout.easy_install DEBUG

        required by demo 0.3.

    zc.buildout.easy_install DEBUG

      We have the best distribution that satisfies 'demoneeded'.

    zc.buildout.easy_install DEBUG

      Picked: demoneeded = 1.1

    >>> handler.uninstall()

    >>> logging.getLogger('zc.buildout.easy_install').propagate = True

We can request that we get an error if versions are picked:

    >>> zc.buildout.easy_install.allow_picked_versions(False)

    True

(The old setting is returned.)

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo'], dest, links=[link_server], index=link_server+'index/',

    ...     )

    Traceback (most recent call last):

    ...

    UserError: Picked: demo = 0.3

    >>> zc.buildout.easy_install.allow_picked_versions(True)

    False

The function default_versions can be used to get and set default

version information to be used when no version information is passes.

If called with an argument, it sets the default versions:

    >>> zc.buildout.easy_install.default_versions(dict(demoneeded='1'))

    {}

It always returns the previous default versions.  If called without an

argument, it simply returns the default versions without changing

them:

    >>> zc.buildout.easy_install.default_versions()

    {'demoneeded': '1'}

So with the default versions set, we'll get the requested version even

if the versions option isn't used:

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo'], dest, links=[link_server], index=link_server+'index/',

    ...     )

    >>> [d.version for d in ws]

    ['0.3', '1.0']

Of course, we can unset the default versions by passing an empty

dictionary:

    >>> zc.buildout.easy_install.default_versions({})

    {'demoneeded': '1'}

    >>> ws = zc.buildout.easy_install.install(

    ...     ['demo'], dest, links=[link_server], index=link_server+'index/',

    ...     )

    >>> [d.version for d in ws]

    ['0.3', '1.1']

Dependencies in Site Packages

-----------------------------

Paths outside of Python's standard library--or more precisely, those that are

not included when Python is started with the -S argument--are loosely referred

to as "site-packages" here.  These site-packages are searched by default for

distributions.  This can be disabled, so that, for instance, a system Python

can be used with buildout, cleaned of any packages installed by a user or

system package manager.

The default behavior can be controlled and introspected using

zc.buildout.easy_install.include_site_packages.

    >>> zc.buildout.easy_install.include_site_packages()

    True

Here's an example of using a Python executable that includes our dependencies.

Our "py_path" will have the "demoneeded," and "demo" packages available.

 We'll simply be asking for "demoneeded" here, but without any external

 index or links.

    >>> from zc.buildout.tests import create_sample_sys_install

    >>> py_path, site_packages_path = make_py()

    >>> create_sample_sys_install(site_packages_path)

    >>> example_dest = tmpdir('site-packages-example-install')

    >>> workingset = zc.buildout.easy_install.install(

    ...     ['demoneeded'], example_dest, links=[], executable=py_path,

    ...     index=None)

    >>> [dist.project_name for dist in workingset]

    ['demoneeded']

That worked fine.  Let's try again with site packages not allowed.  We'll

change the policy by changing the default.  Notice that the function for

changing the default value returns the previous value.

    >>> zc.buildout.easy_install.include_site_packages(False)

    True

    >>> zc.buildout.easy_install.include_site_packages()

    False

    >>> zc.buildout.easy_install.clear_index_cache()

    >>> rmdir(example_dest)

    >>> example_dest = tmpdir('site-packages-example-install')

    >>> workingset = zc.buildout.easy_install.install(

    ...     ['demoneeded'], example_dest, links=[], executable=py_path,

    ...     index=None)

    Traceback (most recent call last):

        ...

    MissingDistribution: Couldn't find a distribution for 'demoneeded'.

    >>> zc.buildout.easy_install.clear_index_cache()

Now we'll reset the default.

    >>> zc.buildout.easy_install.include_site_packages(True)

    False

    >>> zc.buildout.easy_install.include_site_packages()

    True

Dependency links

----------------

Setuptools allows metadata that describes where to search for package

dependencies. This option is called dependency_links. Buildout has its

own notion of where to look for dependencies, but it also uses the

setup tools dependency_links information if it's available.

Let's demo this by creating an egg that specifies dependency_links.

To begin, let's create a new egg repository. This repository hold a

newer version of the 'demoneeded' egg than the sample repository does.

    >>> repoloc = tmpdir('repo')

    >>> from zc.buildout.tests import create_egg

    >>> create_egg('demoneeded', '1.2', repoloc)

    >>> link_server2 = start_server(repoloc)

Turn on logging on this server so that we can see when eggs are pulled

from it.

    >>> get(link_server2 + 'enable_server_logging')

    GET 200 /enable_server_logging

    ''

Now we can create an egg that specifies that its dependencies are

found on this server.

    >>> repoloc = tmpdir('repo2')

    >>> create_egg('hasdeps', '1.0', repoloc,

    ...            install_requires = "'demoneeded'",

    ...            dependency_links = [link_server2])

Let's add the egg to another repository.

    >>> link_server3 = start_server(repoloc)

Now let's install the egg.

    >>> example_dest = tmpdir('example-install')

    >>> workingset = zc.buildout.easy_install.install(

    ...     ['hasdeps'], example_dest,

    ...     links=[link_server3], index=link_server3+'index/')

    GET 200 /

    GET 200 /demoneeded-1.2-pyN.N.egg

The server logs show that the dependency was retrieved from the server

specified in the dependency_links.

Now let's see what happens if we provide two different ways to retrieve

the dependencies.

    >>> rmdir(example_dest)

    >>> example_dest = tmpdir('example-install')

    >>> workingset = zc.buildout.easy_install.install(

    ...     ['hasdeps'], example_dest, index=link_server+'index/',