Caching extended configuration
==============================
As mentioned in the general buildout documentation, configuration files can
extend each other, including the ability to download configuration being
extended from a URL. If desired, zc.buildout caches downloaded configuration
in order to be able to use it when run offline.
As we're going to talk about downloading things, let's start an HTTP server.
Also, all of the following will take place inside the sample buildout.
>>> server_data = tmpdir('server_data')
>>> server_url = start_server(server_data)
>>> cd(sample_buildout)
We also use a fresh directory for temporary files in order to make sure that
all temporary files have been cleaned up in the end:
>>> import tempfile
>>> old_tempdir = tempfile.tempdir
>>> tempfile.tempdir = tmpdir('tmp')
Basic use of the extends cache
------------------------------
We put some base configuration on a server and reference it from a sample
buildout:
>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... foo = bar
... """)
>>> write('buildout.cfg', """\
... [buildout]
... extends = %sbase.cfg
... """ % server_url)
When trying to run this buildout offline, we'll find that we cannot read all
of the required configuration:
>>> print system(buildout + ' -o')
While:
Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.
Trying the same online, we can:
>>> print system(buildout)
Unused options for buildout: 'foo'.
As long as we haven't said anything about caching downloaded configuration,
nothing gets cached. Offline mode will still cause the buildout to fail:
>>> print system(buildout + ' -o')
While:
Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.
Let's now specify a cache for base configuration files. This cache is
different from the download cache used by recipes for caching distributions
and other files; one might, however, use a namespace subdirectory of the
download cache for it. The configuration cache we specify will be created when
running buildout and the base.cfg file will be put in it (with the file name
being a hash of the complete URL):
>>> mkdir('cache')
>>> write('buildout.cfg', """\
... [buildout]
... extends = %sbase.cfg
... extends-cache = cache
... """ % server_url)
>>> print system(buildout)
Unused options for buildout: 'foo'.
>>> cache = join(sample_buildout, 'cache')
>>> ls(cache)
- 5aedc98d7e769290a29d654a591a3a45
>>> import os
>>> cat(cache, os.listdir(cache)[0])
[buildout]
parts =
foo = bar
We can now run buildout offline as it will read base.cfg from the cache:
>>> print system(buildout + ' -o')
Unused options for buildout: 'foo'.
The cache is being used purely as a fall-back in case we are offline or don't
have access to a configuration file to be downloaded. As long as we are
online, buildout attempts to download a fresh copy of each file even if a
cached copy of the file exists. To see this, we put different configuration in
the same place on the server and run buildout in offline mode so it takes
base.cfg from the cache:
>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... bar = baz
... """)
>>> print system(buildout + ' -o')
Unused options for buildout: 'foo'.
In online mode, buildout will download and use the modified version:
>>> print system(buildout)
Unused options for buildout: 'bar'.
Trying offline mode again, the new version will be used as it has been put in
the cache now:
>>> print system(buildout + ' -o')
Unused options for buildout: 'bar'.
Clean up:
>>> rmdir(cache)
Specifying extends cache and offline mode
-----------------------------------------
Normally, the values of buildout options such as the location of a download
cache or whether to use offline mode are determined by first reading the
user's default configuration, updating it with the project's configuration and
finally applying command-line options. User and project configuration are
assembled by reading a file such as ``~/.buildout/default.cfg``,
``buildout.cfg`` or a URL given on the command line, recursively (depth-first)
downloading any base configuration specified by the ``buildout:extends``
option read from each of those config files, and finally evaluating each
config file to provide default values for options not yet read.
This works fine for all options that do not influence how configuration is
downloaded in the first place. The ``extends-cache`` and ``offline`` options,
however, are treated differently from the procedure described in order to make
it simple and obvious to see where a particular configuration file came from
under any particular circumstances.
- Offline and extends-cache settings are read from the two root config files
exclusively. Otherwise one could construct configuration files that, when
read, imply that they should have been read from a different source than
they have. Also, specifying the extends cache within a file that might have
to be taken from the cache before being read wouldn't make a lot of sense.
- Offline and extends-cache settings given by the user's defaults apply to the
process of assembling the project's configuration. If no extends cache has
been specified by the user's default configuration, the project's root
config file must be available, be it from disk or from the net.
- Offline mode turned on by the ``-o`` command line option is honoured from
the beginning even though command line options are applied to the
configuration last. If offline mode is not requested by the command line, it
may be switched on by either the user's or the project's config root.
Extends cache
~~~~~~~~~~~~~
Let's see the above rules in action. We create a new home directory for our
user and write user and project configuration that recursively extends online
bases, using different caches:
>>> mkdir('home')
>>> mkdir('home', '.buildout')
>>> mkdir('cache')
>>> mkdir('user-cache')
>>> os.environ['HOME'] = join(sample_buildout, 'home')
>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... extends-cache = user-cache
... """)
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... """ % server_url)
>>> write(server_data, 'base_default.cfg', """\
... [buildout]
... foo = bar
... offline = false
... """)
>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... extends-cache = cache
... """)
>>> write('fancy.cfg', """\
... [buildout]
... extends = %sbase.cfg
... """ % server_url)
>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... offline = false
... """)
Buildout will now assemble its configuration from all of these 6 files,
defaults first. The online resources end up in the respective extends caches:
>>> print system(buildout)
Unused options for buildout: 'foo'.
>>> ls('user-cache')
- 10e772cf422123ef6c64ae770f555740
>>> cat('user-cache', os.listdir('user-cache')[0])
[buildout]
foo = bar
offline = false
>>> ls('cache')
- c72213127e6eb2208a3e1fc1dba771a7
>>> cat('cache', os.listdir('cache')[0])
[buildout]
parts =
offline = false
If, on the other hand, the extends caches are specified in files that get
extended themselves, they won't be used for assembling the configuration they
belong to (user's or project's, resp.). The extends cache specified by the
user's defaults does, however, apply to downloading project configuration.
Let's rewrite the config files, clean out the caches and re-run buildout:
>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... """)
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... extends-cache = user-cache
... """ % server_url)
>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... """)
>>> write('fancy.cfg', """\
... [buildout]
... extends = %sbase.cfg
... extends-cache = cache
... """ % server_url)
>>> remove('user-cache', os.listdir('user-cache')[0])
>>> remove('cache', os.listdir('cache')[0])
>>> print system(buildout)
Unused options for buildout: 'foo'.
>>> ls('user-cache')
- 0548bad6002359532de37385bb532e26
>>> cat('user-cache', os.listdir('user-cache')[0])
[buildout]
parts =
offline = false
>>> ls('cache')
Clean up:
>>> rmdir('user-cache')
>>> rmdir('cache')
Offline mode and installation from cache
----------------------------~~~~~~~~~~~~
If we run buildout in offline mode now, it will fail because it cannot get at
the remote configuration file needed by the user's defaults:
>>> print system(buildout + ' -o')
While:
Initializing.
Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.
Let's now successively turn on offline mode by different parts of the
configuration and see when buildout applies this setting in each case:
>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... offline = true
... """)
>>> print system(buildout)
While:
Initializing.
Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.
>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... """)
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... offline = true
... """ % server_url)
>>> print system(buildout)
While:
Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... """ % server_url)
>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... offline = true
... """)
>>> print system(buildout)
While:
Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.
>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... """)
>>> write('fancy.cfg', """\
... [buildout]
... extends = %sbase.cfg
... offline = true
... """ % server_url)
>>> print system(buildout)
Unused options for buildout: 'foo'.
The ``install-from-cache`` option is treated accordingly:
>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... install-from-cache = true
... """)
>>> print system(buildout)
While:
Initializing.
Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.
>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... """)
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... install-from-cache = true
... """ % server_url)
>>> print system(buildout)
While:
Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... """ % server_url)
>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... install-from-cache = true
... """)
>>> print system(buildout)
While:
Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.
>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... """)
>>> write('fancy.cfg', """\
... [buildout]
... extends = %sbase.cfg
... install-from-cache = true
... """ % server_url)
>>> print system(buildout)
While:
Installing.
Checking for upgrades.
An internal error occurred ...
ValueError: install_from_cache set to true with no download cache
Clean up
--------
We should have cleaned up all temporary files created by downloading things:
>>> ls(tempfile.tempdir)
Reset the global temporary directory:
>>> tempfile.tempdir = old_tempdir