|
1 Caching extended configuration |
|
2 ============================== |
|
3 |
|
4 As mentioned in the general buildout documentation, configuration files can |
|
5 extend each other, including the ability to download configuration being |
|
6 extended from a URL. If desired, zc.buildout caches downloaded configuration |
|
7 in order to be able to use it when run offline. |
|
8 |
|
9 As we're going to talk about downloading things, let's start an HTTP server. |
|
10 Also, all of the following will take place inside the sample buildout. |
|
11 |
|
12 >>> server_data = tmpdir('server_data') |
|
13 >>> server_url = start_server(server_data) |
|
14 >>> cd(sample_buildout) |
|
15 |
|
16 We also use a fresh directory for temporary files in order to make sure that |
|
17 all temporary files have been cleaned up in the end: |
|
18 |
|
19 >>> import tempfile |
|
20 >>> old_tempdir = tempfile.tempdir |
|
21 >>> tempfile.tempdir = tmpdir('tmp') |
|
22 |
|
23 |
|
24 Basic use of the extends cache |
|
25 ------------------------------ |
|
26 |
|
27 We put some base configuration on a server and reference it from a sample |
|
28 buildout: |
|
29 |
|
30 >>> write(server_data, 'base.cfg', """\ |
|
31 ... [buildout] |
|
32 ... parts = |
|
33 ... foo = bar |
|
34 ... """) |
|
35 |
|
36 >>> write('buildout.cfg', """\ |
|
37 ... [buildout] |
|
38 ... extends = %sbase.cfg |
|
39 ... """ % server_url) |
|
40 |
|
41 When trying to run this buildout offline, we'll find that we cannot read all |
|
42 of the required configuration: |
|
43 |
|
44 >>> print system(buildout + ' -o') |
|
45 While: |
|
46 Initializing. |
|
47 Error: Couldn't download 'http://localhost/base.cfg' in offline mode. |
|
48 |
|
49 Trying the same online, we can: |
|
50 |
|
51 >>> print system(buildout) |
|
52 Unused options for buildout: 'foo'. |
|
53 |
|
54 As long as we haven't said anything about caching downloaded configuration, |
|
55 nothing gets cached. Offline mode will still cause the buildout to fail: |
|
56 |
|
57 >>> print system(buildout + ' -o') |
|
58 While: |
|
59 Initializing. |
|
60 Error: Couldn't download 'http://localhost/base.cfg' in offline mode. |
|
61 |
|
62 Let's now specify a cache for base configuration files. This cache is |
|
63 different from the download cache used by recipes for caching distributions |
|
64 and other files; one might, however, use a namespace subdirectory of the |
|
65 download cache for it. The configuration cache we specify will be created when |
|
66 running buildout and the base.cfg file will be put in it (with the file name |
|
67 being a hash of the complete URL): |
|
68 |
|
69 >>> mkdir('cache') |
|
70 >>> write('buildout.cfg', """\ |
|
71 ... [buildout] |
|
72 ... extends = %sbase.cfg |
|
73 ... extends-cache = cache |
|
74 ... """ % server_url) |
|
75 |
|
76 >>> print system(buildout) |
|
77 Unused options for buildout: 'foo'. |
|
78 |
|
79 >>> cache = join(sample_buildout, 'cache') |
|
80 >>> ls(cache) |
|
81 - 5aedc98d7e769290a29d654a591a3a45 |
|
82 |
|
83 >>> import os |
|
84 >>> cat(cache, os.listdir(cache)[0]) |
|
85 [buildout] |
|
86 parts = |
|
87 foo = bar |
|
88 |
|
89 We can now run buildout offline as it will read base.cfg from the cache: |
|
90 |
|
91 >>> print system(buildout + ' -o') |
|
92 Unused options for buildout: 'foo'. |
|
93 |
|
94 The cache is being used purely as a fall-back in case we are offline or don't |
|
95 have access to a configuration file to be downloaded. As long as we are |
|
96 online, buildout attempts to download a fresh copy of each file even if a |
|
97 cached copy of the file exists. To see this, we put different configuration in |
|
98 the same place on the server and run buildout in offline mode so it takes |
|
99 base.cfg from the cache: |
|
100 |
|
101 >>> write(server_data, 'base.cfg', """\ |
|
102 ... [buildout] |
|
103 ... parts = |
|
104 ... bar = baz |
|
105 ... """) |
|
106 |
|
107 >>> print system(buildout + ' -o') |
|
108 Unused options for buildout: 'foo'. |
|
109 |
|
110 In online mode, buildout will download and use the modified version: |
|
111 |
|
112 >>> print system(buildout) |
|
113 Unused options for buildout: 'bar'. |
|
114 |
|
115 Trying offline mode again, the new version will be used as it has been put in |
|
116 the cache now: |
|
117 |
|
118 >>> print system(buildout + ' -o') |
|
119 Unused options for buildout: 'bar'. |
|
120 |
|
121 Clean up: |
|
122 |
|
123 >>> rmdir(cache) |
|
124 |
|
125 |
|
126 Specifying extends cache and offline mode |
|
127 ----------------------------------------- |
|
128 |
|
129 Normally, the values of buildout options such as the location of a download |
|
130 cache or whether to use offline mode are determined by first reading the |
|
131 user's default configuration, updating it with the project's configuration and |
|
132 finally applying command-line options. User and project configuration are |
|
133 assembled by reading a file such as ``~/.buildout/default.cfg``, |
|
134 ``buildout.cfg`` or a URL given on the command line, recursively (depth-first) |
|
135 downloading any base configuration specified by the ``buildout:extends`` |
|
136 option read from each of those config files, and finally evaluating each |
|
137 config file to provide default values for options not yet read. |
|
138 |
|
139 This works fine for all options that do not influence how configuration is |
|
140 downloaded in the first place. The ``extends-cache`` and ``offline`` options, |
|
141 however, are treated differently from the procedure described in order to make |
|
142 it simple and obvious to see where a particular configuration file came from |
|
143 under any particular circumstances. |
|
144 |
|
145 - Offline and extends-cache settings are read from the two root config files |
|
146 exclusively. Otherwise one could construct configuration files that, when |
|
147 read, imply that they should have been read from a different source than |
|
148 they have. Also, specifying the extends cache within a file that might have |
|
149 to be taken from the cache before being read wouldn't make a lot of sense. |
|
150 |
|
151 - Offline and extends-cache settings given by the user's defaults apply to the |
|
152 process of assembling the project's configuration. If no extends cache has |
|
153 been specified by the user's default configuration, the project's root |
|
154 config file must be available, be it from disk or from the net. |
|
155 |
|
156 - Offline mode turned on by the ``-o`` command line option is honoured from |
|
157 the beginning even though command line options are applied to the |
|
158 configuration last. If offline mode is not requested by the command line, it |
|
159 may be switched on by either the user's or the project's config root. |
|
160 |
|
161 Extends cache |
|
162 ~~~~~~~~~~~~~ |
|
163 |
|
164 Let's see the above rules in action. We create a new home directory for our |
|
165 user and write user and project configuration that recursively extends online |
|
166 bases, using different caches: |
|
167 |
|
168 >>> mkdir('home') |
|
169 >>> mkdir('home', '.buildout') |
|
170 >>> mkdir('cache') |
|
171 >>> mkdir('user-cache') |
|
172 >>> os.environ['HOME'] = join(sample_buildout, 'home') |
|
173 >>> write('home', '.buildout', 'default.cfg', """\ |
|
174 ... [buildout] |
|
175 ... extends = fancy_default.cfg |
|
176 ... extends-cache = user-cache |
|
177 ... """) |
|
178 >>> write('home', '.buildout', 'fancy_default.cfg', """\ |
|
179 ... [buildout] |
|
180 ... extends = %sbase_default.cfg |
|
181 ... """ % server_url) |
|
182 >>> write(server_data, 'base_default.cfg', """\ |
|
183 ... [buildout] |
|
184 ... foo = bar |
|
185 ... offline = false |
|
186 ... """) |
|
187 |
|
188 >>> write('buildout.cfg', """\ |
|
189 ... [buildout] |
|
190 ... extends = fancy.cfg |
|
191 ... extends-cache = cache |
|
192 ... """) |
|
193 >>> write('fancy.cfg', """\ |
|
194 ... [buildout] |
|
195 ... extends = %sbase.cfg |
|
196 ... """ % server_url) |
|
197 >>> write(server_data, 'base.cfg', """\ |
|
198 ... [buildout] |
|
199 ... parts = |
|
200 ... offline = false |
|
201 ... """) |
|
202 |
|
203 Buildout will now assemble its configuration from all of these 6 files, |
|
204 defaults first. The online resources end up in the respective extends caches: |
|
205 |
|
206 >>> print system(buildout) |
|
207 Unused options for buildout: 'foo'. |
|
208 |
|
209 >>> ls('user-cache') |
|
210 - 10e772cf422123ef6c64ae770f555740 |
|
211 >>> cat('user-cache', os.listdir('user-cache')[0]) |
|
212 [buildout] |
|
213 foo = bar |
|
214 offline = false |
|
215 |
|
216 >>> ls('cache') |
|
217 - c72213127e6eb2208a3e1fc1dba771a7 |
|
218 >>> cat('cache', os.listdir('cache')[0]) |
|
219 [buildout] |
|
220 parts = |
|
221 offline = false |
|
222 |
|
223 If, on the other hand, the extends caches are specified in files that get |
|
224 extended themselves, they won't be used for assembling the configuration they |
|
225 belong to (user's or project's, resp.). The extends cache specified by the |
|
226 user's defaults does, however, apply to downloading project configuration. |
|
227 Let's rewrite the config files, clean out the caches and re-run buildout: |
|
228 |
|
229 >>> write('home', '.buildout', 'default.cfg', """\ |
|
230 ... [buildout] |
|
231 ... extends = fancy_default.cfg |
|
232 ... """) |
|
233 >>> write('home', '.buildout', 'fancy_default.cfg', """\ |
|
234 ... [buildout] |
|
235 ... extends = %sbase_default.cfg |
|
236 ... extends-cache = user-cache |
|
237 ... """ % server_url) |
|
238 |
|
239 >>> write('buildout.cfg', """\ |
|
240 ... [buildout] |
|
241 ... extends = fancy.cfg |
|
242 ... """) |
|
243 >>> write('fancy.cfg', """\ |
|
244 ... [buildout] |
|
245 ... extends = %sbase.cfg |
|
246 ... extends-cache = cache |
|
247 ... """ % server_url) |
|
248 |
|
249 >>> remove('user-cache', os.listdir('user-cache')[0]) |
|
250 >>> remove('cache', os.listdir('cache')[0]) |
|
251 |
|
252 >>> print system(buildout) |
|
253 Unused options for buildout: 'foo'. |
|
254 |
|
255 >>> ls('user-cache') |
|
256 - 0548bad6002359532de37385bb532e26 |
|
257 >>> cat('user-cache', os.listdir('user-cache')[0]) |
|
258 [buildout] |
|
259 parts = |
|
260 offline = false |
|
261 |
|
262 >>> ls('cache') |
|
263 |
|
264 Clean up: |
|
265 |
|
266 >>> rmdir('user-cache') |
|
267 >>> rmdir('cache') |
|
268 |
|
269 Offline mode and installation from cache |
|
270 ----------------------------~~~~~~~~~~~~ |
|
271 |
|
272 If we run buildout in offline mode now, it will fail because it cannot get at |
|
273 the remote configuration file needed by the user's defaults: |
|
274 |
|
275 >>> print system(buildout + ' -o') |
|
276 While: |
|
277 Initializing. |
|
278 Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode. |
|
279 |
|
280 Let's now successively turn on offline mode by different parts of the |
|
281 configuration and see when buildout applies this setting in each case: |
|
282 |
|
283 >>> write('home', '.buildout', 'default.cfg', """\ |
|
284 ... [buildout] |
|
285 ... extends = fancy_default.cfg |
|
286 ... offline = true |
|
287 ... """) |
|
288 >>> print system(buildout) |
|
289 While: |
|
290 Initializing. |
|
291 Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode. |
|
292 |
|
293 >>> write('home', '.buildout', 'default.cfg', """\ |
|
294 ... [buildout] |
|
295 ... extends = fancy_default.cfg |
|
296 ... """) |
|
297 >>> write('home', '.buildout', 'fancy_default.cfg', """\ |
|
298 ... [buildout] |
|
299 ... extends = %sbase_default.cfg |
|
300 ... offline = true |
|
301 ... """ % server_url) |
|
302 >>> print system(buildout) |
|
303 While: |
|
304 Initializing. |
|
305 Error: Couldn't download 'http://localhost/base.cfg' in offline mode. |
|
306 |
|
307 >>> write('home', '.buildout', 'fancy_default.cfg', """\ |
|
308 ... [buildout] |
|
309 ... extends = %sbase_default.cfg |
|
310 ... """ % server_url) |
|
311 >>> write('buildout.cfg', """\ |
|
312 ... [buildout] |
|
313 ... extends = fancy.cfg |
|
314 ... offline = true |
|
315 ... """) |
|
316 >>> print system(buildout) |
|
317 While: |
|
318 Initializing. |
|
319 Error: Couldn't download 'http://localhost/base.cfg' in offline mode. |
|
320 |
|
321 >>> write('buildout.cfg', """\ |
|
322 ... [buildout] |
|
323 ... extends = fancy.cfg |
|
324 ... """) |
|
325 >>> write('fancy.cfg', """\ |
|
326 ... [buildout] |
|
327 ... extends = %sbase.cfg |
|
328 ... offline = true |
|
329 ... """ % server_url) |
|
330 >>> print system(buildout) |
|
331 Unused options for buildout: 'foo'. |
|
332 |
|
333 The ``install-from-cache`` option is treated accordingly: |
|
334 |
|
335 >>> write('home', '.buildout', 'default.cfg', """\ |
|
336 ... [buildout] |
|
337 ... extends = fancy_default.cfg |
|
338 ... install-from-cache = true |
|
339 ... """) |
|
340 >>> print system(buildout) |
|
341 While: |
|
342 Initializing. |
|
343 Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode. |
|
344 |
|
345 >>> write('home', '.buildout', 'default.cfg', """\ |
|
346 ... [buildout] |
|
347 ... extends = fancy_default.cfg |
|
348 ... """) |
|
349 >>> write('home', '.buildout', 'fancy_default.cfg', """\ |
|
350 ... [buildout] |
|
351 ... extends = %sbase_default.cfg |
|
352 ... install-from-cache = true |
|
353 ... """ % server_url) |
|
354 >>> print system(buildout) |
|
355 While: |
|
356 Initializing. |
|
357 Error: Couldn't download 'http://localhost/base.cfg' in offline mode. |
|
358 |
|
359 >>> write('home', '.buildout', 'fancy_default.cfg', """\ |
|
360 ... [buildout] |
|
361 ... extends = %sbase_default.cfg |
|
362 ... """ % server_url) |
|
363 >>> write('buildout.cfg', """\ |
|
364 ... [buildout] |
|
365 ... extends = fancy.cfg |
|
366 ... install-from-cache = true |
|
367 ... """) |
|
368 >>> print system(buildout) |
|
369 While: |
|
370 Initializing. |
|
371 Error: Couldn't download 'http://localhost/base.cfg' in offline mode. |
|
372 |
|
373 >>> write('buildout.cfg', """\ |
|
374 ... [buildout] |
|
375 ... extends = fancy.cfg |
|
376 ... """) |
|
377 >>> write('fancy.cfg', """\ |
|
378 ... [buildout] |
|
379 ... extends = %sbase.cfg |
|
380 ... install-from-cache = true |
|
381 ... """ % server_url) |
|
382 >>> print system(buildout) |
|
383 While: |
|
384 Installing. |
|
385 Checking for upgrades. |
|
386 An internal error occurred ... |
|
387 ValueError: install_from_cache set to true with no download cache |
|
388 |
|
389 |
|
390 Clean up |
|
391 -------- |
|
392 |
|
393 We should have cleaned up all temporary files created by downloading things: |
|
394 |
|
395 >>> ls(tempfile.tempdir) |
|
396 |
|
397 Reset the global temporary directory: |
|
398 |
|
399 >>> tempfile.tempdir = old_tempdir |