|
1 Version 1.5.0 of buildout (and higher) provides the ability to use |
|
2 buildout directly with a system Python if you use z3c.recipe.scripts or |
|
3 other isolation-aware recipes that use the sitepackage_safe_scripts function. |
|
4 |
|
5 Some people use virtualenv to provide similar functionality. |
|
6 Unfortunately, a problem with the virtualenv executable as of this |
|
7 writing means that -S will not work properly with it (see |
|
8 https://bugs.launchpad.net/virtualenv/+bug/572545). This breaks |
|
9 buildout's approach to providing isolation. |
|
10 |
|
11 Because of this, if buildout detects an executable with a broken -S |
|
12 option, it will revert to its pre-1.5.0 behavior. If buildout has been |
|
13 asked to provide isolation, it will warn the user that isolation will |
|
14 not be provided by buildout, but proceed. This should give full |
|
15 backwards compatibility to virtualenv users. |
|
16 |
|
17 The only minor annoyance in the future may be recipes that explicitly |
|
18 use the new buildout functionality to provide isolation: as described |
|
19 above, the builds will proceed, but users will receive warnings that |
|
20 buildout is not providing isolation itself. The warnings themselves can |
|
21 be squelched when running bin/buildout with the ``-s`` option or with a |
|
22 lower verbosity than usual (e.g., one or more ``-q`` options). |
|
23 |
|
24 For tests, then, we can examine several things. We'll focus on four. |
|
25 |
|
26 - Running bootstrap with an executable broken in this way will not try to do |
|
27 any -S tricks. |
|
28 |
|
29 - Running sitepackage_safe_scripts with a virtualenv will create an |
|
30 old-style script. This will affect the bin/buildout script that is |
|
31 created, for instance. If the sitepackage_safe_scripts function is asked |
|
32 to provide isolation under these circumstances, it will warn that isolation |
|
33 will not be available, but still create the desired script. |
|
34 |
|
35 - Using the easy_install Installer or install or build functions and trying |
|
36 to request isolation will generate a warning and then the isolation request |
|
37 will be ignored as it proceeds. |
|
38 |
|
39 - Passing -s (or -q) to the bin/buildout script will squelch warnings. |
|
40 |
|
41 Testing these involves first creating a Python that exhibits the same |
|
42 behavior as the problematic one we care about from virtualenv. Let's do that |
|
43 first. |
|
44 |
|
45 >>> import os, sys |
|
46 >>> from zc.buildout.easy_install import _safe_arg |
|
47 >>> py_path, site_packages_path = make_py() |
|
48 >>> if sys.platform == 'win32': |
|
49 ... py_script_path = py_path + '-script.py' |
|
50 ... else: |
|
51 ... py_script_path = py_path |
|
52 ... |
|
53 >>> py_file = open(py_script_path) |
|
54 >>> py_lines = py_file.readlines() |
|
55 >>> py_file.close() |
|
56 >>> py_file = open(py_script_path, 'w') |
|
57 >>> extra = '''\ |
|
58 ... new_argv = argv[:1] |
|
59 ... for ix, val in enumerate(argv[1:]): |
|
60 ... if val.startswith('--'): |
|
61 ... new_argv.append(val) |
|
62 ... if val.startswith('-') and len(val) > 1: |
|
63 ... if 'S' in val: |
|
64 ... val = val.replace('S', '') |
|
65 ... environ['BROKEN_DASH_S'] = 'Y' |
|
66 ... if val != '-': |
|
67 ... new_argv.append(val) |
|
68 ... if 'c' in val: |
|
69 ... new_argv.extend(argv[ix+2:]) |
|
70 ... break |
|
71 ... else: |
|
72 ... new_argv.extend(argv[ix+1:]) |
|
73 ... argv = new_argv |
|
74 ... ''' |
|
75 >>> for line in py_lines: |
|
76 ... py_file.write(line) |
|
77 ... if line.startswith('environ = os.environ.copy()'): |
|
78 ... py_file.write(extra) |
|
79 ... print 'Rewritten.' |
|
80 ... |
|
81 Rewritten. |
|
82 >>> py_file.close() |
|
83 >>> sitecustomize_path = join(os.path.dirname(site_packages_path), |
|
84 ... 'parts', 'py', 'sitecustomize.py') |
|
85 >>> sitecustomize_file = open(sitecustomize_path, 'a') |
|
86 >>> sitecustomize_file.write(''' |
|
87 ... import os, sys |
|
88 ... sys.executable = %r |
|
89 ... if 'BROKEN_DASH_S' in os.environ: |
|
90 ... class ImportHook: |
|
91 ... site = None |
|
92 ... |
|
93 ... @classmethod |
|
94 ... def find_module(klass, fullname, path=None): |
|
95 ... if klass.site is None and 'site' in sys.modules: |
|
96 ... # Pop site out of sys.modules. This will be a |
|
97 ... # close-enough approximation of site not being |
|
98 ... # loaded for our tests--it lets us provoke the |
|
99 ... # right errors when the fixes are absent, and |
|
100 ... # works well enough when the fixes are present. |
|
101 ... klass.site = sys.modules.pop('site') |
|
102 ... if fullname == 'ConfigParser': |
|
103 ... raise ImportError(fullname) |
|
104 ... elif fullname == 'site': |
|
105 ... # Keep the site module from being processed twice. |
|
106 ... return klass |
|
107 ... |
|
108 ... @classmethod |
|
109 ... def load_module(klass, fullname): |
|
110 ... if fullname == 'site': |
|
111 ... return klass.site |
|
112 ... raise ImportError(fullname) |
|
113 ... |
|
114 ... sys.meta_path.append(ImportHook) |
|
115 ... ''' % (py_path,)) |
|
116 >>> sitecustomize_file.close() |
|
117 >>> print call_py( |
|
118 ... _safe_arg(py_path), |
|
119 ... "import ConfigParser") |
|
120 <BLANKLINE> |
|
121 >>> print 'X'; print call_py( |
|
122 ... _safe_arg(py_path), |
|
123 ... "import ConfigParser", |
|
124 ... '-S') # doctest: +ELLIPSIS |
|
125 X...Traceback (most recent call last): |
|
126 ... |
|
127 ImportError: No module named ConfigParser |
|
128 <BLANKLINE> |
|
129 >>> from zc.buildout.easy_install import _has_broken_dash_S |
|
130 >>> _has_broken_dash_S(py_path) |
|
131 True |
|
132 |
|
133 Well, that was ugly, but it seems to have done the trick. The |
|
134 executable represented by py_path has the same problematic |
|
135 characteristic as the virtualenv one: -S results in a Python that does |
|
136 not allow the import of some packages from the standard library. We'll |
|
137 test with this. |
|
138 |
|
139 First, let's try running bootstrap. |
|
140 |
|
141 >>> from os.path import dirname, join |
|
142 >>> import zc.buildout |
|
143 >>> bootstrap_py = join( |
|
144 ... dirname( |
|
145 ... dirname( |
|
146 ... dirname( |
|
147 ... dirname(zc.buildout.__file__) |
|
148 ... ) |
|
149 ... ) |
|
150 ... ), |
|
151 ... 'bootstrap', 'bootstrap.py') |
|
152 >>> broken_S_buildout = tmpdir('broken_S') |
|
153 >>> os.chdir(broken_S_buildout) |
|
154 >>> write('buildout.cfg', |
|
155 ... ''' |
|
156 ... [buildout] |
|
157 ... parts = |
|
158 ... ''') |
|
159 >>> write('bootstrap.py', open(bootstrap_py).read()) |
|
160 >>> print 'X'; print system( |
|
161 ... _safe_arg(py_path)+' '+ |
|
162 ... 'bootstrap.py'); print 'X' # doctest: +ELLIPSIS |
|
163 X... |
|
164 Generated script '/broken_S/bin/buildout'. |
|
165 ... |
|
166 |
|
167 If bootstrap didn't look out for a broken -S, that would have failed. Moreover, |
|
168 take a look at bin/buildout: |
|
169 |
|
170 >>> cat('bin', 'buildout') # doctest: +NORMALIZE_WHITESPACE |
|
171 #!/executable_buildout/bin/py |
|
172 <BLANKLINE> |
|
173 import sys |
|
174 sys.path[0:0] = [ |
|
175 '/broken_S/eggs/setuptools-0.0-pyN.N.egg', |
|
176 '/broken_S/eggs/zc.buildout-0.0-pyN.N.egg', |
|
177 ] |
|
178 <BLANKLINE> |
|
179 import zc.buildout.buildout |
|
180 <BLANKLINE> |
|
181 if __name__ == '__main__': |
|
182 zc.buildout.buildout.main() |
|
183 |
|
184 That's the old-style buildout script: no changes for users with this issue. |
|
185 |
|
186 Of course, they don't get the new features either, presumably because |
|
187 they don't need or want them. This means that if they use a recipe that |
|
188 tries to use a new feature, the behavior needs to degrade gracefully. |
|
189 |
|
190 Here's an example. We'll switch to another buildout in which it is easier to |
|
191 use local dev versions of zc.buildout and z3c.recipe.scripts. |
|
192 |
|
193 >>> os.chdir(dirname(dirname(buildout))) |
|
194 >>> write('buildout.cfg', |
|
195 ... ''' |
|
196 ... [buildout] |
|
197 ... parts = eggs |
|
198 ... find-links = %(link_server)s |
|
199 ... include-site-packages = false |
|
200 ... |
|
201 ... [primed_python] |
|
202 ... executable = %(py_path)s |
|
203 ... |
|
204 ... [eggs] |
|
205 ... recipe = z3c.recipe.scripts |
|
206 ... python = primed_python |
|
207 ... interpreter = py |
|
208 ... eggs = demo |
|
209 ... ''' % globals()) |
|
210 |
|
211 >>> print system(buildout), # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS |
|
212 Installing eggs. |
|
213 Getting distribution for 'demo'. |
|
214 Got demo 0.4c1. |
|
215 Getting distribution for 'demoneeded'. |
|
216 Got demoneeded 1.2c1. |
|
217 Generated script '/sample-buildout/bin/demo'. |
|
218 Generated interpreter '/sample-buildout/bin/py'. |
|
219 ...UserWarning: Buildout has been asked to exclude or limit site-packages |
|
220 so that builds can be repeatable when using a system Python. However, |
|
221 the chosen Python executable has a broken implementation of -S (see |
|
222 https://bugs.launchpad.net/virtualenv/+bug/572545 for an example |
|
223 problem) and this breaks buildout's ability to isolate site-packages. |
|
224 If the executable already has a clean site-packages (e.g., using |
|
225 virtualenv's ``--no-site-packages`` option) you may be getting |
|
226 equivalent repeatability. To silence this warning, use the -s argument |
|
227 to the buildout script. Alternatively, use a Python executable with a |
|
228 working -S (such as a standard Python binary). |
|
229 warnings.warn(BROKEN_DASH_S_WARNING) |
|
230 |
|
231 So, it did what we asked as best it could, but gave a big warning. If |
|
232 you don't want those warnings for those particular recipes that use the |
|
233 new features, you can use the "-s" option to squelch the warnings. |
|
234 |
|
235 >>> print system(buildout + ' -s'), |
|
236 Updating eggs. |
|
237 |
|
238 A lower verbosity (one or more -q options) also quiets the warning. |
|
239 |
|
240 >>> print system(buildout + ' -q'), |
|
241 |
|
242 Notice that, as we saw before with bin/buildout, the generated scripts |
|
243 are old-style, because the new-style feature gracefully degrades to the |
|
244 previous implementation when it encounters an executable with a broken |
|
245 dash-S. |
|
246 |
|
247 >>> print 'X'; cat('bin', 'py') # doctest: +ELLIPSIS |
|
248 X... |
|
249 <BLANKLINE> |
|
250 import sys |
|
251 <BLANKLINE> |
|
252 sys.path[0:0] = [ |
|
253 '/sample-buildout/eggs/demo-0.4c1-pyN.N.egg', |
|
254 '/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg', |
|
255 ] |
|
256 ... |
|
257 |