October 2018 - openSUSE Packaging - openSUSE Mailing Lists

[opensuse-packaging] Proposal to remove pyc/pyo from Python on TW
by Alberto Planas Dominguez 06 Oct '18

06 Oct '18

Hi, As you know the Python packages are collecting the pyc/pyo precompiled binaries inside the RPM. This is mostly a good idea, as makes the first execution of the Python code faster, as is skipped the stage where the interpreter compile the .py code. But this also makes the Python stack a bit fat. Most of the time this is not a problem, but things are changing. We have JeOS and MicroOS, both minimal images (build with different goals and technology) that search for be small and slim. But we want to include a bit of Python in there, like salt-minion or cloud-init. And now the relative size of Python is evident. For Python 2.7 and 3.7 is possible to remove the pyc code from the system and instruct the interpreter to avoid the recreation of the pyc once the code is executed. The Python interpreter, by default, will compile and store the pyc in the disk for each `import`, but this behavior can be disable when we call Python. But this will make the initial execution of a big Python stack a bit slow, as the pyc needs to be recreated in memory for each invocation. The slowness can be relevant in some situations, so is better to not enable this feature. But in Python 3.8 there is a new feature in place, bpo-33499, that will recognize a new env variable (PYTHONPYCACHEPREFIX) that will change the place where __pycache__ is stored [2]. I backported this feature to 3.7 and create a JeOS image that includes salt-minion. I created an small shim that replace the python3.7 binary to enable this cache prefix feature, to point it to /var/ cache/pycache/<username>, and I removed from the image all the python compiled code. I decided salt-minion as saltsack is a relevant Python codebase. I needed to port to 3.7 150 python libraries to create the first PoC. The PoC works properly locally. I have yet some bits that I need to publish in the repo, but the general idea seems to work OK. I can also publish the gain on size for the ISO with the patch and without the patch, to have more data to compare. I also estimated some gains for different scenarios. For example in a normal TW installation: * Python 2.7 + 3.6 - pyc/pyc: 127M total - py: 109M total * Python 3.6 only - pyc/pyc: 91M total - py: 70M total Python pyc/pyo size is more than the py code size, so we can potentially half the size of the Python 3 stack. Maybe for a normal TW installation the absolute gain is not much (91M). But for other scenarios can be relevant, like in OpenStack Cloud, where the size of the Python code is big. I made some calculations based on all the different OpenStack services: * Python 2.7 OpenStack services - pyc/pyo: 1.2G total - py: 804M total Saving 1.2G each node is a more important number. So, my proposal is to remove the pyc from the Python 3 packages and enable the cache layer on Tumbleweed since Python 3.7. I do not know if do that by default or under certain configurations, as I am not sure how to that feature optional. Any ideas? Any suggestions? What do you think if I follow this path? Some ideas that I have are add a new %pycache-clean macro that will remove the __pycache__ from the RPM, add a new rpmlint check to make sure that there are not pyc for a python3 package, update the wiki and update the py2pack code to generate good python3 spec files for openSUSE. But if most of the community do not agree with this approach, I can drop the idea : ) [1] https://bugs.python.org/issue33499 [2] https://docs.python.org/3.8/whatsnew/3.8.html p.s: Sorry if the message is delivered two times, one from the .com account. My bad. -- SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-packaging+owner(a)opensuse.org

12 28

[opensuse-packaging] "quilt setup" considered harmful
by Matthias Gerstner 06 Oct '18

06 Oct '18

Hello packagers, the SUSE security team wants to draw your attention to a potential security threat involving the use of `quilt setup ...` on untrusted RPM spec files. For many of us calling `quilt setup $PACKAGE.spec` is probably a frequent part of our daily workflows. In contrast to building a package on the server in an isolated VM or on the client in a chroot via `osc build`, the `quilt setup` runs without any isolation on the host in the calling user's context. As it turns out this operation easily allows to execute code in the following ways: - The statements in the `%prep` section of the RPM spec file are unconditionally executed in the context of the calling user. - Arbitrary flags can be passed to `patch` via `%define _default_patch_flags ...` in the spec file. By embedding semicolons into the flags also arbitrary commands can be injected this way. - By combining the available vectors, difficult to spot malicious code can be hidden in RPM spec files. For example patch can be caused to follow symlinks, thereby "patching" files in a user's home directory as demonstrated in [1]. A demonstration works like this: ```sh $ osc co home:mgerstner/surprise $ cd home:mgerstner/surprise $ quilt setup surprise.spec # notice the surprise $ bash ``` We have posted about this on the oss-security mailing list [2] to start a discussion about possible countermeasures. A first aid could be to run the `quilt setup` inside a docker container or in a similar isolated environment. We are currently testing isolation of quilt with nsjail [3]. nsjail RPMs are available for Leap 15 and Tumbleweed from its devel project [4]. It is currently not found in Factory/Leap directly. Via the wrapper "squilt" [5], nsjail is utilized to confine quilt to read only the files it needs and is only able to write in the current directory. According to our initial testing it can be used as a drop in replacement and should reduce the attack surface significantly. Our foremost intent is to make you aware of this so you don't run `quilt setup` unsuspectingly on untrusted packages that did not go through review. Furthermore to make you aware of how malicious code that targets this can be embedded in spec files and patches. This should be taken into account when reviewing package submissions. If you have any questions or suggestions then please let's start a discussion. [1]: https://build.opensuse.org/package/show/home:mgerstner/surprise [2]: https://www.openwall.com/lists/oss-security/2018/09/27/2 [3]: http://nsjail.com [4]: https://build.opensuse.org/package/show/security/nsjail [5]: https://github.com/jsegitz/squilt -- Matthias Gerstner <matthias.gerstner(a)suse.de> Dipl.-Wirtsch.-Inf. (FH), Security Engineer https://www.suse.com/security Telefon: +49 911 740 53 290 GPG Key ID: 0x14C405C971923553 SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nuernberg)

5 5

Re: [opensuse-packaging] Proposal to remove pyc/pyo from Python on TW
by Alberto Planas Dominguez 05 Oct '18

05 Oct '18

On Thursday, October 4, 2018 5:03:46 PM CEST Michael Ströder wrote: > On 10/4/18 4:52 PM, Alberto Planas Dominguez wrote: > > I created an small shim that replace the > > python3.7 binary to enable this cache prefix feature, to point it to /var/ > > cache/pycache/<username>, and I removed from the image all the python > > compiled code. > > The above sounds to me that compiled code goes into several > user-specific pycache directories. How does that save space? In two ways basically * The rpm is smaller * In the disk, for long services only one version of __pycache__ will be in / var/cache, the one under were the service is running Also /var/cache, by definition, can be clearer frequently, or even reside on RAM. But I agree that this point is relevant, as the user cache is in ~/.cache according to XDG, so this is another alternative. -- SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-packaging+owner(a)opensuse.org

3 5

[opensuse-packaging] py2pack crashes because of code in package's setup.py
by John Paul Adrian Glaubitz 05 Oct '18

05 Oct '18

Hi! I just wanted to package the Python module azure-storage-blob and used py2pack to generate spec file when I ran into this: glaubitz@suse-laptop:~/suse/home:glaubitz:branches:devel:languages:python:azure/python-azure-storage-blob> py2pack generate azure-storage-blob -f python-azure-storage-blob.spec generating spec file for azure-storage-blob... Traceback (most recent call last): File "setup.py", line 46, in <module> ' Uninstall it with "pip uninstall azure-storage".' Exception: This package is incompatible with azure-storage==0.36.0. Uninstall it with "pip uninstall azure-storage". Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/metaextract/utils.py", line 105, in _setup_py_run_from_dir subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True) File "/usr/lib64/python3.6/subprocess.py", line 336, in check_output **kwargs).stdout File "/usr/lib64/python3.6/subprocess.py", line 418, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '/usr/bin/python3 setup.py -q --command-packages metaextract metaextract -o /tmp/tmpmcsw3yla ' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/bin/py2pack", line 10, in <module> sys.exit(main()) File "/usr/lib/python3.6/site-packages/py2pack/__init__.py", line 341, in main args.func(args) File "/usr/lib/python3.6/site-packages/py2pack/__init__.py", line 243, in generate _augment_data_from_tarball(args, tarball_file[0], data) File "/usr/lib/python3.6/site-packages/py2pack/__init__.py", line 148, in _augment_data_from_tarball data_archive = meta_utils.from_archive(filename) File "/usr/lib/python3.6/site-packages/metaextract/utils.py", line 134, in from_archive data = _setup_py_run_from_dir(root_dir, py_interpreter) File "/usr/lib/python3.6/site-packages/metaextract/utils.py", line 109, in _setup_py_run_from_dir subprocess.check_output(cmd, shell=True) File "/usr/lib64/python3.6/subprocess.py", line 336, in check_output **kwargs).stdout File "/usr/lib64/python3.6/subprocess.py", line 418, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '/usr/bin/python3 setup.py -q --command-packages metaextract metaextract -o /tmp/tmpmcsw3yla ' returned non-zero exit status 1. glaubitz@suse-laptop:~/suse/home:glaubitz:branches:devel:languages:python:azure/python-azure-storage-blob> At first glance it looks the setup.py file generates an exception because the module azure-storage is installed. And, in fact, removing "python-azure*" and "python3-azure*" from my machine fixes the problem immediately. Any suggestion what to do with this issue? I would consider that a bug in "py2pack", OTOH I'm not sure what "py2pack" can do when the setup.py of a Python module generates an exception. Adrian -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-packaging+owner(a)opensuse.org

2 1

[opensuse-packaging] Python macros
by Ansgar Esztermann-Kirchner 03 Oct '18

03 Oct '18

Hello List, I'm just trying to put together a package containing python code. https://en.opensuse.org/openSUSE:Packaging_Python says I can use %{python_sitearch} (except for very old Suse releases), but when I build for tumbleweed, that just expands to the empty string. I have BuildRequires: python-rpm-macros and rpm --showrc spews out pages including -13: python_sitearch %{_python_sysconfig_path %python_flavor platlib} What am I missing? Thanks, A. -- Ansgar Esztermann Sysadmin http://www.mpibpc.mpg.de/grubmueller/esztermann

2 3