New subject: [opensuse-factory] Re: [opensuse-packaging] Proposal to remove pyc/pyo from Python on TW

4 Oct 2018

      Hi,

As you know the Python packages are collecting the pyc/pyo precompiled 
binaries inside the RPM. This is mostly a good idea, as makes the first 
execution of the Python code faster, as is skipped the stage where the 
interpreter compile the .py code.

But this also makes the Python stack a bit fat. Most of the time this is not a 
problem, but things are changing. We have JeOS and MicroOS, both minimal 
images (build with different goals and technology) that search for be small 
and slim. But we want to include a bit of Python in there, like salt-minion or 
cloud-init. And now the relative size of Python is evident.

For Python 2.7 and 3.7 is possible to remove the pyc code from the system and 
instruct the interpreter to avoid the recreation of the pyc once the code is 
executed. The Python interpreter, by default, will compile and store the pyc 
in the disk for each `import`, but this behavior can be disable when we call 
Python.

But this will make the initial execution of a big Python stack a bit slow, as 
the pyc needs to be recreated in memory for each invocation. The slowness can 
be relevant in some situations, so is better to not enable this feature.

But in Python 3.8 there is a new feature in place, bpo-33499, that will 
recognize a new env variable (PYTHONPYCACHEPREFIX) that will change the place 
where __pycache__ is stored [2]. I backported this feature to 3.7 and create a 
JeOS image that includes salt-minion. I created an small shim that replace the 
python3.7 binary to enable this cache prefix feature, to point it to /var/
cache/pycache/<username>, and I removed from the image all the python compiled 
code.

I decided salt-minion as saltsack is a relevant Python codebase. I needed to 
port to 3.7 150 python libraries to create the first PoC.

The PoC works properly locally. I have yet some bits that I need to publish in 
the repo, but the general idea seems to work OK. I can also publish the gain 
on size for the ISO with the patch and without the patch, to have more data to 
compare.

I also estimated some gains for different scenarios. For example in a normal 
TW installation:

* Python 2.7 + 3.6
  - pyc/pyc: 127M    total
  - py: 109M    total

* Python 3.6 only
  - pyc/pyc: 91M     total
  - py: 70M     total

Python pyc/pyo size is more than the py code size, so we can potentially half 
the size of the Python 3 stack.

Maybe for a normal TW installation the absolute gain is not much (91M). But 
for other scenarios can be relevant, like in OpenStack Cloud, where the size 
of the Python code is big. I made some calculations based on all the different 
OpenStack services:

* Python 2.7 OpenStack services
  - pyc/pyo: 1.2G    total
  - py: 804M    total

Saving 1.2G each node is a more important number.

So, my proposal is to remove the pyc from the Python 3 packages and enable the 
cache layer on Tumbleweed since Python 3.7. I do not know if do that by 
default or under certain configurations, as I am not sure how to that feature 
optional.

Any ideas? Any suggestions? What do you think if I follow this path?

Some ideas that I have are add a new %pycache-clean macro that will remove the 
__pycache__ from the RPM, add a new rpmlint check to make sure that there are 
not pyc for a python3 package, update the wiki and update the py2pack code to 
generate good python3 spec files for openSUSE.

But if most of the community do not agree with this approach, I can drop the 
idea : )

[1] https://bugs.python.org/issue33499
[2] https://docs.python.org/3.8/whatsnew/3.8.html

p.s: Sorry if the message is delivered two times, one from the .com account. 
My bad.

-- 
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham 
Norton, HRB 21284 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany

-- 
To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org
To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

[opensuse-factory] Proposal to remove pyc/pyo from Python on TW

tags

participants (18)