Re: [opensuse-packaging] Proposal to remove pyc/pyo from Python on TW
On Thursday, October 4, 2018 5:03:46 PM CEST Michael Ströder wrote:
On 10/4/18 4:52 PM, Alberto Planas Dominguez wrote:
I created an small shim that replace the python3.7 binary to enable this cache prefix feature, to point it to /var/ cache/pycache/<username>, and I removed from the image all the python compiled code.
The above sounds to me that compiled code goes into several user-specific pycache directories. How does that save space?
In two ways basically * The rpm is smaller * In the disk, for long services only one version of __pycache__ will be in / var/cache, the one under were the service is running Also /var/cache, by definition, can be clearer frequently, or even reside on RAM. But I agree that this point is relevant, as the user cache is in ~/.cache according to XDG, so this is another alternative. -- SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/4/18 5:11 PM, Alberto Planas Dominguez wrote:
On Thursday, October 4, 2018 5:03:46 PM CEST Michael Ströder wrote:
On 10/4/18 4:52 PM, Alberto Planas Dominguez wrote:
I created an small shim that replace the python3.7 binary to enable this cache prefix feature, to point it to /var/ cache/pycache/<username>, and I removed from the image all the python compiled code.
The above sounds to me that compiled code goes into several user-specific pycache directories. How does that save space?
In two ways basically
* The rpm is smaller * In the disk, for long services only one version of __pycache__ will be in / var/cache, the one under were the service is running
I typically have several long-running services and CLI tools running under separate service user accounts but using a similar set of modules. So I'd expect your approach to cost more disk space. Also for security reasons I'd not want a tool to write code somewhere. It's bad practice to let a service account write code somewhere. It would seriously trigger my paranoia if someone adds rules to AppArmor profiles allowing that.
Also /var/cache, by definition, can be clearer frequently, or even reside on RAM.
You mentioned small JeOS installations. Do you expect them to have enough RAM for that? Ciao, Michael.
On Thursday, October 4, 2018 5:29:20 PM CEST Michael Ströder wrote:
On 10/4/18 5:11 PM, Alberto Planas Dominguez wrote:
I typically have several long-running services and CLI tools running under separate service user accounts but using a similar set of modules. So I'd expect your approach to cost more disk space.
I think that this is indeed the case for this scenario: running the same code under different users will recreate the pyc under each user id. Also I think that this can be resolved, using some better way to generate the first pyc in the local cache.
Also for security reasons I'd not want a tool to write code somewhere. It's bad practice to let a service account write code somewhere. It would seriously trigger my paranoia if someone adds rules to AppArmor profiles allowing that.
Another good point. -- SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/5/18 10:29 AM, Alberto Planas Dominguez wrote:
On Thursday, October 4, 2018 5:29:20 PM CEST Michael Ströder wrote:
I typically have several long-running services and CLI tools running under separate service user accounts but using a similar set of modules. So I'd expect your approach to cost more disk space.
I think that this is indeed the case for this scenario: running the same code under different users will recreate the pyc under each user id.
Maybe I'm overlooking something but you will end up with at least same disk size being used. So the argument with cloud storage does not count at all.
Also I think that this can be resolved, using some better way to generate the first pyc in the local cache.
Compiling on-the-fly to a central location would possibly also result in security issues. So you would have to create a central pycache by byte-compiling during RPM post-install as root.
Also for security reasons I'd not want a tool to write code somewhere. It's bad practice to let a service account write code somewhere. It would seriously trigger my paranoia if someone adds rules to AppArmor profiles allowing that.
Another good point.
For me that's a show-stopper. Also .py? files shipped in RPMs at least are integrity-protected by RPM signature. Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/4/18 11:11 AM, Alberto Planas Dominguez wrote:
On Thursday, October 4, 2018 5:03:46 PM CEST Michael Ströder wrote:
On 10/4/18 4:52 PM, Alberto Planas Dominguez wrote:
I created an small shim that replace the python3.7 binary to enable this cache prefix feature, to point it to /var/ cache/pycache/<username>, and I removed from the image all the python compiled code.
The above sounds to me that compiled code goes into several user-specific pycache directories. How does that save space?
In two ways basically
* The rpm is smaller
But this is a new requirement? Does the size of the rpm really matter? Previously the focus was on image size, i.e. the size of the result after unpacking all the rpms that go into an image. How does rpm size come into play? Thanks, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU Distinguished Architect LINUX Team Lead Public Cloud rjschwei@suse.com IRC: robjo
On Thursday, October 4, 2018 8:48:32 PM CEST Robert Schweikert wrote:
On 10/4/18 11:11 AM, Alberto Planas Dominguez wrote:
On Thursday, October 4, 2018 5:03:46 PM CEST Michael Ströder wrote:
On 10/4/18 4:52 PM, Alberto Planas Dominguez wrote:
The above sounds to me that compiled code goes into several user-specific pycache directories. How does that save space?
In two ways basically
* The rpm is smaller
But this is a new requirement? Does the size of the rpm really matter? Previously the focus was on image size, i.e. the size of the result after unpacking all the rpms that go into an image.
How does rpm size come into play?
Right, this is only to answer the question 'How does that save space?'. Having smaller rpms is not a requirement IMHO. But this will have some side effects that are nice (faster download, smaller drpms, smaller repos). None of them relevant to the main problem: Python is fat. -- SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
participants (3)
-
Alberto Planas Dominguez -
Michael Ströder -
Robert Schweikert