Re: [opensuse-packaging] How to deal with Python .pyc and .pyo files?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Johannes Meixner napsal(a):
Hello,
On Nov 14 16:00 Marcus Rueckert wrote (shortened):
On Wed, 2007-11-14 at 15:08 +0100, Johannes Meixner wrote:
Hello,
what is the recommended way for packaging Python stuff?
Either have byte-compiled Python .pyc and .pyo files packaged into the RPM or let this be done by the currently actually installed Python system during run-time on the end-user's computer and not by whatever (perhaps different) Python system during package build-time? as we build with the same python version as the user will run it later. you can safely package the .pyc files.
I understand that it is the same Python version directly after a new system installation.
But what happens when the user later change/update his Pyhon?
Think about the "worst case" when the user replaces our Pyhon with whatever self-compiled Pyhon.
Are byte-compiled Python .pyc and .pyo files the same for any Python and/or is any Python sufficiently smart to know when .pyc and/or .pyo files are outdated (even if the matching .py files are unchanged)?
They are (mostly) the same, AND there is some kind of version check in the file. Don't worry about this, python would notice any problem by itself ;e) Second worst case would be if you upgraded to a new major version of python, i.e. from 1.5 to 2.0. In that case python would report errors about bad bytecode, and i'm really not sure if it would recompile automatically. But i think it would. Worst case would be if you downgraded, i don't think python 1.5 could cope... But on the other hand, i don't think this problem is worth considering. That's why python has versioned site-packages. In your other post you noticed that python files are strewn all over the system. In most cases that is bad packaging. Generally speaking, all python files should be located in /usr/lib[64]/pythonx.y/site-packages (of course there are exceptions, but let's not go into that).
i would even recommend packaging them so they get removed if you uninstall the package.
Removal when uninstalling could also be done by whatever other magic (e.g. via %ghost or a %preun scriptlet in RPM).
Bad magic, bad bad magic. We have enough trouble with scriptlets as they are now.
I wonder why in this case small RPMs seem not to count.
For example the installed .pyc and .pyo files in python on a openSUSE 10.3 i386 system are more than 7MB and still about 1.6MB after "bzip2" so that those files should make the python RPM about 1.6MB bigger.
At least according to http://www.python.org/doc/1.5.1p1/tut/node43.html I wonder if there is a noticeable advantage to provide .pyc and .pyo files in the RPM because it reads: "A program doesn't run any faster when it is read from a ".pyc" or ".pyo" file than when it is read from a ".py" file; the only thing that's faster about ".pyc" or ".pyo" files is the speed with which they are loaded."
True, and that's why we provide them. Think about it this way: python programs need to be compiled too. We provide the "binaries" form so that users' computers don't need to do the compilation themselves. But even more important is that regular user doesn't have write access to wherever the .py files are stored. You would need to run each python program as root first, otherwise python would need to compile the sources every time the program is run.
I am no Python expert at all and I would be happy if a Python expert could provide some background information.
is this background sufficient? ;e)
Kind Regards Johannes Meixner
regards jan matejek -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4-svn0 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFHPJmGjBrWA+AvBr8RAu4nAJ44givXOzCG9LY1ge9DWoSouwTFHQCgjlf2 WA9ogJSDXliAvm2NxmAsy2E= =zapY -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Jan Matejek wrote:
Second worst case would be if you upgraded to a new major version of python, i.e. from 1.5 to 2.0. In that case python would report errors about bad bytecode, and i'm really not sure if it would recompile automatically. But i think it would.
So... could python stop reporting errors in such cases and just silently ignore the precompiled files? For mere mortals it's hard to understand that they can use the same package with ELF binaries for years, but have to rebuild python packages each time the python bytecode format changes. "Why scripts and bytecode files have to be in lib64" comes as next ;) Michal --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hello Jan, On Nov 15 20:09 Jan Matejek wrote (shortened):
Johannes Meixner napsal(a):
Are byte-compiled Python .pyc and .pyo files the same for any Python and/or is any Python sufficiently smart to know when .pyc and/or .pyo files are outdated (even if the matching .py files are unchanged)?
They are (mostly) the same, AND there is some kind of version check in the file. Don't worry about this, python would notice any problem by itself ;e)
Second worst case would be if you upgraded to a new major version of python, i.e. from 1.5 to 2.0. In that case python would report errors about bad bytecode, and i'm really not sure if it would recompile automatically. But i think it would.
Worst case would be if you downgraded, i don't think python 1.5 could cope... But on the other hand, i don't think this problem is worth considering.
That's why python has versioned site-packages. In your other post you noticed that python files are strewn all over the system. In most cases that is bad packaging. Generally speaking, all python files should be located in /usr/lib[64]/pythonx.y/site-packages (of course there are exceptions, but let's not go into that).
Many thanks for the explanation! Now I understand the stuff and now I understand why and what I must fix in particular in my hplip package which installs its .py files (but no .pyc or .pyo file at all) under /usr/share/hplip/* Currently I just run HP's "make install" (HPLIP is made by HP) and I need to find a solution together with HP to get it fixed. By the way: Isn't there a general problem with "python files strewn all over the system"? How does Python deal with .py files from whatever personal Python projects in user's home directories? The user has write access so that he gets .pyc or .pyo files stored where his .py files are. What happens if the system admin upgrades to a new major version of Python? I think the same problem (perhaps even in a worse shape) exists for whatever third-party Python stuff which gets installed somewhere under /opt/ or /usr/local/. Think about proprietary third-party software which cannot be changed and which exists for more than one SLES version. I don't know if the Python license allows to have proprietary Python stuff but I think you understand the basic idea behind. http://www.python.org/doc/1.5.1p1/tut/node43.html seems to indicate that at least semi-proprietary Python stuff can exist because it reads: --------------------------------------------------------------- It is possible to have a file called "spam.pyc" (or "spam.pyo" when -O is used) without a module "spam.py" in the same module. This can be used to distribute a library of Python code in a form that is moderately hard to reverse engineer. --------------------------------------------------------------- There could be even "binary-only" third-party Python stuff where no .py files exist. What happens if a system upgrade (e.g. SLES N -> SLES N+1) upgrades Python to a new major version?
But even more important is that regular user doesn't have write access to wherever the .py files are stored. You would need to run each python program as root first, otherwise python would need to compile the sources every time the program is run.
This has another consequence: It doesn't help not to provide .pyc or .pyo files in the RPM to avoid problems with outdated .pyc or .pyo files if the Python major version changes because when root runs whatever .py files, the matching .pyc or .pyo files are created in any case.
I am no Python expert at all and I would be happy if a Python expert could provide some background information.
is this background sufficient? ;e)
Yes, it is perfect and I am happy. Kind Regards Johannes Meixner -- SUSE LINUX Products GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany AG Nuernberg, HRB 16746, GF: Markus Rex --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
participants (3)
-
Jan Matejek -
Johannes Meixner -
Michal Marek