[opensuse-packaging] Python3 packaging inconsistencies/compiled files ignored
[Intentionally disabled line wrapping for the strace snippets] Hi, While debugging an ugly segfaulting issue last week, I noticed something strange in the traces when comparing local builds with system builds (the ones, that were produced from OBS): -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/__pycache__/__init__.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/__init__.py", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/__pycache__/_version.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/_version.py", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/__pycache__/utilities.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/utilities.py", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/__pycache__/__init__.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3 +openat(AT_FDCWD, "/home/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/__pycache__/_version.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/__pycache__/utilities.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 [...] -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/cpp/__pycache__/__init__.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/cpp/__init__.py", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/cpp", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/cpp/version.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/cpp/__pycache__/__init__.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/cpp", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3 +openat(AT_FDCWD, "/home/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/cpp/version.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 [...] -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/__pycache__/document.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/document.py", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/cpp/document.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/cpp/destination.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/cpp/embedded_file.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/cpp/font.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/cpp/global_.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/cpp/page.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/cpp/page_transition.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/cpp/rectangle.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/cpp/toc.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/__pycache__/destination.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/destination.py", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/__pycache__/embeddedfile.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/embeddedfile.py", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/__pycache__/font.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/font.py", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/__pycache__/page.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/page.py", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/__pycache__/pagetransition.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/pagetransition.py", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/__pycache__/rectangle.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/rectangle.py", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/__pycache__/toc.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 -openat(AT_FDCWD, "/usr/lib64/python3.8/site-packages/poppler/toc.py", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/__pycache__/document.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/cpp/document.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/cpp/destination.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/cpp/embedded_file.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/cpp/font.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/cpp/global_.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/cpp/page.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/cpp/page_transition.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/cpp/rectangle.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/cpp/toc.cpython-38-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/__pycache__/destination.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/__pycache__/embeddedfile.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/__pycache__/font.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/__pycache__/page.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/__pycache__/pagetransition.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/__pycache__/rectangle.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 +openat(AT_FDCWD, "/home/nfs/hp/src/git/python-poppler/build/lib.linux-x86_64-3.8/poppler/__pycache__/toc.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3 After opening the .pyc files, the interpreter always opens the .py files for the system package. For some reason, it doesn't accept the compiled files. The local package behave as expected. Needless to say, that this behavior is quite inefficient and not in the sense of the inventor. It would be even more efficient to to do without the compiled version and interpret every python file on load. Is that to be expected? Here's the package in question, but I fear, this is a generic issue: https://build.opensuse.org/package/show/home:frispete:python/python-python-p... I'm assuming, this is somewhat related to the attempts of reproducible builds. If somebody could confirm this issue independently, I will create a bug. Cheers, Pete TW 20201002 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Dienstag, 6. Oktober 2020 19:03:39 CEST Hans-Peter Jansen wrote:
[Intentionally disabled line wrapping for the strace snippets]
Hi,
While debugging an ugly segfaulting issue last week, I noticed something strange in the traces when comparing local builds with system builds (the ones, that were produced from OBS):
[...]
After opening the .pyc files, the interpreter always opens the .py files for the system package. For some reason, it doesn't accept the compiled files. The local package behave as expected.
The local files behave *differently*, but none is wrong, see: https://www.python.org/dev/peps/pep-0552/ The distribution pycs have both the 'hash_based' and 'check_source' flags set (5th byte == 0x3), so python has to read the original source to generate and verify the hash.
Needless to say, that this behavior is quite inefficient and not in the sense of the inventor. It would be even more efficient to to do without the compiled version and interpret every python file on load.
Is that to be expected?
Yes. Hashing the source is quite fast, much faster than generating the bytecode from scratch. Though, for pycs distributed along the sources, the 'check_source' flag is not strictly needed: "For hash-based pycs with the check_source unset, Python will simply load the pyc without checking the hash of the source file. The expectation in this case is that some external system (e.g., the local Linux distribution’s package manager) is responsible for keeping pycs up to date, so Python itself doesn’t have to check." Having the flag set is slightly paranoid, but not wrong per se. Kind regards, Stefan -- Stefan Brüns / Bergstraße 21 / 52062 Aachen home: +49 241 53809034 mobile: +49 151 50412019
Am Dienstag, 6. Oktober 2020, 20:05:22 CEST schrieb Stefan Brüns:
On Dienstag, 6. Oktober 2020 19:03:39 CEST Hans-Peter Jansen wrote:
[Intentionally disabled line wrapping for the strace snippets]
Hi,
While debugging an ugly segfaulting issue last week, I noticed something strange in the traces when comparing local builds with system builds (the
ones, that were produced from OBS): [...]
After opening the .pyc files, the interpreter always opens the .py files for the system package. For some reason, it doesn't accept the compiled files. The local package behave as expected.
The local files behave *differently*, but none is wrong, see: https://www.python.org/dev/peps/pep-0552/
The distribution pycs have both the 'hash_based' and 'check_source' flags set (5th byte == 0x3), so python has to read the original source to generate and verify the hash.
Needless to say, that this behavior is quite inefficient and not in the sense of the inventor. It would be even more efficient to to do without the compiled version and interpret every python file on load.
Is that to be expected?
Yes.
Hashing the source is quite fast, much faster than generating the bytecode from scratch.
Though, for pycs distributed along the sources, the 'check_source' flag is not strictly needed:
"For hash-based pycs with the check_source unset, Python will simply load the pyc without checking the hash of the source file. The expectation in this case is that some external system (e.g., the local Linux distribution’s package manager) is responsible for keeping pycs up to date, so Python itself doesn’t have to check."
Having the flag set is slightly paranoid, but not wrong per se.
Thanks a lot, Stefan, for explanation and pointers. One question remains, why don't we trust rpm enough to omit the hash checking for the compiled python modules for our packages? Cheers, Pete -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/6/20 11:59 AM, Hans-Peter Jansen wrote:
Am Dienstag, 6. Oktober 2020, 20:05:22 CEST schrieb Stefan Brüns:
On Dienstag, 6. Oktober 2020 19:03:39 CEST Hans-Peter Jansen wrote:
[Intentionally disabled line wrapping for the strace snippets]
Hi,
While debugging an ugly segfaulting issue last week, I noticed something strange in the traces when comparing local builds with system builds (the
ones, that were produced from OBS): [...]
After opening the .pyc files, the interpreter always opens the .py files for the system package. For some reason, it doesn't accept the compiled files. The local package behave as expected.
The local files behave *differently*, but none is wrong, see: https://www.python.org/dev/peps/pep-0552/
The distribution pycs have both the 'hash_based' and 'check_source' flags set (5th byte == 0x3), so python has to read the original source to generate and verify the hash.
Needless to say, that this behavior is quite inefficient and not in the sense of the inventor. It would be even more efficient to to do without the compiled version and interpret every python file on load.
Is that to be expected?
Yes.
Hashing the source is quite fast, much faster than generating the bytecode from scratch.
Though, for pycs distributed along the sources, the 'check_source' flag is not strictly needed:
"For hash-based pycs with the check_source unset, Python will simply load the pyc without checking the hash of the source file. The expectation in this case is that some external system (e.g., the local Linux distribution’s package manager) is responsible for keeping pycs up to date, so Python itself doesn’t have to check."
Having the flag set is slightly paranoid, but not wrong per se.
Thanks a lot, Stefan, for explanation and pointers.
One question remains, why don't we trust rpm enough to omit the hash checking for the compiled python modules for our packages?
One advantage of leaving the flag set in my opinion is that if you, as a user or developer, go in and change the "py" file, python re-compiles it for you. If you leave the flag off, then such an update would have to be manually compiled to take effect, and might be a surprise to some.
Cheers, Pete
-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Am Dienstag, 6. Oktober 2020, 21:02:16 CEST schrieb Lee Duncan:
On 10/6/20 11:59 AM, Hans-Peter Jansen wrote:
One question remains, why don't we trust rpm enough to omit the hash checking for the compiled python modules for our packages?
One advantage of leaving the flag set in my opinion is that if you, as a user or developer, go in and change the "py" file, python re-compiles it for you. If you leave the flag off, then such an update would have to be manually compiled to take effect, and might be a surprise to some.
Missed your reply, Lee, sorry. Yes, that's exactly the reason to keep it of course. And sorry for the churn. Cheers, Pete -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Am Dienstag, 6. Oktober 2020, 20:59:22 CEST schrieb Hans-Peter Jansen:
Am Dienstag, 6. Oktober 2020, 20:05:22 CEST schrieb Stefan Brüns:
Having the flag set is slightly paranoid, but not wrong per se.
Thanks a lot, Stefan, for explanation and pointers.
One question remains, why don't we trust rpm enough to omit the hash checking for the compiled python modules for our packages?
Thinking about this a bit longer, I understand, that this is not about trust in the system, and more about trust in the user. Without checking the hash, editing a .py file would most probably result in such changes to be ignored. Cheers, Pete -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
participants (3)
-
Hans-Peter Jansen -
Lee Duncan -
Stefan Brüns