setup.py test -> pytest: stripped/unstripped python module
Hello, I have following issue I have met second time now and I would like to ask you for an explanation and possibly for an advice. I am trying to migrate python-bitstruct and when I use %pytest instead of setup.py test I get following error: [ 7s] import bitstruct.c [ 7s] E ModuleNotFoundError: No module named 'bitstruct.c' In case I run %check %python_expand PYTHONPATH=%{buildroot}%{$python_sitearch} $python setup.py test %pytest_arch pytest_arch succedes. It boiled down that the reason is, for 3.6 for example: copying build/lib.linux-x86_64-3.6/bitstruct/c.cpython-36m-x86_64-linux-gnu.so -> bitstruct Note though, that bitstruct/c.cpython-36m-x86_64-linux-gnu.so is available to %pytest even if setup.py test is not called: $ file /home/abuild/rpmbuild/BUILDROOT/python-bitstruct-8.11.0-0.x86_64/usr/lib64/python3.6/site-packages/bitstruct/c.cpython-36m-x86_64-linux-gnu.so /home/abuild/rpmbuild/BUILDROOT/python-bitstruct-8.11.0-0.x86_64/usr/lib64/python3.6/site-packages/bitstruct/c.cpython-36m-x86_64-linux-gnu.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=98a2f9f5618416992f7d49c3980a2b2f83b4df6e, stripped $ But $ file _build.python36/lib.linux-x86_64-3.6/bitstruct/c.cpython-36m-x86_64-linux-gnu.so _build.python36/lib.linux-x86_64-3.6/bitstruct/c.cpython-36m-x86_64-linux-gnu.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=9a390d431fd0b0a8d895384da74f5324848f83cf, with debug_info, not stripped $ What I am missing? Petr -- Have a lot of fun!
On Fri, May 21, 2021 at 12:48:10PM +0200, pgajdos wrote:
Hello,
I have following issue I have met second time now and I would like to ask you for an explanation and possibly for an advice.
In other words: :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # export PYTHONPATH=/home/abuild/rpmbuild/BUILDROOT/python-bitstruct-8.11.0-0.x86_64/usr/lib64/python3.9/site-packages :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # export PYTHONDONTWRITEBYTECODE=1 :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # strace -f -s512 pytest-3.9 --ignore=_build.python36 --ignore=_build.python39 --ignore=_build.python38 -v > strace.log 2>&1 :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # grep 'bitstruct.*\.so' strace.log newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/tests/__init__.cpython-39-x86_64-linux-gnu.so", 0x7ffdb56b5310, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/tests/__init__.abi3.so", 0x7ffdb56b5310, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/tests/__init__.so", 0x7ffdb56b5310, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/__init__.cpython-39-x86_64-linux-gnu.so", 0x7ffdb56b4f20, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/__init__.abi3.so", 0x7ffdb56b4f20, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/__init__.so", 0x7ffdb56b4f20, 0) = -1 ENOENT (No such file or directory) :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # cp build/lib.linux-x86_64-3.9/bitstruct/c.cpython-39-x86_64-linux-gnu.so bitstruct :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # strace -f -s512 pytest-3.9 --ignore=_build.python36 --ignore=_build.python39 --ignore=_build.python38 -v > strace.log 2>&1 :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # vim strace.log (failed reverse-i-search)`sed': vim ^Crace.log :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # grep 'bitstruct.*\.so' strace.log newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/c.cpython-39-x86_64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=142960, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/c.cpython-39-x86_64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=142960, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/c.cpython-39-x86_64-linux-gnu.so/bin", 0x7ffe5fab9550, 0) = -1 ENOTDIR (Not a directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/tests/__init__.cpython-39-x86_64-linux-gnu.so", 0x7ffe5fab8270, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/tests/__init__.abi3.so", 0x7ffe5fab8270, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/tests/__init__.so", 0x7ffe5fab8270, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/__init__.cpython-39-x86_64-linux-gnu.so", 0x7ffe5fab7e80, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/__init__.abi3.so", 0x7ffe5fab7e80, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/__init__.so", 0x7ffe5fab7e80, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/c.cpython-39-x86_64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=142960, ...}, 0) = 0 openat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/c.cpython-39-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 11 :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # grep '^======' strace.log ============================== 49 passed in 0.33s ============================== :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # ls /home/abuild/rpmbuild/BUILDROOT/python-bitstruct-8.11.0-0.x86_64/usr/lib64/python3.9/site-packages/bitstruct __init__.py __pycache__ c.cpython-39-x86_64-linux-gnu.so version.py :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # Petr -- Have a lot of fun!
On Fri, May 21, 2021 at 02:40:16PM +0200, pgajdos wrote:
:/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # export PYTHONPATH=/home/abuild/rpmbuild/BUILDROOT/python-bitstruct-8.11.0-0.x86_64/usr/lib64/python3.9/site-packages [..] :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # ls /home/abuild/rpmbuild/BUILDROOT/python-bitstruct-8.11.0-0.x86_64/usr/lib64/python3.9/site-packages/bitstruct __init__.py __pycache__ c.cpython-39-x86_64-linux-gnu.so version.py :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 #
But: :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # cp /home/abuild/rpmbuild/BUILDROOT/python-bitstruct-8.11.0-0.x86_64/usr/lib64/python3.9/site-packages/bitstruct/c.cpython-39-x86_64-linux-gnu.so bitstruct :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # pytest-3.9 --ignore=_build.python36 --ignore=_build.python39 --ignore=_build.python38 -v [...] ============================= 49 passed in 0.12s ========================= So export PYTHONPATH=/home/abuild/rpmbuild/BUILDROOT/python-bitstruct-8.11.0-0.x86_64/usr/lib64/python3.9/site-packages is wrong or something else? Petr -- Have a lot of fun!
Am 21.05.21 um 12:48 schrieb pgajdos:
%check %python_expand PYTHONPATH=%{buildroot}%{$python_sitearch} $python setup.py test %pytest_arch
So you run the tests twice? Does %pytest_arch here really succeed? But without setup.py tests before it does not? Am 21.05.21 um 14:40 schrieb pgajdos:
On Fri, May 21, 2021 at 12:48:10PM +0200, pgajdos wrote:
Hello,
I have following issue I have met second time now and I would like to ask you for an explanation and possibly for an advice.
In other words:
:/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # export PYTHONPATH=/home/abuild/rpmbuild/BUILDROOT/python-bitstruct-8.11.0-0.x86_64/usr/lib64/python3.9/site-packages :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # export PYTHONDONTWRITEBYTECODE=1 :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # strace -f -s512 pytest-3.9 --ignore=_build.python36 --ignore=_build.python39 --ignore=_build.python38 -v > strace.log 2>&1 :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # grep 'bitstruct.*\.so' strace.log newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/tests/__init__.cpython-39-x86_64-linux-gnu.so", 0x7ffdb56b5310, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/tests/__init__.abi3.so", 0x7ffdb56b5310, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/tests/__init__.so", 0x7ffdb56b5310, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/__init__.cpython-39-x86_64-linux-gnu.so", 0x7ffdb56b4f20, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/__init__.abi3.so", 0x7ffdb56b4f20, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/__init__.so", 0x7ffdb56b4f20, 0) = -1 ENOENT (No such file or directory) :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 #
:/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # cp build/lib.linux-x86_64-3.9/bitstruct/c.cpython-39-x86_64-linux-gnu.so bitstruct :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # strace -f -s512 pytest-3.9 --ignore=_build.python36 --ignore=_build.python39 --ignore=_build.python38 -v > strace.log 2>&1 :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # vim strace.log (failed reverse-i-search)`sed': vim ^Crace.log :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # grep 'bitstruct.*\.so' strace.log newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/c.cpython-39-x86_64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=142960, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/c.cpython-39-x86_64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=142960, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/c.cpython-39-x86_64-linux-gnu.so/bin", 0x7ffe5fab9550, 0) = -1 ENOTDIR (Not a directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/tests/__init__.cpython-39-x86_64-linux-gnu.so", 0x7ffe5fab8270, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/tests/__init__.abi3.so", 0x7ffe5fab8270, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/tests/__init__.so", 0x7ffe5fab8270, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/__init__.cpython-39-x86_64-linux-gnu.so", 0x7ffe5fab7e80, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/__init__.abi3.so", 0x7ffe5fab7e80, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/__init__.so", 0x7ffe5fab7e80, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/c.cpython-39-x86_64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=142960, ...}, 0) = 0 openat(AT_FDCWD, "/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0/bitstruct/c.cpython-39-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 11 :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # grep '^======' strace.log ============================== 49 passed in 0.33s ============================== :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 #
:/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 # ls /home/abuild/rpmbuild/BUILDROOT/python-bitstruct-8.11.0-0.x86_64/usr/lib64/python3.9/site-packages/bitstruct __init__.py __pycache__ c.cpython-39-x86_64-linux-gnu.so version.py :/home/abuild/rpmbuild/BUILD/bitstruct-8.11.0 #
Petr
Both strace.log show that the import is from BUILD, not from BUILDROOT. So even although you have PYTHONPATH=...BUILDROOT..., the tests still try to import from BUILD (cwd). - check whether the config file for pytest (setup.cfg, pytest.ini, tox.ini, it is reported in the header) modify the import mode. - Check whether conftest.py or any other code tamper with sys.path. * try different pytest --import-mode parameters * Nuclear option: rename/remove the bitstruct directory in BUILD, so that the bitstruct module is really imported from BUILDROOT For future reference: Be clear whethe you use %pytest pr %pytest_arch. Different PYTHONPATH settings. purelib (/usr/lib) vs platlib (%{_libdir}, /usr/lib64). Cheers, Ben
Ben, thanks for your reply! On Fri, May 21, 2021 at 05:36:17PM +0200, Ben Greiner wrote:
- check whether the config file for pytest (setup.cfg, pytest.ini, tox.ini, it is reported in the header) modify the import mode. - Check whether conftest.py or any other code tamper with sys.path.
As far as I quickly checked, that is not the case.
* try different pytest --import-mode parameters * Nuclear option: rename/remove the bitstruct directory in BUILD, so that the bitstruct module is really imported from BUILDROOT
It seems that's it: %check mv bitstruct bitstruct.orig %pytest_arch This works. Also %check %pytest_arch --import-mode=append works (as well as --import-mode=importlib). See home:pgajdos:python/python-bitstruct What is preferred solution for the spec file? Petr -- Have a lot of fun!
Relevant reads: https://docs.pytest.org/en/latest/explanation/goodpractices.html#choosing-a-... https://docs.pytest.org/en/latest/explanation/pythonpath.html
the tests will run against the installed version of pkg_under_test when --import-mode=append is used whereas with prepend they would pick up the local version.
Am 24.05.21 um 09:42 schrieb pgajdos:
See home:pgajdos:python/python-bitstruct
What is preferred solution for the spec file?
Why use pytest at all? The recommended way to replace `python setup.py test` is to call `python -m unittest` directly. If it is supported by all build targets (you need to adjust your Leap 15.2 target to include backports), you can use the macro: mv bitstruct bitstruct.mv %pyunittest_arch discover -v tests - Ben
Dne 24. 05. 21 v 11:31 Ben Greiner napsal(a):
Why use pytest at all? The recommended way to replace `python setup.py test` is to call `python -m unittest` directly. If it is supported by all build targets (you need to adjust your Leap 15.2 target to include backports), you can use the macro:
pytest has better error reports, but yes, unless the upstream uses pytest, I usually rather stay with %pyunittest macro. Matěj -- https://matej.ceplovi.cz/blog/, Jabber: mcepl@ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 If in desperation, read the documentation! -- Brian D. Ripley, on R-help list
participants (3)
-
Ben Greiner -
Matěj Cepl -
pgajdos