Help needed (part 2): ansible-runner hangs (indefinitely) after first round of checks
Hi all, next one: I updated ansible-runner today, but it hangs after the first round of pytests.
https://build.opensuse.org/package/show/home:ojkastl_buildservice:Branch_sys...
I aborted the builds on OBS after 4770s. Any ideas? Kind Regards, Johannes This are the last lines before it stalls:
[ 156s] [ 156s] ============================= slowest 10 durations ============================= [ 156s] 4.14s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_keepalive_setting[0] [ 156s] 3.39s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_keepalive_setting[1e-09] [ 156s] 3.24s call test/integration/test_runner.py::test_pexpect_timeout [ 156s] 3.13s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_keepalive_setting[None] [ 156s] 3.12s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_keepalive_setting[1] [ 156s] 2.75s call test/integration/test_display_callback.py::test_callback_plugin_no_log_filters[playbook4] [ 156s] 2.55s call test/integration/test___main__.py::test_cmdline_playbook [ 156s] 2.52s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_remote_job_interface[adhoc] [ 156s] 2.46s call test/integration/test_interface.py::test_run_role [ 156s] 2.12s call test/integration/test___main__.py::test_cmdline_cmdline_override [ 156s] =========================== short test summary info ============================ [ 156s] SKIPPED [4] test/integration/test_core_integration.py:15: docker is not installed [ 156s] SKIPPED [4] test/integration/test_core_integration.py:15: podman is not installed [ 156s] SKIPPED [4] test/integration/test_core_integration.py:39: docker is not installed [ 156s] SKIPPED [4] test/integration/test_core_integration.py:39: podman is not installed [ 156s] SKIPPED [2] test/integration/test_events.py:7: docker is not installed [ 156s] SKIPPED [2] test/integration/test_events.py:7: podman is not installed [ 156s] SKIPPED [2] test/integration/test_events.py:52: docker is not installed [ 156s] SKIPPED [2] test/integration/test_events.py:52: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:143: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:143: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:242: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:242: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:256: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:256: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:277: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:277: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:339: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:339: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:365: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:365: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:404: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:404: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:463: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:463: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:522: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:522: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cleanup_images.py:12: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cleanup_images.py:12: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:12: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:12: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:26: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:26: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:45: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:45: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:57: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:57: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:52: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:52: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:80: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:80: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:101: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:101: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:151: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:151: podman is not installed [ 156s] SKIPPED [1] test/unit/test_runner.py:165: Writing to stdout can be flaky, probably due to some pexpect bug [ 156s] ========== 2083 passed, 61 skipped, 7 deselected in 148.18s (0:02:28) ==========
Hi all, this is still an issue. Due to the fact that we do not follow upstream's way of running the tests they are not inclined to help. Also, they have not had any issues like long-running / hanging tests before. So, any help would be highly appreciated. Kind Regards, Johannes On 18.05.24 19:08 Johannes Kastl wrote:
Hi all,
next one: I updated ansible-runner today, but it hangs after the first round of pytests.
https://build.opensuse.org/package/show/home:ojkastl_buildservice:Branch_sys...
I aborted the builds on OBS after 4770s.
Any ideas?
Kind Regards, Johannes
This are the last lines before it stalls:
[ 156s] [ 156s] ============================= slowest 10 durations ============================= [ 156s] 4.14s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_keepalive_setting[0] [ 156s] 3.39s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_keepalive_setting[1e-09] [ 156s] 3.24s call test/integration/test_runner.py::test_pexpect_timeout [ 156s] 3.13s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_keepalive_setting[None] [ 156s] 3.12s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_keepalive_setting[1] [ 156s] 2.75s call test/integration/test_display_callback.py::test_callback_plugin_no_log_filters[playbook4] [ 156s] 2.55s call test/integration/test___main__.py::test_cmdline_playbook [ 156s] 2.52s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_remote_job_interface[adhoc] [ 156s] 2.46s call test/integration/test_interface.py::test_run_role [ 156s] 2.12s call test/integration/test___main__.py::test_cmdline_cmdline_override [ 156s] =========================== short test summary info ============================ [ 156s] SKIPPED [4] test/integration/test_core_integration.py:15: docker is not installed [ 156s] SKIPPED [4] test/integration/test_core_integration.py:15: podman is not installed [ 156s] SKIPPED [4] test/integration/test_core_integration.py:39: docker is not installed [ 156s] SKIPPED [4] test/integration/test_core_integration.py:39: podman is not installed [ 156s] SKIPPED [2] test/integration/test_events.py:7: docker is not installed [ 156s] SKIPPED [2] test/integration/test_events.py:7: podman is not installed [ 156s] SKIPPED [2] test/integration/test_events.py:52: docker is not installed [ 156s] SKIPPED [2] test/integration/test_events.py:52: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:143: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:143: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:242: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:242: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:256: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:256: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:277: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:277: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:339: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:339: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:365: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:365: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:404: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:404: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:463: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:463: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:522: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:522: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cleanup_images.py:12: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cleanup_images.py:12: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:12: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:12: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:26: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:26: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:45: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:45: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:57: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:57: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:52: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:52: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:80: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:80: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:101: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:101: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:151: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:151: podman is not installed [ 156s] SKIPPED [1] test/unit/test_runner.py:165: Writing to stdout can be flaky, probably due to some pexpect bug [ 156s] ========== 2083 passed, 61 skipped, 7 deselected in 148.18s (0:02:28) ==========
Hi Johannes, you already had pytest-xdist installed but did not use it. The `-n auto` flag activates it and somehow it makes sure all parallel processes get terminated after the test suite is finished. sr#1179575 Regards, Ben Am 09.06.24 um 13:46 schrieb Johannes Kastl:
Hi all,
this is still an issue. Due to the fact that we do not follow upstream's way of running the tests they are not inclined to help. Also, they have not had any issues like long-running / hanging tests before.
So, any help would be highly appreciated.
Kind Regards, Johannes
On 18.05.24 19:08 Johannes Kastl wrote:
Hi all,
next one: I updated ansible-runner today, but it hangs after the first round of pytests.
https://build.opensuse.org/package/show/home:ojkastl_buildservice:Branch_sys... I aborted the builds on OBS after 4770s.
Any ideas?
Kind Regards, Johannes
This are the last lines before it stalls:
[ 156s] [ 156s] ============================= slowest 10 durations ============================= [ 156s] 4.14s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_keepalive_setting[0] [ 156s] 3.39s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_keepalive_setting[1e-09] [ 156s] 3.24s call test/integration/test_runner.py::test_pexpect_timeout [ 156s] 3.13s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_keepalive_setting[None] [ 156s] 3.12s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_keepalive_setting[1] [ 156s] 2.75s call test/integration/test_display_callback.py::test_callback_plugin_no_log_filters[playbook4] [ 156s] 2.55s call test/integration/test___main__.py::test_cmdline_playbook [ 156s] 2.52s call test/integration/test_transmit_worker_process.py::TestStreamingUsage::test_remote_job_interface[adhoc] [ 156s] 2.46s call test/integration/test_interface.py::test_run_role [ 156s] 2.12s call test/integration/test___main__.py::test_cmdline_cmdline_override [ 156s] =========================== short test summary info ============================ [ 156s] SKIPPED [4] test/integration/test_core_integration.py:15: docker is not installed [ 156s] SKIPPED [4] test/integration/test_core_integration.py:15: podman is not installed [ 156s] SKIPPED [4] test/integration/test_core_integration.py:39: docker is not installed [ 156s] SKIPPED [4] test/integration/test_core_integration.py:39: podman is not installed [ 156s] SKIPPED [2] test/integration/test_events.py:7: docker is not installed [ 156s] SKIPPED [2] test/integration/test_events.py:7: podman is not installed [ 156s] SKIPPED [2] test/integration/test_events.py:52: docker is not installed [ 156s] SKIPPED [2] test/integration/test_events.py:52: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:143: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:143: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:242: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:242: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:256: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:256: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:277: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:277: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:339: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:339: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:365: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:365: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:404: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:404: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:463: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:463: podman is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:522: docker is not installed [ 156s] SKIPPED [1] test/integration/test_interface.py:522: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cleanup_images.py:12: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cleanup_images.py:12: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:12: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:12: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:26: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:26: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:45: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:45: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:57: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_cli_containerized.py:57: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:52: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:52: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:80: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:80: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:101: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:101: podman is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:151: docker is not installed [ 156s] SKIPPED [1] test/integration/containerized/test_container_management.py:151: podman is not installed [ 156s] SKIPPED [1] test/unit/test_runner.py:165: Writing to stdout can be flaky, probably due to some pexpect bug [ 156s] ========== 2083 passed, 61 skipped, 7 deselected in 148.18s (0:02:28) ==========
Hi Ben, On 09.06.24 18:52 Ben Greiner wrote:
Hi Johannes,
you already had pytest-xdist installed but did not use it. The `-n auto` flag activates it and somehow it makes sure all parallel processes get terminated after the test suite is finished.
sr#1179575
Thanks a lot for the fix, Ben! And also for the spec cleanup you also included! Have a nice week! Johannes
participants (2)
-
Ben Greiner -
Johannes Kastl