[Bug 1097006] New: dwz debuginfo splitting is one-thread and very very slow on packages which have too many libs
http://bugzilla.suse.com/show_bug.cgi?id=1097006 Bug ID: 1097006 Summary: dwz debuginfo splitting is one-thread and very very slow on packages which have too many libs Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Basesystem Assignee: pth@suse.com Reporter: tchvatal@suse.com QA Contact: qa-bugs@suse.de CC: mliska@suse.cz Found By: --- Blocker: --- Obs generates dwz compression for all items at once to reduce space: [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/auth_0x0100.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/adt_null.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/auth_ed25519.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/auth_gssapi.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/auth_pam.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/auth_socket.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/disks.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/client_ed25519.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/dialog_examples.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/auth_test_plugin.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/ha_blackhole.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/file_key_management.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/example_key_management.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/ha_connect.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/ha_archive.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/ha_example.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/ha_federated.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/ha_mroonga.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/ha_oqgraph.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/ha_federatedx.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/ha_rocksdb.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/ha_sphinx.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/ha_spider.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/handlersocket.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/libdaemon_example.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/locales.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/ha_test_sql_discovery.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/metadata_lock_info.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/qa_auth_client.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/query_cache_info.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/mypluglib.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/qa_auth_interface.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/qa_auth_server.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/query_response_time.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/semisync_slave.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/sql_errlog.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/semisync_master.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/server_audit.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/sbin/mysqld [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/simple_password_check.so [ 209s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/wsrep_info.so [ 497s] /usr/lib/rpm/sepdebugcrcfix: Updated 80 CRC32s, 0 CRC32s did match. As you can see whole build took 209s while dwz split another 288s utilizing just one core. The -multifile mode tries to reduce the size even more, but question is if there are enough gains to justify the slowdown. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1097006 http://bugzilla.suse.com/show_bug.cgi?id=1097006#c1 Martin Liška <martin.liska@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.liska@suse.com, | |matz@suse.com, | |tchvatal@suse.com, | |tdevries@suse.de Flags| |needinfo?(tchvatal@suse.com | |) --- Comment #1 from Martin Liška <martin.liska@suse.com> --- Can you Tomas please attach some numbers how big is debug info w/ and w/o dwz? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1097006 http://bugzilla.suse.com/show_bug.cgi?id=1097006#c2 Tomáš Chvátal <tchvatal@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(tchvatal@suse.com | |) | --- Comment #2 from Tomáš Chvátal <tchvatal@suse.com> --- [ 98s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/lib64/mysql/plugin/simple_password_check.so [ 384s] /usr/lib/rpm/sepdebugcrcfix: Updated 80 CRC32s, 0 CRC32s did match. tchvatal@kunlun: /dev/shm/tchvatal/tmp/build-root/openSUSE_Tumbleweed-x86_64/home/abuild/rpmbuild/RPMS/x86_64 $ ls -laF *debuginfo* -rw-r--r-- 1 399 399 18M 12. čen 10.21 libmysqld19-debuginfo-10.2.15-0.x86_64.rpm -rw-r--r-- 1 399 399 511K 12. čen 10.21 mariadb-bench-debuginfo-10.2.15-0.x86_64.rpm -rw-r--r-- 1 399 399 1,6M 12. čen 10.21 mariadb-client-debuginfo-10.2.15-0.x86_64.rpm -rw-r--r-- 1 399 399 109M 12. čen 10.20 mariadb-debuginfo-10.2.15-0.x86_64.rpm -rw-r--r-- 1 399 399 37M 12. čen 10.22 mariadb-test-debuginfo-10.2.15-0.x86_64.rpm -rw-r--r-- 1 399 399 19M 12. čen 10.22 mariadb-tools-debuginfo-10.2.15-0.x86_64.rpm [ 98s] extracting debug info from /home/abuild/rpmbuild/BUILDROOT/mariadb-10.2.15-0.x86_64/usr/sbin/mysqld [ 105s] cpio: build/storage/include/fts0blex.h: Cannot stat: No such file or directory tchvatal@kunlun: /dev/shm/tchvatal/tmp/build-root/openSUSE_Tumbleweed-x86_64/home/abuild/rpmbuild/RPMS/x86_64 $ ls -laF *debuginfo* -rw-r--r-- 1 399 399 28M 12. čen 10.43 libmysqld19-debuginfo-10.2.15-0.x86_64.rpm -rw-r--r-- 1 399 399 721K 12. čen 10.43 mariadb-bench-debuginfo-10.2.15-0.x86_64.rpm -rw-r--r-- 1 399 399 2,1M 12. čen 10.43 mariadb-client-debuginfo-10.2.15-0.x86_64.rpm -rw-r--r-- 1 399 399 139M 12. čen 10.42 mariadb-debuginfo-10.2.15-0.x86_64.rpm -rw-r--r-- 1 399 399 58M 12. čen 10.44 mariadb-test-debuginfo-10.2.15-0.x86_64.rpm -rw-r--r-- 1 399 399 29M 12. čen 10.45 mariadb-tools-debuginfo-10.2.15-0.x86_64.rpm -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1097006 http://bugzilla.suse.com/show_bug.cgi?id=1097006#c3 --- Comment #3 from Richard Biener <rguenther@suse.com> --- DWZ is limited by some params but indeed it is single-threaded. I'm not sure how much of it could be done in parallel (I suspect nothing without a significant rewrite). Do we really care so much about the build time? I ask because debuginfo extraction (esp. debugedit) is consuming a big amount of time as well. Note in single-file mode the dwz invocations could be done in parallel (likewise the debug info extraction). That would be rpms (find-debuginfo.sh) job. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1097006 http://bugzilla.suse.com/show_bug.cgi?id=1097006#c4 --- Comment #4 from Tomáš Chvátal <tchvatal@suse.com> --- (In reply to Richard Biener from comment #3)
DWZ is limited by some params but indeed it is single-threaded. I'm not sure how much of it could be done in parallel (I suspect nothing without a significant rewrite). Do we really care so much about the build time? I ask because debuginfo extraction (esp. debugedit) is consuming a big amount of time as well. Note in single-file mode the dwz invocations could be done in parallel (likewise the debug info extraction). That would be rpms (find-debuginfo.sh) job.
It is slowing all the staging projects/etc. Which we rebuild daily. If we take look only on stock SLE rebuild counter we rebuild everything quite significantly http://ostrich.suse.de/build-times/project/SUSE:SLE-15:GA/index.html thus if we spent 3 mins on avg doing something it is unfortunate. Maybe we should disable dwz on staging projects to speed up at least those common rebuilds. FWIW there are even bigger slowdowns in lieu of brp checks and rpmlint but the plan is to look at them somehow too. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1097006 http://bugzilla.suse.com/show_bug.cgi?id=1097006#c5 Philipp Thomas <pth@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |WONTFIX --- Comment #5 from Philipp Thomas <pth@suse.com> --- I will look into this but I won't do any major changes unless that code comes from upstream. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1097006 http://bugzilla.suse.com/show_bug.cgi?id=1097006#c6 Martin Liška <martin.liska@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|WONTFIX |--- --- Comment #6 from Martin Liška <martin.liska@suse.com> --- (In reply to Philipp Thomas from comment #5)
I will look into this but I won't do any major changes unless that code comes from upstream.
I wouldn't close it as wontfix. We now have a new team member (Tom) and dwz is definitely one thing he can potentially work on. Thus re-opening. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1097006 http://bugzilla.suse.com/show_bug.cgi?id=1097006#c7 --- Comment #7 from Martin Liška <martin.liska@suse.com> --- (In reply to Richard Biener from comment #3)
DWZ is limited by some params but indeed it is single-threaded. I'm not sure how much of it could be done in parallel (I suspect nothing without a significant rewrite). Do we really care so much about the build time? I ask because debuginfo extraction (esp. debugedit) is consuming a big amount of time as well.
That's currently being rewritten, so it should speed up soon.
Note in single-file mode the dwz invocations could be done in parallel (likewise the debug info extraction). That would be rpms (find-debuginfo.sh) job.
But if I'm correct, most benefits are achieved when run on all binaries, right? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1097006 http://bugzilla.suse.com/show_bug.cgi?id=1097006#c8 --- Comment #8 from Richard Biener <rguenther@suse.com> --- (In reply to Martin Liška from comment #7)
(In reply to Richard Biener from comment #3)
DWZ is limited by some params but indeed it is single-threaded. I'm not sure how much of it could be done in parallel (I suspect nothing without a significant rewrite). Do we really care so much about the build time? I ask because debuginfo extraction (esp. debugedit) is consuming a big amount of time as well.
That's currently being rewritten, so it should speed up soon.
Note in single-file mode the dwz invocations could be done in parallel (likewise the debug info extraction). That would be rpms (find-debuginfo.sh) job.
But if I'm correct, most benefits are achieved when run on all binaries, right?
Yes. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1097006 Stefan Fent <stefan.fent@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |stefan.fent@suse.com Assignee|pth@suse.com |tdevries@suse.com -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1097006 https://bugzilla.suse.com/show_bug.cgi?id=1097006#c10 --- Comment #10 from Tom de Vries <tdevries@suse.com> --- See also upstream PR25951 - support for parallel processing ( https://sourceware.org/bugzilla/show_bug.cgi?id=25951 ). -- You are receiving this mail because: You are on the CC list for the bug.
participants (2)
-
bugzilla_noreply＠novell.com -
bugzilla_noreply＠suse.com