[Bug 786318] New: mkinitrd inefficiently called multiple times per updates session
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c0 Summary: mkinitrd inefficiently called multiple times per updates session Classification: openSUSE Product: openSUSE Factory Version: 12.3 Milestone 0 Platform: PC OS/Version: Other Status: NEW Severity: Enhancement Priority: P5 - None Component: Basesystem AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: mrmazda@earthlink.net QAContact: qa-bugs@suse.de Found By: --- Blocker: --- Created an attachment (id=510522) --> (http://bugzilla.novell.com/attachment.cgi?id=510522) 12.2 zypp history mailing list thread starter: http://lists.opensuse.org/opensuse-kernel/2012-10/msg00048.html Each of the following packages triggered mkinitrd to build a new initrd for each installed kernel: mkinitrd udev mdadm lvm2 multipath-tools to reproduce: 1-zypper up or zypper dup where more than a few basesystem packages are updated actual behavior: 1-mkinitrd is called to product a new initrd for each installed kernel each time a package is installed that affects what goes into an initrd expected behavior: 1-mkinitrd is called no more than once per zypper dup or zypper up -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c1 andreas bittner <abittner@abittner.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |abittner@abittner.de --- Comment #1 from andreas bittner <abittner@abittner.de> 2012-10-23 18:14:19 UTC --- I think there were quite a number of posts in the past on this matter. This month alone there was an earlier discussion yet again on the main English list. <http://lists.opensuse.org/opensuse/2012-10/msg00222.html> -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c Jiaying ren <jren@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jren@novell.com AssignedTo|bnc-team-screening@forge.pr |mmarek@suse.com |ovo.novell.com | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c2 Olaf Hering <ohering@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ohering@suse.com --- Comment #2 from Olaf Hering <ohering@suse.com> 2013-07-11 20:21:15 CEST --- that needs wider discussion about what to needs to be done (I have not read these threads yet). There are pros and cons for each way to do it. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c3 Olaf Hering <ohering@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mmarek@suse.com Component|Basesystem |libzypp Version|13.1 Milestone 0 |13.1 Milestone 4 AssignedTo|mmarek@suse.com |zypp-maintainers@forge.prov | |o.novell.com --- Comment #3 from Olaf Hering <ohering@suse.com> 2013-09-02 20:34:37 CEST --- I suggest libzypp exports some variable so mkinitrd knows its called from within a zypper dup (or whatever) session/transaction. Once that session/transaction is done libzypp could call a mkinitrd-owned helper script which replays the remaining required mkinitrd calls. So in the end mkinitrd is only called once per zypper dup. Reassigning to libzypp maintainers for comments. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c4 Michael Andres <ma@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |FEATURE --- Comment #4 from Michael Andres <ma@suse.com> 2013-09-03 12:14:49 CEST --- This is adressed by FATE#313506: Collect and execute rpm %posttrans scripts in zypp Execution of rpm %posttrans scripts will then be delayed until all packages have been installed by zypp. This will enable scripts like mkinitrd to detect that 'nothing changed since the previous run' (as no packages were installed inbetween). Execution of the scripts will be rpm-compatible, i.e. we will not unify multiple occurrences of the same script. It's the scripts task to optimize here. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c5 --- Comment #5 from Felix Miata <mrmazda@earthlink.net> 2013-09-03 13:25:30 UTC --- "'313506' is not a feature id... Your query: Find features with '313506' in title or description that are not closed" What is FATE#313506? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c6 Michael Andres <ma@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FEATURE | Summary|mkinitrd inefficiently |libzypp does not handle |called multiple times per |%posttrans scriptlets (was: |updates session |mkinitrd inefficiently | |called multiple times per | |updates session) --- Comment #6 from Michael Andres <ma@suse.com> 2013-09-03 16:51:12 CEST --- Sorry, it's in the internal version of features.opensuse.org, not public accessible. We can leave this bug open as public reference. The feature is (most probably) targeting openSUSE 13.2. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c Michael Andres <ma@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #510522|0 |1 is obsolete| | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c Michael Andres <ma@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P2 - High -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c7 Michael Andres <ma@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jslaby@suse.com --- Comment #7 from Michael Andres <ma@suse.com> 2013-09-03 16:54:39 CEST --- *** Bug 773575 has been marked as a duplicate of this bug. *** http://bugzilla.novell.com/show_bug.cgi?id=773575 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c8 --- Comment #8 from Felix Miata <mrmazda@earthlink.net> 2013-12-12 10:37:58 UTC --- FWIW, I just noticed doing yum update in Fedora 19 that the new vmlinuz* is written to /boot somewhere near the middle of transaction processing (150+ packages), but the new kernel's initrd doesn't show up in /boot until the final seconds, possibly the last thing before reporting processing completion. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c9 --- Comment #9 from Felix Miata <mrmazda@earthlink.net> 2014-03-17 13:07:32 EDT --- Now with mkinitrd being retired it is a dracut inefficiently being called multiple times per update session. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c10 --- Comment #10 from Michael Schröder <mls@suse.com> 2014-03-17 17:47:24 UTC --- But AFAIK libzypp in Factory/13.2 does handle %posttrans correctly... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c11 --- Comment #11 from Felix Miata <mrmazda@earthlink.net> 2014-03-17 14:19:02 EDT --- The behavior I reported in comment 0 still exists. I didn't change the summary, and don't believe changing it was appropriate. The behavior as reported still exists, as evidenced by the number of times dracut fails at upgrades time in bug 866454. I reduce the annoyance somewhat by removing kernels other than the running one myself prior to each zypper dup, and keep a kernel* lock set until after everything else gets updated, but Fedora's yum upgrade seems to have avoided this annoyance since farther back than I can remember. Maybe that has something to do with always downloading all packages and running a transaction test before installing anything and would be a problem for zypper with options alternative to DownloadInAdvance. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c12 Takashi Iwai <tiwai@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |Ulrich.Windl@rz.uni-regensb | |urg.de --- Comment #12 from Takashi Iwai <tiwai@suse.com> 2014-03-18 14:37:26 UTC --- *** Bug 858840 has been marked as a duplicate of this bug. *** http://bugzilla.novell.com/show_bug.cgi?id=858840 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c13 Michael Andres <ma@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution| |FIXED --- Comment #13 from Michael Andres <ma@suse.com> 2014-03-18 16:11:53 CET --- (In reply to comment #9)
Now with mkinitrd being retired it is a dracut inefficiently being called multiple times per update session.
Scripts to execute (%pre/%post/%posttrans/..) are part of the packages. libzypp now executes the defined scripts at the correct point in time. It is not up to libzypp to decide whether scripts from different packages are equal/equivalent: %post a <same?> %post b echo "HI" echo "HI" ^ ^^ Also zypp can't know whether it would be ok to execute them only once. If there are 100 requests for SCRIPT, we must execute SCRIPT 100 times. SCRIPT should be optimized to be cheap if there is nothing todo, but that's within the scope of the packages using it, not within libzypp. I'm leaving the bug as FIXED (from libzypps POV the original 'called multiple times' issue is INVALID) If dracut isn't fast if there is nothing todo, a bug should be opened for dracut. If it can't be optimized, packages must change the way they use dracut in their scripts. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c14 --- Comment #14 from Felix Miata <mrmazda@earthlink.net> 2014-03-18 12:34:46 EDT --- (In reply to comment #13)
(from libzypps POV the original 'called multiple times' issue is INVALID)
From user POV, what happens is what matters, not whose job it is to do whatever needs to be done, or its name. Original language "called multiple times" may not be technically what you want to hear, but that Fedora (and Mageia, and probably most other major distros) only writes to /boot one new initrd per hundreds of updates process (yum upgrade) while openSUSE builds as many as 5 initrds (per installed kernel) performing an essentially identical procedure (zypper dup) suggests an as yet though solvable openSUSE shortcoming that may indeed be better described than "called multiple times". Need there be a new bug opened with a summary such as "each installed kernel's initrd is rebuilt up to 5 times within less than an hour while various updates that might or do affect initrd content are installed", or something similarly wordy? I think that should not be necessary. The behavior described in comment 0 has not been perceptibly improved, and thus this bug is not fixed, unless it has happened in the past several days and hasn't made it to the oss mirrors yet.
-- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c15 --- Comment #15 from Ulrich Windl <Ulrich.Windl@rz.uni-regensburg.de> 2014-03-19 08:57:03 CET --- In the past people always thought something cannot be improved until someone did it. For example in older HP-UX a kernel was rebuilt replacing the current kernel when a kernel patch was installed. But still at that time there was a separate "configure" phase in addition to the "installation" phase that caused the kernel to be built only once (in configure phase). More recent versions can even decide whether a module that is patched can be replaced without reboot (unloaded/loaded when it's not used, or if processes that use it can be restarted). Now I hear from SUSE that we'll have to live with things how they presently are, even if those things got worse with recent changes. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c Michael Andres <ma@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |858840 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c16 --- Comment #16 from Michael Schröder <mls@suse.com> 2014-03-19 10:38:00 UTC --- Sure, but you're talking to the wrong people here. From the libzypp side there's nothing more we can do about the posttrans scriptlets. The mechanism to reduce the amount of time spent in posttrans is now in place, it just has to be used by the packages. Please create a new bug about the handling of mkinitrd in the kernel and kmp packages and assign it to the kernel folks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c17 --- Comment #17 from Michael Andres <ma@suse.com> 2014-03-19 11:51:18 CET --- (In reply to comment #16)
Please create a new bug about the handling of mkinitrd in the kernel and kmp packages and assign it to the kernel folks.
That's covered by bug#858840 which was closed as dup of this, though it's rather been blocked by this. I already suggested to reopen it in case the scripts are not yet fixed. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c18 --- Comment #18 from Felix Miata <mrmazda@earthlink.net> 2014-06-05 23:46:13 EDT --- There is as yet in Factory only marginal evidence this has been improved at all, with a zypper dup causing one run of initrd creation (new initrd for every installed kernel) ending each dup session. -> reopen I precede each zypper dup thus: zypper -v in zypper libzypp libsolv-tools rpm glibc udev systemd dracut That generates rebuilds of initrds for each installed kernel as many as five times, according to how many of the packages listed in comment 0 (in Factory: -mkinitrd; +dracut) have updates available. Developers with their new computers and SSDs might not see this as a problem, but wasted time is considerable with older single core CPUs and spinning storage media. So I'm now setting each initrd immutable to work around this nuisance, which means I'm unlikely to ever discover on my own if this ever really gets fixed. This is made more annoying by the apparent exclusive openSUSE behavior of obliterating all existing functional initrds for every upgrade of the various packages that can affect what an initrd contains. With e.g. *buntu, Rawhide & Cauldron, initrds are generated automatically only at kernel installation time, never rebuilt as an automatic part of a distribution upgrade process. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c19 --- Comment #19 from Michael Andres <ma@suse.com> 2014-06-06 09:17:45 CEST ---
There is as yet in Factory only marginal evidence this has been improved at all, with a zypper dup causing one run of initrd creation
To clarify: Executing %posttrans correctly will _NOT_ cause one run of initrd creation. The zypp fix only asserts that all %posttrans scripts are collected and executed correctly at the end of the transaction. But if N %posttrans scripts run initrd, it will be exectuted N times. It's task of the %posttrans script (thus of the package maintainer) to prevent superfluous actions. (e.g. touching an indicator file in each %postinstall and let the %posttrans perform only if the file exists; removing it afterwards) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c20 --- Comment #20 from Felix Miata <mrmazda@earthlink.net> 2014-06-06 03:31:49 EDT --- So now it's necessary to determine which packages need this fixup. Does a separate bug need to be filed for each, or can we use this bug, at least as a metabug to see to it that the comment 0 complaint is satisfied, maybe CC here maintainers of errant packages? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c21 --- Comment #21 from Michael Andres <ma@suse.com> 2014-06-06 11:22:58 CEST --- See comment#17 here and bug#858840#8 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c22 --- Comment #22 from Felix Miata <mrmazda@earthlink.net> 2014-06-06 05:46:22 EDT --- (In reply to comment #21)
See comment#17 here and bug#858840#8
Those really don't help much, more or less pointing to each other. The difference is this was filed against basesystem in Factory, which could be fixed, unlike 858840, filed against "update problems" in 12.3 , which would go out of support next month if 13.2's target released hadn't been postponed, and wouldn't make sense to fix unless 13.1 (future Evergreen) and/or Factory get fixed soon, and bringing the fix back to 12.3 would be simple. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c23 --- Comment #23 from Michael Andres <ma@suse.com> 2014-06-06 11:56:30 CEST --- But 858840 already targets the mkinitrd problem; is IMO wrongly resolved as dupl. of this; could be reopened or cloned for factory. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c24 Christian Boltz <suse-beta@cboltz.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED CC| |suse-beta@cboltz.de Resolution|FIXED | --- Comment #24 from Christian Boltz <suse-beta@cboltz.de> 2014-06-06 12:36:46 CEST --- (In reply to comment #19)
To clarify: Executing %posttrans correctly will _NOT_ cause one run of initrd creation.
The zypp fix only asserts that all %posttrans scripts are collected and executed correctly at the end of the transaction. But if N %posttrans scripts run initrd, it will be exectuted N times.
It's task of the %posttrans script (thus of the package maintainer) to prevent superfluous actions. (e.g. touching an indicator file in each %postinstall and let the %posttrans perform only if the file exists; removing it afterwards)
Are you serious about this? It sounds like "broken by design" to me because it basically means all packages containing a %posttrans script need to be modified. IMHO at least %posttrans scripts that have exactly the same content should be de-duplicated automatically by libzypp. I'd guess this would catch most of the %posttrans scripts we have. And: yes, I'm also aware that it won't catch all duplicates. But it would improve the situation a lot without causing lots of work to the package maintainers. BTW: Even if you don't plan to change this behaviour, it should at least be documented in the wiki. The only page containing posttrans is http://en.opensuse.org/openSUSE:Packaging_scriptlet_snippets (well, and "Cron rename") so this sounds like a good place to document it. Reopening to make sure it's documented (or ideally the behaviour gets improved). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c25 Michael Andres <ma@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution| |FIXED --- Comment #25 from Michael Andres <ma@suse.com> 2014-06-06 14:00:47 CEST ---
IMHO at least %posttrans scripts that have exactly the same content should be de-duplicated
No, they are executed the same way as plain rpm does. Apart from that, there's no way for the installer to tell whether de-duplication is appropriate for a specific script or not.
it should at least be documented in the wiki
Authority for %posttrans is rpm and rpm is well documented. It's been a bug, that due to the way we use rpm, %posttrans scripts did not work as expected when transactions are executed by zypp. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=786318 https://bugzilla.novell.com/show_bug.cgi?id=786318#c Felix Miata <mrmazda@earthlink.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |881780 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com