[Bug 853293] New: boot hang after 12.3 -> 13.1 upgrade adds libpcre dependency to udevadm
https://bugzilla.novell.com/show_bug.cgi?id=853293 https://bugzilla.novell.com/show_bug.cgi?id=853293#c0 Summary: boot hang after 12.3 -> 13.1 upgrade adds libpcre dependency to udevadm Classification: openSUSE Product: openSUSE 13.1 Version: Final Platform: x86-64 OS/Version: Other Status: NEW Severity: Minor Priority: P5 - None Component: Basesystem AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: ar16@imapmail.org QAContact: qa-bugs@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0 in opensuse 12.3 ldd /sbin/udevadm linux-vdso.so.1 (0x00007fffe3d44000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fa23e9a2000) libblkid.so.1 => /usr/lib64/libblkid.so.1 (0x00007fa23e77a000) libkmod.so.2 => /lib64/libkmod.so.2 (0x00007fa23e565000) libacl.so.1 => /lib64/libacl.so.1 (0x00007fa23e35c000) librt.so.1 => /lib64/librt.so.1 (0x00007fa23e154000) libc.so.6 => /lib64/libc.so.6 (0x00007fa23dda6000) /lib64/ld-linux-x86-64.so.2 (0x00007fa23ebc2000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fa23dba2000) libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x00007fa23d99d000) liblzma.so.5 => /usr/lib64/liblzma.so.5 (0x00007fa23d777000) libz.so.1 => /lib64/libz.so.1 (0x00007fa23d561000) libattr.so.1 => /lib64/libattr.so.1 (0x00007fa23d35c000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa23d140000) with my usual (for a long time) ld.so.conf, cat /etc/ld.so.conf /usr/local/lib64 /usr/local/lib include /etc/ld.so.conf.d/*.conf # /lib64, /lib, /usr/lib64 and /usr/lib gets added # automatically by ldconfig after parsing this file. # So, they do not need to be listed. and a locally installed pcre I use for dev in, ls -al /usr/local/lib64/*pcre* -rwxr-xr-x 1 root root 965 Nov 27 14:10 /usr/local/lib64/libpcrecpp.la* lrwxrwxrwx 1 root root 19 Nov 27 14:10 /usr/local/lib64/libpcrecpp.so -> libpcrecpp.so.0.0.0* lrwxrwxrwx 1 root root 19 Nov 27 14:10 /usr/local/lib64/libpcrecpp.so.0 -> libpcrecpp.so.0.0.0* -rwxr-xr-x 1 root root 47K Nov 27 14:10 /usr/local/lib64/libpcrecpp.so.0.0.0* -rwxr-xr-x 1 root root 919 Nov 27 14:10 /usr/local/lib64/libpcre.la* -rwxr-xr-x 1 root root 977 Nov 27 14:10 /usr/local/lib64/libpcreposix.la* lrwxrwxrwx 1 root root 21 Nov 27 14:10 /usr/local/lib64/libpcreposix.so -> libpcreposix.so.0.0.2* lrwxrwxrwx 1 root root 21 Nov 27 14:10 /usr/local/lib64/libpcreposix.so.0 -> libpcreposix.so.0.0.2* -rwxr-xr-x 1 root root 14K Nov 27 14:10 /usr/local/lib64/libpcreposix.so.0.0.2* lrwxrwxrwx 1 root root 16 Nov 27 14:10 /usr/local/lib64/libpcre.so -> libpcre.so.1.2.2* lrwxrwxrwx 1 root root 16 Nov 27 14:10 /usr/local/lib64/libpcre.so.1 -> libpcre.so.1.2.2* -rwxr-xr-x 1 root root 425K Nov 27 14:10 /usr/local/lib64/libpcre.so.1.2.2* after rm -f /etc/ld.so.cache ldconfig mkinitrd 12.3 boots OK. but, after upgrade to 13.1, the system hangs on boot, streaming (to serial console) messages about a missing "libpcre.so.1" apparently, in opensuse 13.1, a dependency on libpcre.so.1 has been added if I boot from LiveKDE, chroot to 13.1, and vi /etc/ld.so.conf + /lib64 /usr/local/lib64 /usr/local/lib include /etc/ld.so.conf.d/*.conf # /lib64, /lib, /usr/lib64 and /usr/lib gets added # automatically by ldconfig after parsing this file. # So, they do not need to be listed. ln -sf /usr/lib64/*pcre* /lib64 rm -f /etc/ld.so.cache ldconfig mkinitrd then reboot ... boot to 13.1 is OK again. noting on the booted 13.1 system, ldd /sbin/udevadm linux-vdso.so.1 (0x00007fff77ffe000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fe54b5d6000) librt.so.1 => /lib64/librt.so.1 (0x00007fe54b3ce000) libblkid.so.1 => /usr/lib64/libblkid.so.1 (0x00007fe54b193000) libkmod.so.2 => /lib64/libkmod.so.2 (0x00007fe54af7d000) libacl.so.1 => /lib64/libacl.so.1 (0x00007fe54ad74000) libc.so.6 => /lib64/libc.so.6 (0x00007fe54a9c5000) /lib64/ld-linux-x86-64.so.2 (0x00007fe54b7fa000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fe54a7c1000) !!! libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fe54a55b000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe54a33d000) libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x00007fe54a138000) liblzma.so.5 => /usr/lib64/liblzma.so.5 (0x00007fe549f12000) libz.so.1 => /lib64/libz.so.1 (0x00007fe549cfc000) libattr.so.1 => /lib64/libattr.so.1 (0x00007fe549af7000) There's clearly a conflict, or missing dep, re: pcre. The workaround I've used above is likely not the best approach. Not sure what is. Reproducible: Always Steps to Reproduce: 1. 2. 3. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c
A R
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c
zhang jiajun
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c1
Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c
A R
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c2
--- Comment #2 from A R
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c3
Dr. Werner Fink
ldd `which udevadm` | grep pcre
this does not show the fully truth, try LD_DEBUG=all udevadm --help 2>&1 | less +/libpcre and you may see that the first library which requires libpcre is /lib64/libselinux.so.1 .. also you may see from werner@linux:~> nm -D /lib64/libselinux.so.1 | grep pcre U pcre_compile U pcre_exec U pcre_free U pcre_free_study U pcre_study werner@linux:~> nm -D /sbin/udevadm | grep pcre werner@linux:~> that rise the question *how* you have updated your system and *if* you have ignored some dependencies during the update. This because werner@linux:~> rpm -qRf /lib64/libselinux.so.1 | grep pcre libpcre.so.1()(64bit) shows me that the package libselinux1 *does* require the libpcre.so.1 and the resolver does expand this to the package libpcre1. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c4
A R
this does not show the fully truth, try
LD_DEBUG=all udevadm --help 2>&1 | less +/libpcre
and you may see that the first library which requires libpcre is
/lib64/libselinux.so.1
as suggested, LD_DEBUG=all udevadm --help 2>&1 | less +/libpcre 89 8340: file=libpcre.so.1 [0]; needed by /lib64/libselinux.so.1 [0] 90 8340: find library=libpcre.so.1 [0]; searching 91 8340: search path=/usr/lib64/mpi/gcc/openmpi/lib64 (LD_LIBRARY_PATH) 92 8340: trying file=/usr/lib64/mpi/gcc/openmpi/lib64/libpcre.so.1 93 8340: search cache=/etc/ld.so.cache 94 8340: trying file=/usr/lib64/libpcre.so.1 95 8340: 96 8340: file=libpcre.so.1 [0]; generating link map 97 8340: dynamic: 0x00007f62fc7fad30 base: 0x00007f62fc596000 size: 0x0000000000265228 98 8340: entry: 0x00007f62fc597900 phdr: 0x00007f62fc596040 phnum: 7 99 8340: 100 8340: ...
... also you may see from
werner@linux:~> nm -D /lib64/libselinux.so.1 | grep pcre U pcre_compile U pcre_exec U pcre_free U pcre_free_study U pcre_study werner@linux:~> nm -D /sbin/udevadm | grep pcre werner@linux:~>
and again, nm -D /lib64/libselinux.so.1 | grep pcre U pcre_compile U pcre_exec U pcre_free U pcre_free_study U pcre_study nm -D /sbin/udevadm | grep pcre (empty)
that rise the question *how* you have updated your system and *if* you have ignored some dependencies during the update. This because
werner@linux:~> rpm -qRf /lib64/libselinux.so.1 | grep pcre libpcre.so.1()(64bit)
Well, I now see the source of the issue as you;ve pointed it out -- the dep on libpcre1 by libselinux.so.1
shows me that the package libselinux1 *does* require the libpcre.so.1 and the resolver does expand this to the package libpcre1.
I'm not sure what update you're asking about. This issue occurs on multiple systems. All of which have been upgraded from 12.3 -> 13.1 using zypper dup with a 13.1 install-DVD, then updated online. I'd not knowingly ignored any dependencies during the update process ... and, I noted no errors on any upgrade. noting, ldd /lib64/libselinux.so.1 linux-vdso.so.1 (0x00007fffe42ec000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f04ec2d8000) !! libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f04ec071000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f04ebe53000) libc.so.6 => /lib64/libc.so.6 (0x00007f04ebaa4000) /lib64/ld-linux-x86-64.so.2 (0x00007f04ec745000) in the case of, cat /etc/ld.so.conf /usr/lib64 /usr/local/lib64 /usr/local/lib include /etc/ld.so.conf.d/*.conf ldconfig mkinitrd unless that 1st /usr/lib64 *precedes* the other entries, I get the boot-fail as above. I do note that few libs in /lib64 reference other libs in /usr, ldd /lib64/* | grep usr ldd: /lib64/ast: not regular file ldd: /lib64/device-mapper: not regular file ldd: /lib64/engines: not regular file ldd: /lib64/ksh: not regular file ldd: warning: you do not have execution permission for `/lib64/libapparmor.a' ldd: warning: you do not have execution permission for `/lib64/libattr.so.1' ldd: warning: you do not have execution permission for `/lib64/libattr.so.1.1.0' libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007fd1fd8c8000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fd1fd257000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007f7700f73000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f7700902000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007fb3061e2000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fb305b71000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007fc3ed234000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fc3ecbc3000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007f2984e7b000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f298480a000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007ff924714000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007ff9240a3000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007f04f727a000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f04f679f000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007fb3c5c5f000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fb3c5184000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007f06ec0ba000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f06eba49000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007fa8dd88d000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fa8dd21c000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007fa993817000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fa9933ac000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007fd332484000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fd332019000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007fdf02983000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fdf01f4a000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007fbe7b0b9000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fbe7a680000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007ff362827000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007ff36219e000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007fa6bc008000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fa6bb97f000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007f5509436000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f5508fcc000) libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007fef9d670000) libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f223c163000) ldd: warning: you do not have execution permission for `/lib64/libfreebl3.chk' ldd: warning: you do not have execution permission for `/lib64/libimmunix.a' liblzma.so.5 => /usr/lib64/liblzma.so.5 (0x00007f975ab5c000) liblzma.so.5 => /usr/lib64/liblzma.so.5 (0x00007f5a7b130000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007fcb0f5b0000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fcb0e2cd000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007f08451fc000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f0843f19000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007f467b813000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f467a327000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007f356e8b4000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f356d3c8000) libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007ff064ede000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007ff064256000) libaudit.so.1 => /usr/lib64/libaudit.so.1 (0x00007fb14d415000) libaudit.so.1 => /usr/lib64/libaudit.so.1 (0x00007f8bdb0f8000) libaudit.so.1 => /usr/lib64/libaudit.so.1 (0x00007fe4b95d3000) libaudit.so.1 => /usr/lib64/libaudit.so.1 (0x00007f18fe057000) ldd: warning: you do not have execution permission for `/lib64/libpci.so.3' ldd: warning: you do not have execution permission for `/lib64/libpci.so.3.2.0' libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f84d480a000) libbz2.so.1 => /usr/lib64/libbz2.so.1 (0x00007f8432451000) libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f8431a03000) libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f15377c6000) libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007fdbabe32000) ldd: /lib64/multipath: not regular file ldd: /lib64/rsyslog: not regular file ldd: /lib64/security: not regular file whereas most other refs are to /lib64/* I'm not clear if that matters -- should libpcre.so.1, libudev.so.1 be in /lib64 by default? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c5
--- Comment #5 from Michael Schröder
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c6
--- Comment #6 from A R
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c
Xiyuan Liu
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c7
--- Comment #7 from A R
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c
A R
https://bugzilla.novell.com/show_bug.cgi?id=853293 https://bugzilla.novell.com/show_bug.cgi?id=853293#c8 --- Comment #8 from bug pool <883f6ce9@opayq.com> 2014-03-14 15:58:52 UTC --- Any interest from Suse devs in fixing this? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c9
--- Comment #9 from Dr. Werner Fink
From the side of systemd I can not do that much:
linux /suse/werner> rpm -q --whatrequires 'libpcre.so.1()(64bit)' libselinux1-2.1.13-4.1.21.x86_64 librasqal3-0.9.30-2.1.2.x86_64 grep-2.14-5.1.2.x86_64 postfix-2.9.6-7.4.1.x86_64 libpcrecpp0-8.33-2.1.2.x86_64 libpcreposix0-8.33-2.1.2.x86_64 swig-2.0.10-2.1.2.x86_64 zsh-5.0.2-8.1.4.x86_64 git-core-1.8.4-3.1.4.x86_64 git-svn-1.8.4-3.1.4.x86_64 libglib-2_0-0-2.38.2-8.2.x86_64 libkde4-4.11.3-52.3.x86_64 that is the libpcre is required in second order due libselinux: linux /suse/werner> ldd /usr/lib/systemd/systemd | grep libselinux libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fcf9c001000) and the libselinux is required linux /suse/werner> rpm -qR systemd | grep libselinux libselinux.so.1()(64bit) linux /suse/werner> rpm -qR udev | grep libselinux libselinux.so.1()(64bit) maybe we could use somehow `Requires(post)' in an automatic way to make sure that libselinux and all its requirements are there at the time the %post scriplet is executed to regenerate the initrd. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c10
--- Comment #10 from Stephan Kulow
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c12
Stephan Kulow
https://bugzilla.novell.com/show_bug.cgi?id=853293 https://bugzilla.novell.com/show_bug.cgi?id=853293#c14 bug pool <883f6ce9@opayq.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|883f6ce9@opayq.com | --- Comment #14 from bug pool <883f6ce9@opayq.com> 2014-03-30 03:28:14 UTC --- cd $(mktemp -d) gunzip -c /boot/initrd | cpio -i 99736 blocks ls bin/ config/ etc/ lib/ mkinitrd.config root/ sbin/ tmp/ var/ boot/ dev/ init* lib64/ proc/ run_all.sh sys/ usr/ find . | grep ld.so (empty) does that suggest a fix in user-land? or is there a fix in opensuse's mkinitrd implementation? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c15
--- Comment #15 from Stephan Kulow
https://bugzilla.novell.com/show_bug.cgi?id=853293 https://bugzilla.novell.com/show_bug.cgi?id=853293#c16 --- Comment #16 from bug pool <883f6ce9@opayq.com> 2014-05-18 18:37:05 UTC ---
mkinitrd should also cope with it
Is this bug, then, correctly assigned to:
Assignee: Thomas Renninger
https://bugzilla.novell.com/show_bug.cgi?id=853293
https://bugzilla.novell.com/show_bug.cgi?id=853293#c17
grant k
http://bugzilla.novell.com/show_bug.cgi?id=853293 bug pool <883f6ce9@opayq.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |trenn@suse.com Flags| |needinfo?(trenn@suse.com) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=853293
Thomas Renninger
a locally installed pcre I use for dev in, ls -al /usr/local/lib64
mkinitrd is gone for 13.2, it's not worth to look at this "very specific": "a locally installed pcre I use for dev in" problem. May be worth if this still happens in 13.2 (locally installed library fails to construct a working intitrd or similar). But there a totally other software stack is used today: dracut -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=853293
grant k
http://bugzilla.novell.com/show_bug.cgi?id=853293
Thomas Renninger
From your comment above: /etc/ld.so.conf --- /usr/lib64 +++ #/usr/lib64 /usr/lib64 does not exist in 13.2 and is not needed. Did you want to do what I did above?
There may be some misunderstanding and I do not get your point, but for me everything works as it should be. I close the bug again won't fix (for 13.1). If you still think something is wrong, please email me privately. Let's fiddle some details out first, before polluting the public bug... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=853293
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=853293
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=853293
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=853293
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=853293
http://bugzilla.novell.com/show_bug.cgi?id=853293#c21
--- Comment #21 from Swamp Workflow Management
participants (1)
-
bugzilla_noreply@novell.com