[Bug 972471] New: Slow boot: 33s waiting for wicked interfaces to start
http://bugzilla.suse.com/show_bug.cgi?id=972471 Bug ID: 972471 Summary: Slow boot: 33s waiting for wicked interfaces to start Classification: openSUSE Product: openSUSE Distribution Version: Leap 42.1 Hardware: x86-64 OS: openSUSE 42.1 Status: NEW Severity: Major Priority: P5 - None Component: Network Assignee: bnc-team-screening@forge.provo.novell.com Reporter: studio@anchev.net QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- A few days ago I ran a system update through 'zypper up' and I noticed that systemd and wicked were updated. Since then every time my system boots slowly because of this: Code: A start job is running for wicked managed network interfaces (33s / no limit)[ OK ] I haven't changed any network settings whatsoever. Before the update my system used to boot in about 15 seconds and I never had to wait for anything like that. Now the boot is x3 slower which is quite annoying as I am using a dual boot. Others confirmed the same problem in the forums: https://forums.opensuse.org/showthread.php/514501-Slow-boot-33s-waiting-for-... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
Chenzi Cao
http://bugzilla.suse.com/show_bug.cgi?id=972471
Per Jessen
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c1
Dave Howorth
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c2
--- Comment #2 from George Anchev
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c3
--- Comment #3 from Dave Howorth
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c4
Pawel Wieczorkiewicz
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c5
--- Comment #5 from George Anchev
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c6
Marius Tomaschewski
I have encountered this problem on a new install of Leap 42.1 on an Acer XC-705. In my case the wicked delay went away after I disabled IPv6 in YaST.
There is still a noticeable delay that seems to be attributable to ModemManager and/or SuSE_firewall2
SuSEfirewall2 as wicked is not using ModemManager yet. Does it also happen when you disable SuSEfirewall2(_init) services? Also, try to set: /etc/sysconfig/SuSEfirewall2, FW_BOOT_FULL_INIT=yes The SuSEfirewall2_init script blocks most icmpv6 traffic including IPv6 RA used by the kernel to set default route + autoip and start dhcpv6 [managed mode] processing in wicked. FW_BOOT_FULL_INIT=yes causes to apply final rules already in the init phase, so the packets are not blocked while the setup. Except of this, the kernel forgets to send a NEWLINK when it receives RA in some cases. When there is also a prefix in the RA, wicked will workaround it, but when not, it does not get informed. See also: http://patchwork.ozlabs.org/patch/512486/ http://patchwork.ozlabs.org/patch/512646/ which still aren't applied to the kernels... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c7
George Anchev
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c8
--- Comment #8 from Marius Tomaschewski
http://bugzilla.suse.com/show_bug.cgi?id=972471
Marius Tomaschewski
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c9
--- Comment #9 from George Anchev
Please disable SuSEfirewall2, enable debug, reboot and attach "wicked show-config" and "journalctl -b -o short-precise" outputs when it still happens with disabled SuSEfirewall2, see also:
The link explains how to enable debug. How do I disable it after that and restore things back to the original (non-debugging) state? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c10
--- Comment #10 from Marius Tomaschewski
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c11
--- Comment #11 from Per Jessen
Please disable SuSEfirewall2, enable debug, reboot and attach "wicked show-config" and "journalctl -b -o short-precise" outputs when it still happens with disabled SuSEfirewall2, see also:
FYI, in don't use the susefirewall, but I can confirm the 30s wait as described by George and Dave. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c15
--- Comment #15 from Pawel Wieczorkiewicz
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c16
--- Comment #16 from George Anchev
After upgrade to Leap the interface naming scheme changes: interfaces by default are named again as eth0, eth1, ... .
I see you still have olf config files: ifcfg-eno1 etc. Therefore a wicked client waits for $WAIT_FOR_INTERFACES (30s) timeout for all interfaces that have configs specified (even though the interface with old names are not to show up). Please verify what you interfaces names are and leave only relevant configs (or change the STARTMODE to off or manual in the old ones).
When I open YaST->Network settings I see eth0 and eth1. What are you talking about? Can you please explain how to fix the thing you explain? (sorry, I am not an expert) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c17
--- Comment #17 from Pawel Wieczorkiewicz
When I open YaST->Network settings I see eth0 and eth1. What are you talking about?
Can you please explain how to fix the thing you explain? (sorry, I am not an expert)
Sure. Please inspect manually a directory /etc/sysconfig/network. Look for ifcfg-NAME files that do not correspond to the interfaces present in your system. I see in the logs you provided that your system still has: /etc/sysconfig/network/ifcfg-enp2s0 /etc/sysconfig/network/ifcfg-eno1 (remove them) along the new ones: /etc/sysconfig/network/ifcfg-eth0 /etc/sysconfig/network/ifcfg-eth1 (which are ok and should stay) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c18
--- Comment #18 from George Anchev
Sure. Please inspect manually a directory /etc/sysconfig/network. Look for ifcfg-NAME files that do not correspond to the interfaces present in your system. I see in the logs you provided that your system still has: /etc/sysconfig/network/ifcfg-enp2s0 /etc/sysconfig/network/ifcfg-eno1 (remove them)
along the new ones: /etc/sysconfig/network/ifcfg-eth0 /etc/sysconfig/network/ifcfg-eth1 (which are ok and should stay)
Thanks. Done and rebooted. No change at all - still waiting 33 sec for wicked... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c19
--- Comment #19 from Pawel Wieczorkiewicz
Thanks. Done and rebooted. No change at all - still waiting 33 sec for wicked...
I see that eth0 has no link up event: Apr 04 12:24:43.304117 i7 wickedd-nanny[1350]: eth0: state=device-up want=network-up, wait-for=link-up Is this interface connected to some network? If not (or there is no carrier on the cable of that network), please append: LINK_REQUIRED=no into /etc/sysconfig/network/ifcfg-eth0 and try over. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c20
--- Comment #20 from George Anchev
I see that eth0 has no link up event: Apr 04 12:24:43.304117 i7 wickedd-nanny[1350]: eth0: state=device-up want=network-up, wait-for=link-up
Is this interface connected to some network? If not (or there is no carrier on the cable of that network), please append: LINK_REQUIRED=no into /etc/sysconfig/network/ifcfg-eth0 and try over.
eth0 is my second interface. It is used for a second internal LAN (which is not always available and online, I turn on the network switch only when I need to share the Internet to other computers). I have not changed its configuration in the 6 months at all. So it has always been the way it is and never caused this delay. Should I still do something after this explanation? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c21
--- Comment #21 from Marius Tomaschewski
Created attachment 671428 [details] journalctl -b -o short-precise
Either there is still something blocking router icmpv6 responses or the kernel didn't forwarded it to userspace (patches mentioned in comment 6). It is not visible in the logs, but please disable the firewall using: systemctl disable SuSEfirewall2.service SuSEfirewall2_init.service Maybe this helps to see what's wrong: Write a /etc/wicked/scripts/pre-up/ip6tables-dump script with following content: #!/bin/bash set -e action="$1" ifname="$2" test "X$ifname" != "X" -a -d "/sys/class/net/$ifname" test -x /usr/sbin/ip6tables-save echo "*** $(date -u):" &>> "/tmp/ip6tables.${ifname}.${action}.dump" /usr/sbin/ip6tables-save &>> "/tmp/ip6tables.${ifname}.${action}.dump" echo "" &>> "/tmp/ip6tables.${ifname}.${action}.dump" ip addr show dev "$ifname" &>> "/tmp/ip6tables.${ifname}.${action}.dump" echo "" &>> "/tmp/ip6tables.${ifname}.${action}.dump" make it executable: chmod +x /etc/wicked/scripts/pre-up/ip6tables-dump And add either: PRE_UP_SCRIPT="wicked:pre-up" or PRE_UP_SCRIPT="wicked:pre-up/ip6tables-dump" to ifcfg-eth0 file. After reboot, attach /tmp/ip6tables.eth0.pre-up. Further, installing "radvd" and running "radvdump" for a while (10min) and/or calling "ip link set down eth0 ; ip link set up eth0" would show what is in the RA. Workaround: You can provide the IPv6 RA managed/other-config flags explicitly in the ifcfg-eth0 by adding DHCLIENT6_MODE='managed' to it. (In reply to George Anchev from comment #14)
Here is my wicked debug info:
After disabling debug (restoring to original as explained) I also tested with IPv6 disabled (with and without SuSE_firewall2). In all cases - absolutely no change. Still waiting for 33 seconds.
You have configuration for ifcfg-{eno1,eth0,enp2s0,eth1} ethernet interfaces and even when the eth1 setup basically finished after 4 seconds: Apr 04 12:24:39.018645 i7 wicked[1351]: Executing: /usr/sbin/wicked --systemd ifup all ... Apr 04 12:24:43.303684 i7 wickedd-nanny[1350]: eth1: successfully transitioned from addrconf-up to network-up Apr 04 12:24:43.303904 i7 wicked[1351]: received signal progressInfo; object_path=/org/opensuse/Network/Nanny/Interface/3; target_state=network-up, state_name=network-up it still waits for the other requested interfaces: Apr 04 12:24:43.304117 i7 wickedd-nanny[1350]: eth0: state=device-up want=network-up, wait-for=link-up Apr 04 12:24:43.304330 i7 wickedd-nanny[1350]: waiting for 1 devices to become ready (1 explicitly requested) Apr 04 12:24:43.304571 i7 wickedd-nanny[1350]: ni_nanny_recheck(eno1) Apr 04 12:24:43.304771 i7 wickedd-nanny[1350]: eno1: no applicable policies Apr 04 12:24:43.304983 i7 wickedd-nanny[1350]: ni_nanny_recheck(enp2s0) Apr 04 12:24:43.305200 i7 wickedd-nanny[1350]: enp2s0: no applicable policies until the timeout is reached: Apr 04 12:25:09.087723 i7 wicked[1351]: Interface wait time reached This has not much in common with ipv6. Not ignoring missed STARTMODE=hotplug configs is bug #945601. But in your case, it are even STARTMODE=boot devices, so it works as expected here. An rm -v /etc/sysconfig/network/ifcfg-{eno1,eth0,enp2s0} will stop waiting for eno1 enp2s0 (which are eth0 + eth1 probably) + eth0. When eth0 is used, but currently not connected, you mv ifcfg-eth0 .ifcfg-eth0 as workaround to avoid the timeout until STARTMODE=hotplug & co is fixed. It would be possible to set LINK_REQUIRED=no for eth0, but this is generally IMO not a good idea (except in special cases) and configures IPs + routes on unconnected interfaces (-> black hole) and ignoring network layer order. I'd tend to resolve this report as dup of #945601 (even the config is not using STARTMODE=hotplug). Does it make sense to split the report into multiple? We have already two different scenarios here. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c22
--- Comment #22 from George Anchev
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c23
--- Comment #23 from Per Jessen
(In reply to Per Jessen from comment #13)
Created attachment 671428 [details] journalctl -b -o short-precise
Either there is still something blocking router icmpv6 responses or the kernel didn't forwarded it to userspace (patches mentioned in comment 6).
It is not visible in the logs, but please disable the firewall using: systemctl disable SuSEfirewall2.service SuSEfirewall2_init.service
Hi Marius sorry, I forgot to mention it: # systemctl status SuSEfirewall2 SuSEfirewall2.service - SuSEfirewall2 phase 2 Loaded: loaded (/usr/lib/systemd/system/SuSEfirewall2.service; disabled) Active: inactive (dead)
Maybe this helps to see what's wrong: Write a /etc/wicked/scripts/pre-up/ip6tables-dump script with following content:
[snip]
And add either: PRE_UP_SCRIPT="wicked:pre-up" or PRE_UP_SCRIPT="wicked:pre-up/ip6tables-dump"
to ifcfg-eth0 file. After reboot, attach /tmp/ip6tables.eth0.pre-up.
Okay, willdo. I'm pretty certain there will be nothing there.
Further, installing "radvd" and running "radvdump" for a while (10min) and/or calling "ip link set down eth0 ; ip link set up eth0" would show what is in the RA.
I'm running radvdump now. I have access to radvd on the router too.
Workaround: You can provide the IPv6 RA managed/other-config flags explicitly in the ifcfg-eth0 by adding DHCLIENT6_MODE='managed' to it.
Oh, the boot-time isn't such a big deal for me, it's just weird. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c24
--- Comment #24 from Per Jessen
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c25
--- Comment #25 from Per Jessen
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c51
Marius Tomaschewski
It is in no way related to cables or anything else. As I mentioned - same settings, same hardware, same network, same cables. The only different thing is this extra 33 seconds.
OK, then please install the old *wicked* version which still worked and provide debug as described in comment 44 from both versions. A 'hwinfo --netcard | grep "Driver Modules"' will show you the module names to use in rmmod/modprobe lines instead of tg3. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c52
Marius Tomaschewski
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c53
--- Comment #53 from George Anchev
OK, then please install the old *wicked* version which still worked and provide debug as described in comment 44 from both versions.
Which one is the old? In YaST I search or wicked and see: 0.6.30-6.1-x86_64 from vendor openSUSE (Installed) 0.6.28-3.1-x86_64 from openSUSE-leap/42.1-Update with priority 99 and vendor openSUSE 0.6.27-1.1-x86_64 from Main Repository (OSS) with priority 99 and vendor openSUSE Do I have to downgrade wicked-service and libwicked-0-6 too?
A 'hwinfo --netcard | grep "Driver Modules"' will show you the module names to use in rmmod/modprobe lines instead of tg3.
This gives: Driver Modules: "e1000e" Driver Modules: "r8169" Also do I have to re-debug again for current version as I have already sent the debug for it in an earlier comment? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c55
--- Comment #55 from Marius Tomaschewski
(In reply to Marius Tomaschewski from comment #52)
Per, does the change / RPMs from comment 45 remove the dhcpv6 delays for you?
Yes, I installed yesterday, and observed a delay of about 12secs, much better. Is that about what you expected?
Yes, I'd say 12sec are OK on a cold/unused link - depends on the NIC [how long it needs to initialize, to detect carrier, forwards packets after it reported link UP] and if your router is using a "3sec silence between answers" or not (some do). (In reply to George Anchev from comment #53)
(In reply to Marius Tomaschewski from comment #51) Which one is the old? In YaST I search or wicked and see:
0.6.30-6.1-x86_64 from vendor openSUSE (Installed) 0.6.28-3.1-x86_64 from openSUSE-leap/42.1-Update with priority 99 and vendor openSUSE 0.6.27-1.1-x86_64 from Main Repository (OSS) with priority 99 and vendor openSUSE
I've no idea which version were working for you before. See /var/log/zypp/history, it should show what were installed when.
Do I have to downgrade wicked-service and libwicked-0-6 too?
Yes please. It should be not even possible to omit them.
Also do I have to re-debug again for current version as I have already sent the debug for it in an earlier comment?
I'd say not the current (old 0.6.30 in the meantime), but master build: http://download.opensuse.org/repositories/network:/wicked:/master/openSUSE_L... which contains the missed kernel event workaround fix from comment 46. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c56
--- Comment #56 from George Anchev
(In reply to Per Jessen from comment #54)
(In reply to Marius Tomaschewski from comment #52)
Per, does the change / RPMs from comment 45 remove the dhcpv6 delays for you?
Yes, I installed yesterday, and observed a delay of about 12secs, much better. Is that about what you expected?
Yes, I'd say 12sec are OK on a cold/unused link - depends on the NIC [how long it needs to initialize, to detect carrier, forwards packets after it reported link UP] and if your router is using a "3sec silence between answers" or not (some do).
(In reply to George Anchev from comment #53)
(In reply to Marius Tomaschewski from comment #51) Which one is the old? In YaST I search or wicked and see:
0.6.30-6.1-x86_64 from vendor openSUSE (Installed) 0.6.28-3.1-x86_64 from openSUSE-leap/42.1-Update with priority 99 and vendor openSUSE 0.6.27-1.1-x86_64 from Main Repository (OSS) with priority 99 and vendor openSUSE
I've no idea which version were working for you before. See /var/log/zypp/history, it should show what were installed when.
Do I have to downgrade wicked-service and libwicked-0-6 too?
Yes please. It should be not even possible to omit them.
Also do I have to re-debug again for current version as I have already sent the debug for it in an earlier comment?
I'd say not the current (old 0.6.30 in the meantime), but master build:
http://download.opensuse.org/repositories/network:/wicked:/master/ openSUSE_Leap_42.1/
which contains the missed kernel event workaround fix from comment 46.
Ok, I have tried all 3 versions: 0.6.30 0.6.31 (from master) 0.6.28 (which I found in the zypper history log) - this one I even tried with reverting to the previous version of systemd With all these the delay is still 33 seconds. Bummer. I also tried to set eth0 to "On cable connection" and to "Never". Only when set to boot "Never" the delay is gone. I followed the procedure from comment 44. I didn't reboot between creating the logs. The exact thing I did was: 1. Go to YaST and install the package versions 2. Execute commands: rcwickedd restart tcpdump -envfi any icmp6 2>&1 | systemd-cat -t "tcpdump" & ip monitor link address prefix neigh route | systemd-cat -t "ip-monitor" & rmmod r8169 rmmod e1000e cursor=`journalctl -n 10 -o export | grep ^__CURSOR= | sed -e 's|^__CURSOR=||' | head -1` modprobe r8169 modprobe e1000e sleep 40 ip a s | systemd-cat -t "ip-addrs" ip -6 r s | systemd-cat -t "ip-route6" sysctl -a | grep ^net.ipv6.conf | systemd-cat -t "ipv6-sysctls" kill $(jobs -lp) journalctl -c "$cursor" -o short-precise > journal-XXX.txt where XXX is the version of wicked as you will see in the 3 log files: https://goo.gl/mnSuJx Another thing I would like to ask for further clarification: (In reply to Marius Tomaschewski from comment #49)
(In reply to George Anchev from comment #48)
Also - if I lower manually WAIT_FOR_INTERACES to (say) 3 seconds - what could be the negative impact?
ifup will return after 3 sec, setup will happen in background and when you have services which bind to specific IPs, the'll fail.
How do I check which these services are and if they bind to specific IPs? The internal zone interface (eth0) is used for sharing the internet connection to that internal network + file transfers through sftp and smb. All I need is to make sure that when that network is online (the net switch is turned on) it will be possible to use these network functions + of course get rid of the 33 sec delay. I hope you can help. Thanks. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471 http://bugzilla.suse.com/show_bug.cgi?id=972471#c61 ILYA INDIGO <13ilya@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |13ilya@gmail.com --- Comment #61 from ILYA INDIGO <13ilya@gmail.com> --- I'm on the openSUSE Tumbleweed has long been a similar problem. I'm thinking to configure the network through systemd. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
Sagi Ben Akiva
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c62
--- Comment #62 from Bernhard Wiedemann
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c64
--- Comment #64 from Bernhard Wiedemann
http://bugzilla.suse.com/show_bug.cgi?id=972471
Swamp Workflow Management
http://bugzilla.suse.com/show_bug.cgi?id=972471
Swamp Workflow Management
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c65
Marius Tomaschewski
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c66
Marius Tomaschewski
http://bugzilla.suse.com/show_bug.cgi?id=972471
Swamp Workflow Management
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c67
--- Comment #67 from Swamp Workflow Management
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c68
--- Comment #68 from George Anchev
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c69
--- Comment #69 from Swamp Workflow Management
http://bugzilla.suse.com/show_bug.cgi?id=972471
Swamp Workflow Management
http://bugzilla.suse.com/show_bug.cgi?id=972471
Swamp Workflow Management
http://bugzilla.suse.com/show_bug.cgi?id=972471 Илья Индиго <13ilya@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC|13ilya@gmail.com | -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=972471
Swamp Workflow Management
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c70
--- Comment #70 from Swamp Workflow Management
http://bugzilla.suse.com/show_bug.cgi?id=972471
Swamp Workflow Management
http://bugzilla.suse.com/show_bug.cgi?id=972471
http://bugzilla.suse.com/show_bug.cgi?id=972471#c71
--- Comment #71 from Swamp Workflow Management
http://bugzilla.suse.com/show_bug.cgi?id=972471
Swamp Workflow Management
participants (1)
-
bugzilla_noreply@novell.com