[opensuse-kernel] Why is IPV6 built in?
Is there some good reason why IPV6 is built into the kernel rather than as a module? There are a lot of systems where every DNS lookup has to timeout on the IPV6 lookup, thus IPV6 must be disabled. When it is a module, the YaST setting suffices, but if built in, it must be disabled on the kernel boot line in GRUB. That is confusing. Larry -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Tue, Nov 30, 2010 at 02:09:16PM -0600, Larry Finger wrote:
Is there some good reason why IPV6 is built into the kernel rather than as a module?
boot speed. But I thought we relented and pulled it out recently. What kernel flavor are you using?
There are a lot of systems where every DNS lookup has to timeout on the IPV6 lookup, thus IPV6 must be disabled. When it is a module, the YaST setting suffices, but if built in, it must be disabled on the kernel boot line in GRUB. That is confusing.
Agreed, I thought those issues were resolved already. Wasn't there a bug somewhere for this? thanks, greg k-h -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 11/30/2010 02:28 PM, Greg KH wrote:
On Tue, Nov 30, 2010 at 02:09:16PM -0600, Larry Finger wrote:
Is there some good reason why IPV6 is built into the kernel rather than as a module?
boot speed. But I thought we relented and pulled it out recently. What kernel flavor are you using?
2.6.37-rc3-git1-6-desktop from 11.4 M4 installed via "NET install i586 Build 0909".
There are a lot of systems where every DNS lookup has to timeout on the IPV6 lookup, thus IPV6 must be disabled. When it is a module, the YaST setting suffices, but if built in, it must be disabled on the kernel boot line in GRUB. That is confusing.
Agreed, I thought those issues were resolved already. Wasn't there a bug somewhere for this?
Perhaps, but I have not seen it. Larry -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Tue, Nov 30, 2010 at 02:51:05PM -0600, Larry Finger wrote:
On 11/30/2010 02:28 PM, Greg KH wrote:
On Tue, Nov 30, 2010 at 02:09:16PM -0600, Larry Finger wrote:
Is there some good reason why IPV6 is built into the kernel rather than as a module?
boot speed. But I thought we relented and pulled it out recently. What kernel flavor are you using?
2.6.37-rc3-git1-6-desktop from 11.4 M4 installed via "NET install i586 Build 0909".
Ah, yeah, try the non-desktop version, it should be a module there, right? thanks, greg k-h -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 11/30/2010 03:04 PM, Greg KH wrote:
Ah, yeah, try the non-desktop version, it should be a module there, right?
No, it is built in 2.6.37-rc3-git1-6-default as well. Larry -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Tue, Nov 30, 2010 at 12:28:14PM -0800, Greg KH wrote:
On Tue, Nov 30, 2010 at 02:09:16PM -0600, Larry Finger wrote:
Is there some good reason why IPV6 is built into the kernel rather than as a module?
boot speed. But I thought we relented and pulled it out recently. What kernel flavor are you using?
nope, it should now be =y everywhere commit 9d95188751f25b8244f78c3fec7c2647bd8274a8 Author: Jiri Bohac <jbohac@suse.cz> Date: Fri Mar 19 17:34:50 2010 +0100 - set CONFIG_IPV6=y for all flavours (bnc#561611)
There are a lot of systems where every DNS lookup has to timeout on the IPV6 lookup, thus IPV6 must be disabled. When it is a module, the YaST setting suffices, but if built in, it must be disabled on the kernel boot line in GRUB. That is confusing.
Agreed, I thought those issues were resolved already. Wasn't there a bug somewhere for this?
yes, this is solved, bnc#561611 -- YaST now does it via sysctl. It is important to keep CONFIG_IPV6=y everywhere or it will break again. I need to see why this is currently not true for x86_64/ec2, sparc64/net and i386/ec2. -- Jiri Bohac <jbohac@suse.cz> SUSE Labs, SUSE CZ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/30/2010 04:26 PM, Jiri Bohac wrote:
On Tue, Nov 30, 2010 at 12:28:14PM -0800, Greg KH wrote:
On Tue, Nov 30, 2010 at 02:09:16PM -0600, Larry Finger wrote:
Is there some good reason why IPV6 is built into the kernel rather than as a module?
boot speed. But I thought we relented and pulled it out recently. What kernel flavor are you using?
nope, it should now be =y everywhere
commit 9d95188751f25b8244f78c3fec7c2647bd8274a8 Author: Jiri Bohac <jbohac@suse.cz> Date: Fri Mar 19 17:34:50 2010 +0100
- set CONFIG_IPV6=y for all flavours (bnc#561611)
There are a lot of systems where every DNS lookup has to timeout on the IPV6 lookup, thus IPV6 must be disabled. When it is a module, the YaST setting suffices, but if built in, it must be disabled on the kernel boot line in GRUB. That is confusing.
Agreed, I thought those issues were resolved already. Wasn't there a bug somewhere for this?
yes, this is solved, bnc#561611 -- YaST now does it via sysctl. It is important to keep CONFIG_IPV6=y everywhere or it will break again. I need to see why this is currently not true for x86_64/ec2, sparc64/net and i386/ec2.
The ec2 flavors should be adjusted but sparc64 is under Jan Engelhardt's maintainership. - -Jeff - -- Jeff Mahoney SUSE Labs -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkz1dTQACgkQLPWxlyuTD7JMCwCdH2q/WA7E2U6ZKQib9tU7vleT N54AnjUnyidKCxn0GrTduFLcWbJo8o6P =KaoS -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Tuesday 2010-11-30 23:05, Jeff Mahoney wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/30/2010 04:26 PM, Jiri Bohac wrote:
On Tue, Nov 30, 2010 at 12:28:14PM -0800, Greg KH wrote:
On Tue, Nov 30, 2010 at 02:09:16PM -0600, Larry Finger wrote:
There are a lot of systems where every DNS lookup has to timeout on the IPV6 lookup, thus IPV6 must be disabled.
Where is the problem? Provided you are using v4 to do DNS queries -- if a DNS query does not return an AAAA, all is well. If it does return AAAA, but not have an IPv6 route to the target address, all is well. If you have a v6 route for ::/0, you have v6 Internet, or at least a router which advertises such. Disabling IPv6 on the client system is hardly a fix for router issues, in case the advertisement is wrongful.
Agreed, I thought those issues were resolved already. Wasn't there a bug somewhere for this?
yes, this is solved, bnc#561611 -- YaST now does it via sysctl. It is important to keep CONFIG_IPV6=y everywhere or it will break again. I need to see why this is currently not true for x86_64/ec2, sparc64/net and i386/ec2.
The ec2 flavors should be adjusted but sparc64 is under Jan Engelhardt's maintainership.
sparc64/net is one where every saved byte counts, so it's modules as far as the eye can see. If IPv4 was not so uglily tied into everything, even v4 would be a module. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 11/30/2010 04:22 PM, Jan Engelhardt wrote:
Where is the problem? Provided you are using v4 to do DNS queries -- if a DNS query does not return an AAAA, all is well. If it does return AAAA, but not have an IPv6 route to the target address, all is well. If you have a v6 route for ::/0, you have v6 Internet, or at least a router which advertises such. Disabling IPv6 on the client system is hardly a fix for router issues, in case the advertisement is wrongful.
Agreed, but one doesn't always own the router, or have the skill to change it's behavior. This issue is a recurring one on the Forums. My system works fine with IPv6 enabled. I do not test it often as my self-generated kernels have it disabled to save the build time. Larry -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Wednesday 2010-12-01 00:17, Larry Finger wrote:
On 11/30/2010 04:22 PM, Jan Engelhardt wrote:
Where is the problem? Provided you are using v4 to do DNS queries -- if a DNS query does not return an AAAA, all is well. If it does return AAAA, but not have an IPv6 route to the target address, all is well. If you have a v6 route for ::/0, you have v6 Internet, or at least a router which advertises such. Disabling IPv6 on the client system is hardly a fix for router issues, in case the advertisement is wrongful.
Agreed, but one doesn't always own the router, or have the skill to change it's behavior.
This issue is a recurring one on the Forums. My system works fine with IPv6 enabled. I do not test it often as my self-generated kernels have it disabled to save the build time.
If anything I suggest setting accept_ra=0 in sysctl, which works around this without completely compromising IPv6 functionality. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 30.11.10 at 23:05, Jeff Mahoney <jeffm@suse.de> wrote: On 11/30/2010 04:26 PM, Jiri Bohac wrote: It is important to keep CONFIG_IPV6=y everywhere or it will break again. I need to see why this is currently not true for x86_64/ec2, sparc64/net and i386/ec2.
The ec2 flavors should be adjusted but sparc64 is under Jan Engelhardt's maintainership.
I'm actually opposed to changing this back without knowing that it's actually used in the EC2 environment. Even more, I was actually considering making it (and other stuff - crypto comes to mind immediately) =m again in the -xen configs, unless it is known that these bits are used on the vast majority of systems. After all that's what modules are for, and I don't see the point of a bloated kernel (not even with the argument of boot speed, as I doubt anyone has ever tried to compare the run time effect [accumulated over the perhaps long periods of run time] with the boot time effect) when only a subset of users actually uses the features in question. The argument of sysconfig not being able to cope with IPV6=m is certainly bogus - if so, the tool needs to be fixed. Jan -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Wed, Dec 01, 2010 at 08:08:54AM +0000, Jan Beulich wrote:
On 30.11.10 at 23:05, Jeff Mahoney <jeffm@suse.de> wrote: On 11/30/2010 04:26 PM, Jiri Bohac wrote: It is important to keep CONFIG_IPV6=y everywhere or it will break again. I need to see why this is currently not true for x86_64/ec2, sparc64/net and i386/ec2.
The ec2 flavors should be adjusted but sparc64 is under Jan Engelhardt's maintainership.
I'm actually opposed to changing this back without knowing that it's actually used in the EC2 environment.
I guess you can set it to =n and see if someone complains then. If it turns out it is used, please make it =y. See below why.
Even more, I was actually considering making it (and other stuff - crypto comes to mind immediately) =m again in the -xen configs
Please don't...
unless it is known that these bits are used on the vast majority of systems. After all that's what modules are for, and I don't see the point of a bloated kernel (not even with the argument of boot speed, as I doubt anyone has ever tried to compare the run time effect [accumulated over the perhaps long periods of run time] with the boot time effect) when only a subset of users actually uses the features in question.
The argument of sysconfig not being able to cope with IPV6=m is certainly bogus - if so, the tool needs to be fixed.
... the sysctl configuration interface of IPv6 (and IPv4 as well, FWIW) is really really frustrating wrt the effect and availability of the tunables during boot. A simple example of the horror: Say you want to set accept_ra to 0 to disable autoconfiguration on an interface. To do this, you need to : - have IPv6 available (i.e. loaded if it is a module) - the interface must not yet be up, otherwise it is too late to stop it from autoconfiguring If you want to enable the interface with something like ifplugd/NM/udev when it appears, doing everything in the right order is quite complicated. If the scripts needed to check/load the ipv6 module, this would be a nightmare. Even worse, you may need to configure the network interface in initrd, etc... Disabling IPv6 (through YaST/sysconfig) is another problem. The old method of blacklisting the module does no longer work, because e.g. the bonding module depends on the ipv6 module. Disabling IPv6 with the sysctl cannot be done unless the module is already loaded, at which point, interfaces may have started autoconfiguring. The only option left would be a kernel command-line option. Having ipv6 built-in everywhere minimises the frustration of people putting their own ipv6 setting to /etc/sysctl.conf. Please keep it consistent for all archs/flavours. -- Jiri Bohac <jbohac@suse.cz> SUSE Labs, SUSE CZ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 01.12.10 at 13:18, Jiri Bohac <jbohac@suse.cz> wrote: On Wed, Dec 01, 2010 at 08:08:54AM +0000, Jan Beulich wrote: I'm actually opposed to changing this back without knowing that it's actually used in the EC2 environment.
I guess you can set it to =n and see if someone complains then. If it turns out it is used, please make it =y. See below why.
I don't think we should bluntly disable anything.
The argument of sysconfig not being able to cope with IPV6=m is certainly bogus - if so, the tool needs to be fixed.
... the sysctl configuration interface of IPv6 (and IPv4 as well, FWIW) is really really frustrating wrt the effect and availability of the tunables during boot. A simple example of the horror:
Say you want to set accept_ra to 0 to disable autoconfiguration on an interface. To do this, you need to : - have IPv6 available (i.e. loaded if it is a module) - the interface must not yet be up, otherwise it is too late to stop it from autoconfiguring
If you want to enable the interface with something like ifplugd/NM/udev when it appears, doing everything in the right order is quite complicated. If the scripts needed to check/load the ipv6 module, this would be a nightmare. Even worse, you may need to configure the network interface in initrd, etc...
All of this should be a non-issue as long as ipv6.ko gets loaded early enough on systems that actually need it. I can't see what's that difficult here.
Disabling IPv6 (through YaST/sysconfig) is another problem. The old method of blacklisting the module does no longer work, because e.g. the bonding module depends on the ipv6 module. Disabling IPv6 with the sysctl cannot be done unless the module is already loaded, at which point, interfaces may have started autoconfiguring. The only option left would be a kernel command-line option.
Having ipv6 built-in everywhere minimises the frustration of people putting their own ipv6 setting to /etc/sysctl.conf.
When put there through YaST, it should simultaneously store appropriate module load options or hooks into a modprobe configuration file. When put there manually, people should know what they're doing (i.e. whether they need to put it there, in modprobe's config files, or both). Jan -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Wednesday 2010-12-01 13:18, Jiri Bohac wrote:
... the sysctl configuration interface of IPv6 (and IPv4 as well, FWIW) is really really frustrating wrt the effect and availability of the tunables during boot. A simple example of the horror:
Say you want to set accept_ra to 0 to disable autoconfiguration on an interface. To do this, you need to : - have IPv6 available (i.e. loaded if it is a module) - the interface must not yet be up, otherwise it is too late to stop it from autoconfiguring
If you want to enable the interface with something like ifplugd/NM/udev when it appears, doing everything in the right order is quite complicated. If the scripts needed to check/load the ipv6 module, this would be a nightmare. Even worse, you may need to configure the network interface in initrd, etc...
So why the heck don't we just let accept_ra default to 0 in the kernel source, and enable it when needed? That's a much less error prone approach. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Wed, Dec 01, 2010 at 08:08:54AM +0000, Jan Beulich wrote:
I'm actually opposed to changing this back without knowing that it's actually used in the EC2 environment. Even more, I was actually considering making it (and other stuff - crypto comes to mind immediately) =m again in the -xen configs, unless it is known that these bits are used on the vast majority of systems. After all that's what modules are for, and I don't see the point of a bloated kernel (not even with the argument of boot speed, as I doubt anyone has ever tried to compare the run time effect [accumulated over the perhaps long periods of run time] with the boot time effect) when only a subset of users actually uses the features in question.
I don't understand what you mean by "run time effect" here. How does having the ipv6 module, or any other module, built into the kernel, have any effect on the runtime speed of the kernel over time? confused, greg k-h -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 01.12.10 at 17:08, Greg KH <gregkh@suse.de> wrote: On Wed, Dec 01, 2010 at 08:08:54AM +0000, Jan Beulich wrote: I'm actually opposed to changing this back without knowing that it's actually used in the EC2 environment. Even more, I was actually considering making it (and other stuff - crypto comes to mind immediately) =m again in the -xen configs, unless it is known that these bits are used on the vast majority of systems. After all that's what modules are for, and I don't see the point of a bloated kernel (not even with the argument of boot speed, as I doubt anyone has ever tried to compare the run time effect [accumulated over the perhaps long periods of run time] with the boot time effect) when only a subset of users actually uses the features in question.
I don't understand what you mean by "run time effect" here. How does having the ipv6 module, or any other module, built into the kernel, have any effect on the runtime speed of the kernel over time?
Increased pressure on TLBs and, less significantly (because of the smaller granularity), caches. Jan -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Wed, Dec 01, 2010 at 04:56:51PM +0000, Jan Beulich wrote:
On 01.12.10 at 17:08, Greg KH <gregkh@suse.de> wrote: I don't understand what you mean by "run time effect" here. How does having the ipv6 module, or any other module, built into the kernel, have any effect on the runtime speed of the kernel over time?
Increased pressure on TLBs and, less significantly (because of the smaller granularity), caches.
How come? How does code/data mapped in the kernel address space incerease cache/TLB pressure if it is not being accessed? -- Jiri Bohac <jbohac@suse.cz> SUSE Labs, SUSE CZ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 01.12.10 at 18:33, Jiri Bohac <jbohac@suse.cz> wrote: On Wed, Dec 01, 2010 at 04:56:51PM +0000, Jan Beulich wrote:
On 01.12.10 at 17:08, Greg KH <gregkh@suse.de> wrote: I don't understand what you mean by "run time effect" here. How does having the ipv6 module, or any other module, built into the kernel, have any effect on the runtime speed of the kernel over time?
Increased pressure on TLBs and, less significantly (because of the smaller granularity), caches.
How come? How does code/data mapped in the kernel address space incerease cache/TLB pressure if it is not being accessed?
Unless the used and unused regions of address space are each contiguous (which they certainly aren't), at least the TLBs need to map unused code along with all the used regions. The total size of mapped space is what matters for determining TLB pressure. For the caches, code would likely be separated well enough to mostly avoid bringing in unused bits (apart from the effects of false speculation), but for data the picture may already be different again when considering many small items coming from various components ending up in the same cache line. Jan -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Thursday 2010-12-02 09:49, Jan Beulich wrote:
How come? How does code/data mapped in the kernel address space incerease cache/TLB pressure if it is not being accessed?
Unless the used and unused regions of address space are each contiguous (which they certainly aren't), at least the TLBs need to map unused code along with all the used regions. The total size of mapped space is what matters for determining TLB pressure. For the caches, code would likely be separated well enough to mostly avoid bringing in unused bits (apart from the effects of false speculation), but for data the picture may already be different again when considering many small items coming from various components ending up in the same cache line.
Isn't kernel code, like stuff allocated through kmalloc, contiguous in physical memory too? -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 02.12.10 at 11:54, Jan Engelhardt <jengelh@medozas.de> wrote:
On Thursday 2010-12-02 09:49, Jan Beulich wrote:
How come? How does code/data mapped in the kernel address space incerease cache/TLB pressure if it is not being accessed?
Unless the used and unused regions of address space are each contiguous (which they certainly aren't), at least the TLBs need to map unused code along with all the used regions. The total size of mapped space is what matters for determining TLB pressure. For the caches, code would likely be separated well enough to mostly avoid bringing in unused bits (apart from the effects of false speculation), but for data the picture may already be different again when considering many small items coming from various components ending up in the same cache line.
Isn't kernel code, like stuff allocated through kmalloc, contiguous in physical memory too?
Sure, but what would this have to do with TLB usage? Jan -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
Hi, On Tue, Nov 30, 2010 at 02:09:16PM -0600, Larry Finger wrote:
Is there some good reason why IPV6 is built into the kernel rather than as a module?
originally, IPv6 got moved in the kernel in our moblin kernel branch to speed up booting. We quickly found that having it as a module in some branches and built-in in other branches caused a lot of pain (differrent behaviour when setting various ipv6 sysctl parameters or when disabling it). Thus we decided to unify all the branches by always building it in. Also, the bonding module now depends on the ipv6 module, so the old method of disablling IPv6 by blacklisting the ipv6 module breaks bonding.
There are a lot of systems where every DNS lookup has to timeout on the IPV6 lookup, thus IPV6 must be disabled.
... or the network infrastructure should be fixed, really. Normally, you should not see any timeouts with IPv6 enabled. When a host has both AAAA and A records in the DNS, an attempt to connect to it will(*) try IPv6 first and then fall back to IPv4. When there is no IPv6 connectivity on the network, the connect(2) call should return immediately and cause no delay at all. One common cause of possible timeout delays is badly configured IPv6 router that advertises an IPv6 default route on the local network without providing the connectivity or sending proper ICMPv6 Destination Unreachable messages.
When it is a module, the YaST setting suffices, but if built in, it must be disabled on the kernel boot line in GRUB.
Another method of disabling IPv6 is sysctl net.ipv6.conf.all.disable_ipv6=1 Also, YaST still has the option to disable IPv6 and does this by setting this sysctl option in /etc/sysctl.conf (*) this is an over-simplification the policy can be tuned with gai.conf and also IPv6 does not always have precedence in some cases (e.g. 2002::/16 ... ) HTH, -- Jiri Bohac <jbohac@suse.cz> SUSE Labs, SUSE CZ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Tue, 30 Nov 2010, Jiri Bohac wrote:
There are a lot of systems where every DNS lookup has to timeout on the IPV6 lookup, thus IPV6 must be disabled. ... or the network infrastructure should be fixed, really.
Yes, in principle. Practically, more often than not the network infrastructure is not under your control (especially when using a notebook and travelling) and I can tell that having to await those timeouts _is_ painful.
One common cause of possible timeout delays is badly configured IPv6 router that advertises an IPv6 default route on the local network without providing the connectivity or sending proper ICMPv6 Destination Unreachable messages.
Or a route/DNS server that does not respond to AAAA requests but let's them run into a timeout. Yes, that sucks. Gerald -- Dr. Gerald Pfeifer <gp@novell.com> Director Product Management, SUSE Linux Enterprise, openSUSE, Appliances -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
participants (7)
-
Gerald Pfeifer -
Greg KH -
Jan Beulich -
Jan Engelhardt -
Jeff Mahoney -
Jiri Bohac -
Larry Finger