Feature changed by: Jakub Rusinek (liviopl) Feature #305694, revision 80 Title: Separate Desktop / Server Kernels openSUSE-11.2: Done Priority Requester: Important Projectmanager: Important Requested by: Dean Hilkewich (deanjo13) Description: It seems to me that the current default kernels are somewhat hurting openSUSE's performance perception. Current kernel configs are OK but are not very well suited for desktop usage. In the future I would like to see a kernel package that is optimized for desktop usage. Current timer settings and no preemption really (sometimes drastically) hurts openGL performance and applications such as wine and causes alot of issues such as audio studdering. It would be nice to see a separate desktop performance kernel package with options such as Preemption model set to Preemptable Kenel (low-latency Desktop) and Timer Frequency Set to 1000 Hz, HPET support, Tickless System, disable optimize for size, disable Control Group support and disable Group CPU scheduler. You could also disable items and modules that are extremely rare in a desktop environment such as ATM support, Infiniband etc etc as these are not typically used in a desktop scenario which would be a large majority of openSUSE users. Further performance enhancements would also be done through out the system aimed at desktop use as well such as disabling barriers (even making it a simple checkmark option in the partitioner). Such optimizations for desktop usage can overcome openSUSE's reputation as being slower then the other mainstream distro's. The kernel settings alone can make up to a 30-40% increase in framerates in wine games for example and can cure alot of hiccups in multimedia apps. Discussion: #1: Bernhard Friedreich (bernhard1234) (2009-01-18 15:25:11) wow.. I didn't know about that facts.. I vote for that one! (running both a server and multiple notebooks on openSUSE) #2: Jan Engelhardt (jengelh) (2009-01-19 09:48:49) So why do not you use the RT kernel if your problems are that bad? Also, HPET and tickless is already enabled if you have not noticed. If you suggest disabling Optimize For Size, then I disclaim that you know all of the facts. A smaller kernel results in less instructions to fetch and run, and as such, less execution time to complete a given task. Unless you have specific numbers that an unloaded ATM module slows down your daily operations, it is not going to go away just because you think it is responsible for speed issues. Because as I read the source, it seems to me that it merely introduces one test and branch to the IPv4 ARP code, and that is just so small it is ridiculous to debate about. I do however, would want to see the kernel package being split up so as to have drivers that "probably are not" desktop- related in this decade, like ATM, in a separate installable package. #3: Dean Hilkewich (deanjo13) (2009-01-21 06:12:22) Yes I realize that HPET and tickless are already in there by default. I'm just pointing out settings that do make a difference for desktop use. Optimize for Size (=Os) can in theory reduce the executable size and can in some cases have small increases due to better cache usage, however in real life practice on a desktop you will find that disabling it (-02) does give a real life performance gain. Feel free to benchmark on your own as well (BTW Grub starts looking pretty squished with tones of kernels installed). These settings have been tested tried and true in the mailing lists, forums, etc. The RT kernel still defaults to a 250 Hz timer, Control Group support and Group CPU. All of these hurt multimedia and GL app performance. Easiest way of testing this for yourself is put a demanding game in wine such as command and conquer 3 in wine and play it or try to play high def content on a system with marginal system specs to do so. They make all the world of difference. The reason I recommended removing ATM, Infniband etc etc etc for the same reasons that you would like to see a separate kernel package for little if ever used anymore modules. I'm sure if you went through opensuse-help's own mailing list you can see these configs mentioned more then a few times and the results they give users afterwards. #4: Dmitry Mittov (michael_knight) (2009-01-22 08:40:14) How it will have impact to installation procces? Does user have to choose kernel manually. Or it will be done automatically with help of some criteria? #7: Dean Hilkewich (deanjo13) (2009-01-23 02:26:22) (reply to #4) It doesn't have to impact the installation process at all. What is required to do this is just another .config kernel config file added to the kernel src RPM and when that is built you would have a desktop kernel. The process is really no different then the current kernel choices we have now (rt, pae, default, xen etc). It doesn't even have to really be hosted on the installation DVD/CD medium at all but could be simply one of those extra packages just found on the full OSS repository. No separate installation images are needed but could be made if product creator was massaged a bit more into a end user friendly app or once susestudio goes live one could easily spin a custom dvd for their own use. The key is to get these alternative kernel packages on the official OSS repo. #5: Vasiliy Astanin (madcad) (2009-01-22 17:49:21) May be we should separate not only kernel, but entire images? openSUSE- Desktop.iso contains KDE, Gnome and a lot of userspace software, but no any Apaches, tomcat, or so. + it contains kernel-desktop openSUSE- Server.iso contains no heavy DE, no OpenOffice, no any other large desktop packages - only some extremely light WM - like icewm, but with all server features. + it contains kernel-default, -server, or similar Such splitted images will be much less in size, so maybe server will even fit on CD. This way we'll get something similar to SLES/SLED, but it may be a good tactics ;) #6: Todd R (theblackcat) (2009-01-22 22:19:33) Perhaps when someone is installing their system they can have a selection of a few different pre-configured systems, such as Desktop/Laptop and Server, and then an "Other" section that includes Netbook, Text-only Desktop/Laptop, Text-only Server, Lightweight Desktop/Laptop, Lightweight Server, Real-time System, Workstation, and perhaps a few more advanced configurations that only advanced users would want. Alternatively, you could ask when someone is first installing the system "Are you planning to use this computer primarily as a:" and they can pick either "Desktop/Laptop" or Server. For established setups, having kernel-desktop and kernel-server versions available in the software management could take care of switching, but perhaps a generic Kernel Setup YaST module that allows you to pick what sorts of things you want to serve as well as allowing you to switch to the server kernel (or back) would help. #8: Stephan Kleine (bitshuffler) (2009-02-23 21:05:37) Apparently doing stuff like enabling PREEMPT and setting HZ to 1000 would not only result in a snappier system but also e.g. solve the glitch problems with pulse audio: https://tango.0pointer.de/pipermail/pulseaudio-discuss/2009-February/003150.... (considering the amount of bitching about PA that alone should be a good enough reason to do it ;D) Even the kernel devs themself say, if I'm not completely mistaken, that the one kernel for all approach isn't a good idea and that it makes sense to have different flavors for -desktop & -server with their settings tuned for the usecase. #9: Stephan Kulow (coolo) (2009-02-24 11:05:04) (reply to #8) Can't anyone of you check if they can get real data for the openSUSE kernel config if changed like that? #11: Karsten König (remur) (2009-02-24 17:15:40) (reply to #9) maybe the kernel team could spin a -desktop kernel build for testing and release it in kotd? This would make testing much easier then building a custom one on our own systems and people could do test runs with comparing them to each other. #12: Stephan Kulow (coolo) (2009-02-25 11:54:31) (reply to #11) well, I want to see numbers that are improved. Because we can simply rename -default to -highperformance and people find it faster. It's always the same story. #14: Jeff Mahoney (jeff_mahoney) (2009-02-25 10:42:10) (reply to #12) The thing is that this is an area where the "numbers" are subjective. We currently make a trade-off between throughput and latency, where throughput is preferred on the server environment. I've heard requests from both sides of the fence - the server users don't want the overhead of the preemption and the desktop users don't want to sacrifice a responsive interactive experience to the server users. This is something I thought about during a recent on-site. While I won't commit to actually shipping separate kernels in the next release, it's worth researching. I can tweak the configs in -master and see what shakes out. For testing, I'm inclined to split kernel-default (kernel-pae on i386) into kernel-desktop and kernel-server. On i386, perhaps rename kernel- default to kernel-legacy? #15: Jeff Mahoney (jeff_mahoney) (2009-02-25 10:46:09) (reply to #14) There are places where kernel-default is the expected name, though, aren't there? Perhaps kernel-default = kernel-server, and kernel- desktop, then. #16: Stephan Kulow (coolo) (2009-02-26 10:01:43) (reply to #15) Let's get Jiri in here, he should know the yast part. Others part that require "kernel-default" should be pretty trivial to fix (the live cd configs come to mind). But renaming our kernels sounds like a good idea, that kernel-pae is installed by default confused many :) Hmm - now that I think about that aspect: if we have a kernel-desktop, then we need to have pae configurable at runtime first, or we'll have - desktop the -pae and what's now -default will become -legacy. Then you get kernel-server, kernel-legacy and kernel-desktop. But the live cds would still need to stick with kernel-legacy. Damn. But possibly we can have the updaters switch kernel flavors on first kernel update. Sounds all solvable :) #17: Jiri Srain (jsrain) (2009-02-26 14:45:48) (reply to #16) YaST has some hardcoded options to select proper kernel, with fallbacks (if the chosen does not exists). These will need to be adjusted, but that's should be pretty trivial. We will need a screen to select the role of the machine (we have similar in SLES - server, virt. host, virt. guest); perhaps we should also use this screen to preselect patterns - but that's just a side effect of these changes). If we have such screen, preselecting the right kernel will be easier. Regarding the PAE vs. non-PAE kernel: We will install PAE kernel only if the PAE flag is present && (machine has >3GB RAM || NX flag is present), which will avoid the performance hit caused by PAE on machines where it has no use. I'd rather avoid hacking online update to replace legacy kernel with PAE, it would IMO be quite tricky; since we have repos enabled on the live media, replacing the kernel during media deployment should be a safer solution. #20: Karsten König (remur) (2009-02-27 11:40:00) (reply to #12) Lennart Poettering added a alsa-time-test to pulseaudio to check latency, would that be the kind of numbers you want to see? It is of course only a single check for a very specific problem. https://tango.0pointer.de/pipermail/pulseaudio-discuss/2009-February/003158.... #21: Takashi Iwai (tiwai) (2009-02-27 11:57:23) (reply to #20) No, it doesn't help in practice. The latency (responsiveness) can't be measured in such a style because the situation that the test program assumes (a tight busy loop) is completely different from the actual situation where PA is running. It's a good tool to test how stable the sound driver (hardware) is, though. #18: Takashi Iwai (tiwai) (2009-02-26 15:05:14) (reply to #8) Lennart's suggestion (HZ=1000 and CONFIG_PREEMPT) is nonsense. It won't help the latency (for non-RT tasks) much. It rather consumes more power and gets slower. His vocal opinion there is based on the very old data or a placebo. In general, the latency (or responsiveness) depends rather on the CPU and IO schedulers. The RT-latency is a different story, and irrelevant now as long as PA daemon is running as a normal task. I find it's not bad to have a desktop kernel, though. The requirement of the current scheduler and parameters are mostly for the server performance (on SLES11), and they don't always fit with the desktop performance. #19: Jeff Mahoney (jeff_mahoney) (2009-02-26 09:19:24) (reply to #18) Agreed. I had no intention of changing HZ=1000. Rather, I was going to start by changing the preemption settings so that the desktop is super aggressive about it, and back off the server configs. #22: Jan Engelhardt (jengelh) (2009-03-01 16:44:23) (reply to #18) However, HZ=100 (yes, 100, the old "server default") and CONFIG_NO_PREEMPT=y does cause mplayer (non-RT app) to skip sometimes during heavy load, such as `git status` in a Linux tree. So that extreme of the two ends is also not too good ;-) #34: Nick Piggin (npiggin) (2009-07-20 18:22:41) (reply to #18) For something like a game that is running flat-out to get max frame rates, I would not have expected HZ=1000 to help either. However I have heard of some people saying for example video decoding does not like HZ=250 because it is not a good multiple of frame rates so it can cause some dropping. This is why HZ=300 was added. A finer-grained timer and one close to ms like 1000 I don't think is such a bad thing (provided we have tickless idle). CONFIG_PREEMPT -- we actually did some numbers when turning off voluntary preempt (basically just adds helps more cond_resched points thoughtout the kernel but costs performance), and VP did have noticably better latencies then PREEMPT_NONE. I would expect full preempt to be even slightly better (because VP can still miss some preempt points -- such as large memory copies). I do agree that CPU and IO schedulers and VM and filesystems are going to have a much bigger impact most of the time. But also maybe in some cases those latencies will be more accepted by the user -- like when loading a new application or trying to run stuff with compile job running in the background -- people might expect and accept some reduced interactivity in those cases. #10: andrea florio (anubisg1) (2009-02-24 11:26:47) i agree with it, but please consider to keep PAE function enabled on desktop kernel... most pc have 4GB ram, and if i install a 32bit system i would like to use all my ram. #13: Theodoros Nikolakopoulos (theodoros) (2009-02-25 16:24:49) I agree, as far as there isn't any significant sacrifice on security features from each. #23: Dean Hilkewich (deanjo13) (2009-03-01 17:07:18) I think it's important to keep in mind that a vast majority of openSUSE users are desktop users. Even though openSUSE is used for the building blocks for SLE where server usage is at a higher percentage, it's your non-corporate desktop user that are the ones primarily using openSUSE. Having separate desktop and server kernels allow you to cater to the larger desktop community as well have a solution for the much smaller server community. Lets face it when you see a review of openSUSE it's almost always focused on the desktop usage and not on the server side. That's where openSUSE's focus of improvement should be geared at. #24: William Simon Lewis (williamsimonlewis) (2009-03-08 13:04:51) Please correct me if I wrong but one advantage of having separate installations for desktops and servers is that desktop users could have noatime and nodiratime options set by default. According to Kerneltrap this would bring a 10% speed increase in the journaling file systems on desktops... #25: Dean Hilkewich (deanjo13) (2009-03-20 18:22:37) William, this certainly could be done as well with a desktop profile. (as well as disabling barriers as other desktop distributions do) but kernel modification isn't needed to do that. #26: De Ganseman Amaury (amaurydg) (2009-04-01 08:55:44) I think it will be good if someone can do some "real life" benchmark to sse if enable preempt or HZ=1000 or ..... can be a benefit for a desktop. Today the more annoying bug that cause VERY BIG latency is http://bugzilla.kernel.org/show_bug.cgi?id=12309 and I don't see any patches or fix on the lkml at the moment. This bug exist since 2.6.18 so...a lot of time ! #27: Dean Hilkewich (deanjo13) (2009-04-23 06:40:07) (reply to #26) Go for it. Perfect tool to test them out has already been created. http://www.phoronix-test-suite.com/ #28: Grozdan Nikolov (microchip8) (2009-05-20 14:07:47) (reply to #26) also, you may consider using the interactivity benchmark 'interbench' by Con Kolivas. Best is to, on the same setup, benchmark SUSE default kernel agains a self-compiled kernel with full preempt, 1000 Hz and -O2 compilation. Use the exact same kernel versions if possible #29: andrea florio (anubisg1) (2009-06-06 10:29:08) any news? i really think that one to be an important features if we really want to "defeate" desktop distros like *buntu #30: Piotrek Juzwiak (benderbendingrodriguez) (2009-06-18 23:20:04) It's not about defating a distro but making the desktop feel snappy. I agree that there is need to separate the kernel for desktop and server. #31: Jeff Mahoney (jeff_mahoney) (2009-07-17 21:23:59) I've added a desktop flavor on i386 and x86_64 for master and the v2. 6.31 branch. It includes full preemption, HZ=1000, and disabled optimize for size, cgroups, and the group scheduler. #32: Jan Engelhardt (jengelh) (2009-07-20 17:25:34) (reply to #31) Unless you have specific numbers that -O2 is faster than -Os, I am not going to buy it. #33: Nick Piggin (npiggin) (2009-07-20 18:05:59) (reply to #32) I'm actually of the opinion that we should disable optimize for size in our server kernel as well. I will try to recall the particular sles bug report I have with some numbers, but we have an ISV customer doing some virtual memory intensive workloads (basically mmap/page fault/munmap) and they found their real world performance is improved very significantly by using -O2 in SLES11. I can't remember exactly, but it is several 10s of % IIRC. The reasoning for -Os in the kernel has seemed a bit flawed to me (as I have written other times before). icache issues are almost no different in userspace applications or libraries. There will always be various combinations of uncommon, common, large, small code being run -- the gcc guys are presumably always trying to make good tradeoffs based on that, and "performance" for them is including icache misses. Specifying -Os would seem to tell gcc that we care more about just binary size rather than actual performance. If the kernel has commonly used code, we absolutely want it to be optimized as highly as possible. Uncommonly used code sure would be nice to reduce in size, but if it is uncommonly executed then by definition it should have smaller (temporal) icache footprint. Now I don't have any numbers or reason to believe -O2 should lead in the desktop flavour -- unless like a staging step to enable it in the server flavour. I don't know of desktop workload where the kernel is going to be very costly, but actually I don't really profile 3d rendering which is one thing that might benefit from -O2. If anyone is gathering these kinds of framerate numbers, then it would be very interesting to test the difference between -Os and -O2. #35: Nick Piggin (npiggin) (2009-07-21 12:06:27) (reply to #33) OK it is SLES bug 482887 (https://bugzilla.novell.com/show_bug.cgi?id=482887) . ISV reports VM intensive microbenchmark slows down by about 45%, and real world (for them) performance by 10-20% by using -Os rather than -O2 in SLES11 kernel. #36: Nick Piggin (npiggin) (2009-07-22 08:36:16) (reply to #35) We have another result from a hardware vendor showing an important database workload is actually improving by 1% (system-wide throughput) by compiling kernel with -O2 rather than -Os. Their result is a little sparse on details, and I can't share some details to public, but I think it is a meaningful result. #37: Takashi Iwai (tiwai) (2009-07-22 16:15:12) (reply to #36) IIRC, the decision for -Os was due to a significant performance difference on PowerPC a few years ago. There was little difference between -Os and -O2 on x86, thus we chose -Os. But, your current number is much more convincing. We should go for -O2 indeed. #38: Piotrek Juzwiak (benderbendingrodriguez) (2009-07-30 23:57:03) I can see that it has been done and such kernels have already been created, now it would be a great idea to make the desktop kernel the default kernel for live CDs and also for DVDs. The "server" kernel will still be included on the DVD right? #39: Jiri Srain (jsrain) (2009-07-31 07:19:35) (reply to #38) The code to make desktop kernel default is already in, I just could not test it (since the latest i586 image I had did not contain the desktop kernel). #40: Stephan Kulow (coolo) (2009-07-31 10:08:02) (reply to #38) the i586 live cd can't use it as default as it will only work on pae hardware - but the DVD has both flavors on both architecture (but no longer a kernel-pae - that's in the online repo only). The x86_64 live cd should be able to use the kernel-desktop as default. #41: Jiri Srain (jsrain) (2009-07-31 10:39:52) (reply to #40) That's why I wanted to know more about the kernels configurations. Installer pre-selects the desktop kernel as default if otherwise PAE kernel would be used. If PAE flag is not present or (NX flag is not present and system has <3GB RAM), there is no change from 11.1. + #42: Jakub Rusinek (liviopl) (2009-08-10 22:18:09) + I don't really know anything about kernel performance increasing and I + don't even want to know, but making separate kernel config for desktop + seems pretty resonable and wouldn't really hurt installation as LiveCD + is meant to be only for desktop users, while DVD can have both kernels + included. + I'll be glad if -desktop kernel will really bring more perceived + performance and PulseAudio stability as it's important and -desktop + kernel could make it possible to achieve it. + Thanks for making such a great move! -- openSUSE Feature: https://features.opensuse.org/305694