SUPER optimization flags
Hi, I began building packages for SUPER with optimizations: CFLAGS="-O3 -g -march=i686 -mtune=i686 -fomit-frame-pointer -pipe -ftracer -ffast-math -fforce-addr -falign-functions=64 -momit-leaf-frame-pointer" CXXFLAGS="-O3 -g -march=i686 -mtune=i686 -fomit-frame-pointer -pipe -ftracer -ffast-math -fforce-addr -falign-functions=64 -momit-leaf-frame-pointer -fvisibility-inlines-hidden" LDFLAGS="-Wl,-O1 -Wl,--enable-new-dtags -Wl,--sort-common -s" Until now everything works OK, at least the compiling process (I didn't have time to test them). Those flags are recommended by various Gentoo users, and I also used them about 1-1.5 years ago on Gentoo. Anyone on the list having any idea of better flags? What is your opinion about optimized packages? Should we provide optimized packages or we should continue on the same way Andreas Girardet began and provide just the ck kernel and other patches? Yours faithfully, -- Damian Mihai Liviu Mobile: +40 741 226993; Fax: +1 347-632-4117 Phone : +1 360-526-6441; +1 347-632-4117; +44 0870-3403339 URL: http://liviudm.blogspot.com
If there is a quantifiable difference in performance then it makes sense, if not I think it just increases the likelihood of something being miscompiled and possibly broken. Ian Damian Mihai Liviu wrote:
Hi,
I began building packages for SUPER with optimizations: CFLAGS="-O3 -g -march=i686 -mtune=i686 -fomit-frame-pointer -pipe -ftracer -ffast-math -fforce-addr -falign-functions=64 -momit-leaf-frame-pointer" CXXFLAGS="-O3 -g -march=i686 -mtune=i686 -fomit-frame-pointer -pipe -ftracer -ffast-math -fforce-addr -falign-functions=64 -momit-leaf-frame-pointer -fvisibility-inlines-hidden" LDFLAGS="-Wl,-O1 -Wl,--enable-new-dtags -Wl,--sort-common -s" Until now everything works OK, at least the compiling process (I didn't have time to test them). Those flags are recommended by various Gentoo users, and I also used them about 1-1.5 years ago on Gentoo. Anyone on the list having any idea of better flags? What is your opinion about optimized packages? Should we provide optimized packages or we should continue on the same way Andreas Girardet began and provide just the ck kernel and other patches?
Yours faithfully,
Good morning, On Monday 07 November 2005 18:54, Ian S. Nelson wrote:
If there is a quantifiable difference in performance then it makes sense, if not I think it just increases the likelihood of something being miscompiled and possibly broken.
Sometimes it's two times faster, sometimes the difference is almost unnoticeable, but if you consider the hole system, from my experience there is a good difference. Andreas (Girardet) what is your opinion? You are the project lead, so I'm waiting for yours :-) Cheers, -- Damian Mihai Liviu Mobile: +40 741 226993; Fax: +1 347-632-4117 Phone : +1 360-526-6441; +1 347-632-4117; +44 0870-3403339 URL: http://liviudm.blogspot.com
On Tuesday 08 November 2005 12:17 am, Damian Mihai Liviu wrote:
Good morning,
On Monday 07 November 2005 18:54, Ian S. Nelson wrote:
If there is a quantifiable difference in performance then it makes sense, if not I think it just increases the likelihood of something being miscompiled and possibly broken.
Sometimes it's two times faster, sometimes the difference is almost unnoticeable, but if you consider the hole system, from my experience there is a good difference. Andreas (Girardet) what is your opinion? You are the project lead, so I'm waiting for yours :-)
Cheers,
I think there may be some problems with the use of another compiler. First of all, what compiler does say the KDE group use? If a problem occurs when using the Intel compiler but doesn't show up with GCC, and KDE uses the GCC compiler, then who is responsible for correcting it? I'm sure that KDE would say "Take care of it yourself". Secondly, are there any conflicts in the combining of routines that may have been created using different compilers? That is, a huge system such as KDE has many subroutines. Subroutines compiled with one compiler may not play nice with subroutines compiled with another compiler. These bugs are not simple to find, and may occur in very special and unusual circumstances. I would expect that if you go with the Intel compiler, then all routines in the distribution would be compiled with the Intel compiler. That said, do you create 2 distributions, one Intel, and the other GCC? Thirdly, for the user that wants to use a program available in source only, do they have to install the compiler based upon the distribution? Would you start to include the Intel compiler as part of the repositories? Has anyone developed a list of good programming techniques to enhance the speed of program? Perhaps a route to go would be a project to review individual programs to see if they have any coding that could be improved for speed based upon those techniques. The list could be set to include only high resource usage programs such as image viewers and editors. There are a lot of lions and tigers lurking out in the bushes, so I feel that much caution should use in embarking on such a journey. Just my thoughts... Tom.
On Tue, Nov 08, 2005 at 10:33:26AM -0500, Tom Cada wrote:
I think there may be some problems with the use of another compiler.
Sure! ;-)
First of all, what compiler does say the KDE group use? If a problem occurs when using the Intel compiler but doesn't show up with GCC, and KDE uses the GCC compiler, then who is responsible for correcting it? I'm sure that KDE would say "Take care of it yourself".
Building a distribution yourself without the knowledge how to fix bugs is stupid anyway in my opinion.
Secondly, are there any conflicts in the combining of routines that may have been created using different compilers? That is, a huge system such as KDE has many subroutines. Subroutines compiled with one compiler may not play nice with subroutines compiled with another compiler. These bugs are not simple to find, and may occur in very special and unusual circumstances.
The Intel compiler is ABI compatible to gcc. Otherwise he couldn't use libstdc++ from gcc anyway. Sure there is still the risk that something breaks due to this problem but then this is a bug in the compiler and Intel would sure be happy about a decent bug report.
Thirdly, for the user that wants to use a program available in source only, do they have to install the compiler based upon the distribution? Would you start to include the Intel compiler as part of the repositories?
That would be illegal. But you can freely mix both compilers.
Has anyone developed a list of good programming techniques to enhance the speed of program? Perhaps a route to go would be a project to review individual programs to see if they have any coding that could be improved for speed based upon those techniques. The list could be set to include only high resource usage programs such as image viewers and editors.
Cleaning up and tuning the code of applications is definitely the more promissing way than just blindly replacing the compiler. Robert -- Robert Schiele Tel.: +49-621-181-2214 Dipl.-Wirtsch.informatiker mailto:rschiele@uni-mannheim.de
Hi, On Monday 07 November 2005 18:54, Ian S. Nelson wrote:
If there is a quantifiable difference in performance then it makes
At the first look amarok is works much better. In the SUSE's release if I was closing it to tray and the open it again (just click on the tray icon) too many times/second amarok was getting stuck, even with small playlists (less than 15 songs). Tried this with my optimized RPM version and there is no problem, even with large playlist (9h+ playtime). Feel free to install it from suser-liviudm and test it. Cheers, -- Damian Mihai Liviu Mobile: +40 741 226993; Fax: +1 347-632-4117 Phone : +1 360-526-6441; +1 347-632-4117; +44 0870-3403339 URL: http://liviudm.blogspot.com
Package works nicely for me. However, to install it I had to
uninstall amarok and comment out all my apt repositories except for
yours, since apt by default will go for the newer version.
Is there a way for me to give optimized / SUPER RPM's priority over
non-optimized newer versions of the package?
On 11/7/05, Damian Mihai Liviu
Hi,
On Monday 07 November 2005 18:54, Ian S. Nelson wrote:
If there is a quantifiable difference in performance then it makes
At the first look amarok is works much better. In the SUSE's release if I was closing it to tray and the open it again (just click on the tray icon) too many times/second amarok was getting stuck, even with small playlists (less than 15 songs). Tried this with my optimized RPM version and there is no problem, even with large playlist (9h+ playtime). Feel free to install it from suser-liviudm and test it.
Cheers, -- Damian Mihai Liviu Mobile: +40 741 226993; Fax: +1 347-632-4117 Phone : +1 360-526-6441; +1 347-632-4117; +44 0870-3403339 URL: http://liviudm.blogspot.com
Hello, On Tuesday 08 November 2005 08:45, Illidan Wabos wrote:
Package works nicely for me. However, to install it I had to uninstall amarok and comment out all my apt repositories except for yours, since apt by default will go for the newer version.
Is there a way for me to give optimized / SUPER RPM's priority over non-optimized newer versions of the package?
I'm glad to hear that. You don't need to uninstall amarok, just do upgrade. The easiest way is to use smart, add your repositories, go to View -> Tree Style and select Channels. Now go to my repository and select the packages you want to install. Smart will tell you that he will upgrade your package. Also in smart you can give priority to the channels you want. More optimized KDE packages will be available around 10:00AM GMT (kdebase3-*, kdenetwork3-*, kdesvn and I hope kdemultimedia3-*) Cheers, -- Damian Mihai Liviu Mobile: +40 741 226993; Fax: +1 347-632-4117 Phone : +1 360-526-6441; +1 347-632-4117; +44 0870-3403339 URL: http://liviudm.blogspot.com
You wrote:
At the first look amarok is works much better. In the SUSE's release if I was closing it to tray and the open it again (just click on the tray icon) too many times/second amarok was getting stuck, even with small playlists (less than 15 songs). Tried this with my optimized RPM version and there is no problem, even with large playlist (9h+ playtime).
I think you are highlighting an interesting problem: benchmarking. When you want to convince somebody (users,Suse) that the changes are beneficial, you should be able to quantify exactly how much faster it is. On my system amarok does not get stuck, no matter how often I click. Another option we might want to explore is using icc (the Intel C++ compiler). There is a free version for college/universtiy students that allows you to compile software with "non-commercial licenses only". I think that applies to this project. The German Linux Magazin recently had a comparison of gcc 4.0 against icc 9.0, where the icc was around 10% faster for bzip2. The effect was smaller or even negative for other software, but at least for _some_ applications you can get considerable speedups using icc. Cheers nordi
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 nordi@addcom.de wrote: ...
Another option we might want to explore is using icc (the Intel C++ compiler). There is a free version for college/universtiy students that allows you to compile software with "non-commercial licenses only". I think that applies to this project. The German Linux Magazin recently had a comparison of gcc 4.0 against icc 9.0, where the icc was around 10% faster for bzip2. The effect was smaller or even negative for other software, but at least for _some_ applications you can get considerable speedups using icc.
Using icc will result in having to install icc runtime libraries, at least for C++.
(see libstdc++ and libgcc for g++)
What licenses apply to the runtime libraries ?
On what architectures are they available ?
How well is RPM currently handling building with icc ? (flags, archs, ...)
Needs quite some investigation upfront, IMO.
cheers
- --
-o) Pascal Bleser http://linux01.gwdg.de/~pbleser/
/\\
On Tuesday 08 November 2005 11:59, Pascal Bleser wrote:
Using icc will result in having to install icc runtime libraries, at least for C++. (see libstdc++ and libgcc for g++) That's not very convenient....
Needs quite some investigation upfront, IMO. I'm not so interested learning about icc but if someone will test it and we all agree that we should use icc in SUPER instead of gcc, I will begin building the packages with icc.
Cheers, -- Damian Mihai Liviu Mobile: +40 741 226993; Fax: +1 347-632-4117 Phone : +1 360-526-6441; +1 347-632-4117; +44 0870-3403339 URL: http://liviudm.blogspot.com
On Tue, Nov 08, 2005 at 10:59:22AM +0100, Pascal Bleser wrote:
Using icc will result in having to install icc runtime libraries, at least for C++. (see libstdc++ and libgcc for g++)
Wrong. On a Linux system icc uses by default the gcc provided libstdc++ if installed and the other support libraries can be linked in statically by the -i-static switch.
What licenses apply to the runtime libraries ?
"C. Subject to all of the terms and conditions of this Agreement, Intel grants to you a non-exclusive, non-assignable copyright license to distribute (except under an Evaluation License as specified below) the Redistributables, or any portions thereof, as part of the product or application you developed using the Materials." Note that all libraries that icc links to by default are "Redistributables".
On what architectures are they available ?
i586, x86_64, and ia64.
How well is RPM currently handling building with icc ? (flags, archs, ...)
Huh? RPM does not really care about the compiler you use.
Needs quite some investigation upfront, IMO.
Technically there is no real problem using this compiler. But you should make sure yourself whether you comply to this: "ii. NONCOMMERCIAL-USE LICENSE: If you are using the Materials under the control of a Noncommercial-Use license, you as an individual may use the Materials only for non-business use where you receive no fee, salary or any other forms of compensation. The Materials may not be used for any other purpose, whether "for profit" or "not for profit." Any work performed or produced as a result of use of the Materials cannot be performed or produced for the benefit of other parties for a fee, compensation or any other reimbursement or remuneration." Robert -- Robert Schiele Tel.: +49-621-181-2214 Dipl.-Wirtsch.informatiker mailto:rschiele@uni-mannheim.de
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Robert Schiele wrote:
On Tue, Nov 08, 2005 at 10:59:22AM +0100, Pascal Bleser wrote:
Using icc will result in having to install icc runtime libraries, at least for C++. (see libstdc++ and libgcc for g++)
Wrong. On a Linux system icc uses by default the gcc provided libstdc++ if installed and the other support libraries can be linked in statically by the -i-static switch.
Ok, nice. So -i-static *must* be passed. Correct ? If not, are there packages of the "other" (non-libstdc++) icc runtime libraries ? (that would have to be shipped as part of SUPER)
What licenses apply to the runtime libraries ? "C. Subject to all of the terms and conditions of this Agreement, Intel grants to you a non-exclusive, non-assignable copyright license to distribute (except under an Evaluation License as specified below) the Redistributables, or any portions thereof, as part of the product or application you developed using the Materials." Note that all libraries that icc links to by default are "Redistributables".
Ok, that's covered as well.
On what architectures are they available ? i586, x86_64, and ia64.
Ok.
How well is RPM currently handling building with icc ? (flags, archs, ...) Huh? RPM does not really care about the compiler you use.
/usr/lib/rpm/macros:%__cxx g++ My point is: if people plan to build RPMs with icc, there must be a common policy on how to tell RPM to use icc instead of g++ IMO it must be external to the spec file (i.e. not write "%define __cxx icc" in the spec file), which may sound obvious, but when I saw Damian writing "BuildArch: i686" there doesn't necessarely seems to be a correct understanding of how RPM is supposed to be used. The spec files should fall back nicely to g++. Also, the %optflags defined for rpmbuild (in /usr/lib/rpm/rpmrc and /usr/lib/rpm/*-linux/macros) are tailored for GCC, not for icc, e.g.: optflags: i686 -O2 -g -march=i686 -mtune=i686 -fmessage-length=0 -D_FORTIFY_SOURCE=2 - -O and -g should be fine with icc (I assume), but -fmessage-length and -mtune most definately not (unless icc provides a g++ front). Would most probably mean adding the following line in ~/.rpmmacros: %define __cc icc %define __cxx icc And adding corresponding options in ~/.rpmrc (see /usr/lib/rpm/rpmrc) And patching plenty of Makefiles that explicity use "gcc" or "g++", instead of using a "CC" and "CXX" make variables you would have to pass as well: %__make CC="%__cc" CXX="%__cxx" CFLAGS="%{optflags}" CXXFLAGS="%{optflags}" For projects that use autoconf, they most probably will use "gcc"/"g++" by default, so you must pass some stuff to ./configure - probably like this: CC="%__cc" CXX="%__cxx" CFLAGS="%{optflags}" CXXFLAGS="%{optflags}" \ ./configure \ ... Even if you use the %configure macro in your spec files (which is highly encouraged), it does *NOT* pass CC nor CXX to ./configure (see yourself: /usr/lib/rpm/macros), which means every spec file has to do the following: CC="%__cc" CXX="%__cxx" %configure ... (and do check the rpmbuild output to see whether it actually uses "icc" and not "gcc"/"g++")
Needs quite some investigation upfront, IMO. Technically there is no real problem using this compiler. But you should make
see above ;)
sure yourself whether you comply to this: "ii. NONCOMMERCIAL-USE LICENSE: If you are using the Materials under the control of a Noncommercial-Use license, you as an individual may use the Materials only for non-business use where you receive no fee, salary or any other forms of compensation. The Materials may not be used for any other purpose, whether "for profit" or "not for profit." Any work performed or produced as a result of use of the Materials cannot be performed or produced for the benefit of other parties for a fee, compensation or any other reimbursement or remuneration."
This is fine for people here who would build RPMs with icc.
But if, say, those RPMs are included into SUPER... what are the implications ?
To me, given the license excerpt above, it would "just" mean that you may not charge a fee for
SUPER. I'm not sure whether that conflicts with the GPL.
If not, then it's just that everyone using and participating in SUPER development must be aware of
this. And everyone must agree it's a "non issue".
cheers
- --
-o) Pascal Bleser http://linux01.gwdg.de/~pbleser/
/\\
On Tue, Nov 08, 2005 at 04:03:58PM +0100, Pascal Bleser wrote:
Ok, nice. So -i-static *must* be passed. Correct ?
Either this or you ship the respective shared libraries.
If not, are there packages of the "other" (non-libstdc++) icc runtime libraries ?
I am not aware of them. Someone needed to build them.
How well is RPM currently handling building with icc ? (flags, archs, ...) Huh? RPM does not really care about the compiler you use.
/usr/lib/rpm/macros:%__cxx g++
My point is: if people plan to build RPMs with icc, there must be a common policy on how to tell RPM to use icc instead of g++
Right, obviously you have to tell RPM to do so but there is nothing that hinders you but the fact that it must be done.
Also, the %optflags defined for rpmbuild (in /usr/lib/rpm/rpmrc and /usr/lib/rpm/*-linux/macros) are tailored for GCC, not for icc, e.g.: optflags: i686 -O2 -g -march=i686 -mtune=i686 -fmessage-length=0 -D_FORTIFY_SOURCE=2
- -O and -g should be fine with icc (I assume), but -fmessage-length and -mtune most definately not (unless icc provides a g++ front).
icc supports many of the gcc params (all from the mentioned above but -fmessage-length). It is even fine with most gcc params that it does not support because it just ignores them in that case with a small warning that the option is unknown.
And patching plenty of Makefiles that explicity use "gcc" or "g++", instead of using a "CC" and "CXX" make variables you would have to pass as well:
%__make CC="%__cc" CXX="%__cxx" CFLAGS="%{optflags}" CXXFLAGS="%{optflags}"
For projects that use autoconf, they most probably will use "gcc"/"g++" by default, so you must pass some stuff to ./configure - probably like this:
CC="%__cc" CXX="%__cxx" CFLAGS="%{optflags}" CXXFLAGS="%{optflags}" \ ./configure \ ...
Even if you use the %configure macro in your spec files (which is highly encouraged), it does *NOT* pass CC nor CXX to ./configure (see yourself: /usr/lib/rpm/macros), which means every spec file has to do the following:
CC="%__cc" CXX="%__cxx" %configure ...
(and do check the rpmbuild output to see whether it actually uses "icc" and not "gcc"/"g++")
Sure but these are issues with the packages and not RPM itself. Actually this is a _lot_ of work for a whole distribution and I personally don't see the motivation to do it because I prefer to have a free compiler in a Linux distribution. In my opinion using icc for special projects where you really _need_ some advantages of it makes sense but I would not make a whole distribution depending on it. If Intel revoked your license tomorrow for some reason all your work would have been useless. Robert -- Robert Schiele Tel.: +49-621-181-2214 Dipl.-Wirtsch.informatiker mailto:rschiele@uni-mannheim.de
On Tuesday 08 November 2005 11:28, nordi@addcom.de wrote:
I think you are highlighting an interesting problem: benchmarking. When you want to convince somebody (users,Suse) that the changes are beneficial, you should be able to quantify exactly how much faster it is. On my system
Someone with enough experience in benchmarking can please do some tests and post the results here? P.S.: The gwdg.de repository is synced again. New packages available. -- Damian Mihai Liviu Mobile: +40 741 226993; Fax: +1 347-632-4117 Phone : +1 360-526-6441; +1 347-632-4117; +44 0870-3403339 URL: http://liviudm.blogspot.com
Am Dienstag, 8. November 2005 11:47 schrieb Damian Mihai Liviu:
P.S.: The gwdg.de repository is synced again. New packages available.
Sorry which repository? suser-agirardet? Because I have no new packages (kde*) as you described... Thanks Heart
On Wednesday 09 November 2005 00:03, Heart wrote:
Sorry which repository? suser-agirardet? Because I have no new packages (kde*) as you described... suser-liviudm
-- Damian Mihai Liviu Mobile: +40 741 226993; Fax: +1 347-632-4117 Phone : +1 360-526-6441; +1 347-632-4117; +44 0870-3403339 URL: http://liviudm.blogspot.com
Ian S. Nelson wrote:
If there is a quantifiable difference in performance then it makes sense
I have played a bit with bzip2 and have managed to get compression 10.4% faster: Compression / Decompression of test file in seconds old: 42.94 / 11.54 new: 38.90 / 11.48 Decompression time improved very slightly. What I changed is some compiler options (nothing fancy, just using -Os and -O3 in the right places) and I made bzip2 static. Building it static seems to help caching, since quite a bit of the extra performance comes through this. Sure, that might use some more memory, but it's just a few kilobytes. All tests were run on my Pentium-M. I'd like to know if other people also get higher performance with this one, especially on AMD processors. If at least perfomance does not _de_crease for anyone else, I hope this can go into SLICK. You can get the rpm at [1] (install with --force) and the source at [2]. Note that I have only added a patch for the makefiles and changed the spec-file. Cheers nordi [1] http://private.addcom.de/nordi/super/bzip2-1.0.3-5.i586.rpm [2] http://private.addcom.de/nordi/super/bzip2-1.0.3-SUPER.tar.gz
participants (9)
-
Damian Mihai Liviu -
Heart -
Ian S. Nelson -
Illidan Wabos -
nordi -
nordi＠addcom.de
-
Pascal Bleser -
Robert Schiele -
Tom Cada