[opensuse-packaging] LLVM package changes

Hi, Some time ago I had to make some changes to the LLVM package. It caused some issues and took some time to stabilize, but I thought it is done and working correctly now. Unfortunatelly, it turned out it is still broken and more changes are needed. Before I start breaking things again, I would like to describe here what happened and what I think still needs to happen. The LLVM package builds several related projects: llvm, clang, libcxx, lldb, lld, and other small ones. These projects can be either placed into a single source tree and built all together, or they can be built individually. From packaging point of view, it would be cleaner if we built them separately, but since we need to bootstrap clang and then use it to build itself, we have to build at least llvm, clang and libcxx together. So it is easier to just build them all together. Each of the projects is divided into components (such as LLVM/Analysis, LLVM/ AsmParser, clang/AST, clang/CodeGen, ...). LLVM can be built in different modes: 1) Every component will compile into a shared library. 2) Every component will compile into a static library. 3) Every project will compile into a single shared library containing all its components. Before I started working on the package, we were building LLVM using mode 1. This caused us some troubles, all that I am aware of are listed in this bug: https://bugzilla.opensuse.org/show_bug.cgi?id=1049703 When trying to fix these issues with upstream, we were told that this build mode is only meant for the llvm developers and the issues would not happen if we used one of the recommended modes (2 or 3). https://github.com/klee/klee/pull/686
Maybe openSUSE have a good reason for doing this but going against a project's recommended practice seems like a bad idea.
You can file a bug on LLVM.org to investigate fixing that, however it is unlikely that much attention will be given to the BUILD_SHARED_LIBS build because it is strongly discouraged.
The same information is also stated in the LLVM build documentation. So I went ahead and changed LLVM package to the mode 3 (single shared library for every project). Unfortunatelly the clang project does not support this mode. So the clang components are instead built as static libraries and distributed in the new clang-devel-static package. This is the current state of the package. One drawback is that the llvm package now takes longer to build and consumes more resources in OBS. (Linking stuff together into big libraries takes longer and the static libraries cause increase file size. Biggest slow down is in extracting debuginfo and compressing the RPM files.) That caused the LLVM package to be a bottleneck in the distribution build and needed to be solved (bnc#1064786). I spent lot of time trying to optimize the build and also had to split Mesa to a llvm and non-llvm part (bnc#1071297) to reduce the amount of packages that transitively depend on llvm. The situation is now better, although still not that good. Recently a new bug got to me: https://bugzilla.opensuse.org/show_bug.cgi?id=1065464 That bug shows that OpenCL is broken if you have multiple OpenCL implementations which use clang installed at once. Note that it is perfectly ok to have them installed at once. Applications are supposed to load them all, query them and pick one to use. However, when a program loads two libraries statically linked with the clang libraries, it aborts. The reason is that each of them try to register a configuration option into the llvm library - but there is only one shared llvm library used by both of them. It aborts during the second attempt to register the same configuration option. There is a request upstream to change the way llvm handles configuration options, but it is a big task and not likely to be done any time soon. So it can only work if both clang and llvm are static or both shared libraries. I am against making them all static. In addition to all the good reasons why we try to stay away from static libraries, it would likely make the llvm build times even worse. Instead, I would like to try approach similar to what Fedora does - they build all of llvm using the mode 3 (each project as single shared library), except for clang which they build with mode 1 ("developer" mode where every component ends up as a shared library). Fedora has easier time doing that because they have them in separate packages, but we can do it too using a small patch to the llvm build system (tested). This should: * Fix the OpenCL bug. * Get rid of static libraries and the clang-devel-static package. * Reduce llvm build time. * Maybe reintroduce some bugs since we will be partially using the unsupported "developer" mode again. But it would be only for clang, llvm and all the other components would still use the supported mode. AFAIK all the bugs I know of were caused by the llvm libraries. I will test for regressions before submitting anything. Comments and suggestions are welcome. Michal Srb -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Hi, I like the idea, it seems like the right way to go. If it works for Fedora, it's probably good enough. I didn't encounter any of the mentioned bugs, but there are a few things that I don't like about the current setup: * The package clang-devel contains headers for both C API (in /usr/include/clang-c) and C++ API (in /usr/include/clang), but libclang.so from the same package only exposes symbols for the former. * Using the C++ API, I have to install clang-devel-static, which is a gigantic package, and linking against it is quite slow. * Unlike libclang.so, libLLVM.so exposes both C and C++ API, which seems inconsistent. So using both with the C++ API I link statically with one and dynamically with another? So technically Clang supports the single dynamic library (3), but that library only exposes the C API. Which makes me think that the advice against using BUILD_SHARED_LIBS only applies to LLVM and not Clang, since otherwise there would be no shared library build exposing Clang's C++ API. That advice is also from the LLVM documentation, and I couldn't find anything similar for Clang. If all else fails, and we really have to go the static library route, we should at least have another version of clang-devel-static without debug info. (Which should then be around 40-50 MB instead of 4 GB.) Best regards, Aaron Am 19.01.2018 um 10:17 schrieb Michal Srb:
Hi,
Some time ago I had to make some changes to the LLVM package. It caused some issues and took some time to stabilize, but I thought it is done and working correctly now. Unfortunatelly, it turned out it is still broken and more changes are needed. Before I start breaking things again, I would like to describe here what happened and what I think still needs to happen.
The LLVM package builds several related projects: llvm, clang, libcxx, lldb, lld, and other small ones. These projects can be either placed into a single source tree and built all together, or they can be built individually. From packaging point of view, it would be cleaner if we built them separately, but since we need to bootstrap clang and then use it to build itself, we have to build at least llvm, clang and libcxx together. So it is easier to just build them all together.
Each of the projects is divided into components (such as LLVM/Analysis, LLVM/ AsmParser, clang/AST, clang/CodeGen, ...).
LLVM can be built in different modes:
1) Every component will compile into a shared library. 2) Every component will compile into a static library. 3) Every project will compile into a single shared library containing all its components.
Before I started working on the package, we were building LLVM using mode 1. This caused us some troubles, all that I am aware of are listed in this bug: https://bugzilla.opensuse.org/show_bug.cgi?id=1049703
When trying to fix these issues with upstream, we were told that this build mode is only meant for the llvm developers and the issues would not happen if we used one of the recommended modes (2 or 3).
https://github.com/klee/klee/pull/686
Maybe openSUSE have a good reason for doing this but going against a project's recommended practice seems like a bad idea.
You can file a bug on LLVM.org to investigate fixing that, however it is unlikely that much attention will be given to the BUILD_SHARED_LIBS build because it is strongly discouraged.
The same information is also stated in the LLVM build documentation. So I went ahead and changed LLVM package to the mode 3 (single shared library for every project). Unfortunatelly the clang project does not support this mode. So the clang components are instead built as static libraries and distributed in the new clang-devel-static package.
This is the current state of the package. One drawback is that the llvm package now takes longer to build and consumes more resources in OBS. (Linking stuff together into big libraries takes longer and the static libraries cause increase file size. Biggest slow down is in extracting debuginfo and compressing the RPM files.) That caused the LLVM package to be a bottleneck in the distribution build and needed to be solved (bnc#1064786). I spent lot of time trying to optimize the build and also had to split Mesa to a llvm and non-llvm part (bnc#1071297) to reduce the amount of packages that transitively depend on llvm. The situation is now better, although still not that good.
Recently a new bug got to me: https://bugzilla.opensuse.org/show_bug.cgi?id=1065464
That bug shows that OpenCL is broken if you have multiple OpenCL implementations which use clang installed at once. Note that it is perfectly ok to have them installed at once. Applications are supposed to load them all, query them and pick one to use. However, when a program loads two libraries statically linked with the clang libraries, it aborts. The reason is that each of them try to register a configuration option into the llvm library - but there is only one shared llvm library used by both of them. It aborts during the second attempt to register the same configuration option. There is a request upstream to change the way llvm handles configuration options, but it is a big task and not likely to be done any time soon.
So it can only work if both clang and llvm are static or both shared libraries. I am against making them all static. In addition to all the good reasons why we try to stay away from static libraries, it would likely make the llvm build times even worse.
Instead, I would like to try approach similar to what Fedora does - they build all of llvm using the mode 3 (each project as single shared library), except for clang which they build with mode 1 ("developer" mode where every component ends up as a shared library). Fedora has easier time doing that because they have them in separate packages, but we can do it too using a small patch to the llvm build system (tested).
This should: * Fix the OpenCL bug. * Get rid of static libraries and the clang-devel-static package. * Reduce llvm build time. * Maybe reintroduce some bugs since we will be partially using the unsupported "developer" mode again. But it would be only for clang, llvm and all the other components would still use the supported mode. AFAIK all the bugs I know of were caused by the llvm libraries. I will test for regressions before submitting anything.
Comments and suggestions are welcome.
Michal Srb
-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On středa 24. ledna 2018 1:47:47 CET Aaron Puchert wrote:
Hi,
I like the idea, it seems like the right way to go. If it works for Fedora, it's probably good enough.
Hi, Thank you for the feedback!
* The package clang-devel contains headers for both C API (in /usr/include/clang-c) and C++ API (in /usr/include/clang), but libclang.so from the same package only exposes symbols for the former. * Using the C++ API, I have to install clang-devel-static, which is a gigantic package, and linking against it is quite slow.
Luckily this problem will go away with the new changes because we'll get rid of the clang-devel-static package and have again only clang-devel.
* Unlike libclang.so, libLLVM.so exposes both C and C++ API, which seems inconsistent. So using both with the C++ API I link statically with one and dynamically with another?
Yeah, in the current setup that is the way to do it. And it can cause bugs like the bnc#1065464 if you get mixed with another library that also linked statically to clang.
So technically Clang supports the single dynamic library (3), but that library only exposes the C API.
As I understand it, the libclang.so is different than libLLVM.so. As you observed, it exposes only the stable C API and so it is linked (statically or dynamically) only to *some* of the clang components. So the remaining components still have to be distributed. If you build LLVM as a single dynamic library, the libLLVM.so contains and exposes all the available LLVM components. I do not know of a way to build libclang.so in such way.
Which makes me think that the advice against using BUILD_SHARED_LIBS only applies to LLVM and not Clang, since otherwise there would be no shared library build exposing Clang's C++ API. That advice is also from the LLVM documentation, and I couldn't find anything similar for Clang.
Hopefully that is true. Michal -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
participants (2)
-
Aaron Puchert
-
Michal Srb