[opensuse-buildservice] Build C++ Files for different distributions
Hello, I´m quite new to building packages on Linux, so I have some very basic, probably stupid questions, but after hours of luckless research this is the last way I see to understand this. I have the task to find out how to avoid the need of 3 build servers for supplying three different distributions. All I read is the open build service would be able to build binary packages from sources for a wide range of operating systems and hardware architectures. But I don't understand what those sources are. Would I be able to compile binaries for different distributions from C++ code with this? And then based on those binaries build the packages? If yes, how can I do it? If not, what is the benefit from just building packages if I need to build the binaries on different distributions? Thank you very much, tol -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi Tobias, building packages for different Distributions and Architectures is a task the openSUSE build service (OBS) comes in very handy. Currently it supports all major distributions, enterprise and community, and most major architectures (MIPS being the only missing one currently). That said, although OBS leverages you from several tasks, most importantly recurring ones, there is still some initial setup you have to do for your software. You said you will be using C++. So lets see how to get from your source code to the installable package: 1. You need a build environment. For the beginning, just install a C++ compiler, e.g. GCC. 2. Next, you have to create a "recipe" how to create a binary from the source. 2.1 If you have just a single source file and are just needing the c++ standard library (libstdc++), all you need is calling: $> g++ -o foo foo.cpp The linking to libstdc++ is implicit, and this creates an "foo" executable. Typical "Hello World!" example. 2.2. Things will become more complicated soon, as you have several source files, start linking to third party libraries and so on .. Now you have to start thinking about automting these steps. Typically, this is done using a Makefile, which stores the recipes, which is executed by GNU make. As writing Makefiles by hand becomes error prone very soon, often a Makefile generator is used. Prominent examples are GNU automake and CMake. I prefer CMake, as writing rules for it is (IMHO) much more straightforward than automake. Assuming you now have some source files and an (autogenerated) makefile. Everytime you execute "make", the sources are recompiled (as needed) and the executable is created. Create an archive from your sources. Assuming you have created an directory "foo" for your sources: $> cd .. $> tar caf foo-0.1.tar.xz foo --- Up until now you have executed the building on your own computer, building an executable for the distribution and architecture you are using. Next, you want to replicate the steps done so far (installing the distribution, installing the build environment, executing the build) for some other target. For the beginning, you should chose just one target and adapt as needed when adding more targets. As I am only familiar with RPM based distributions, the next steps are just given for openSUSE, which is RPM based, but this is just a limitation of myself, not the OBS. 3. Start using the OBS. Create an account, add a new Subproject (e.g. "Bar") to your home Project, and create an empty Package ("Foo") in it. You should also add some base repository to your Project, e.g. openSUSE_13.1 or openSUSE_13.2. Next, you should replicate the current state of the package from the OBS to your local computer. $> osc checkout home:<accountName>:Bar Foo This will create a new directory home:<accountName>:Bar with a subdirectory Foo. Change to this directory. It will be almost empty. Next steps ... 4. Creating the recipe for installing the distribution/build environment and doing the build - aka creating a "specfile" 4.1. Use https://build.opensuse.org/package/view_file/openSUSE:13.2/vim/spec.skeleton... as a specfile template. Rename it to foo.spec 4.2. Fill out the missing Key:'s. Have a look at some other specfiles, and read https://en.opensuse.org/openSUSE:Specfile_guidelines#Preamble The most important keys are Source: and BuildRequires: The Source should be an archive of your sources, e.g. foo-0.1.tar.xz. BuildRequires is at least "g++", but maybe also make and cmake. 4.3. After you have created the specfile foo.spec and copied the source archive to the same directory, you can test things locally: $> osc add foo-0.1.tar.xz $> osc add foo.spec $> osc build Now you will most likely get some errors. Fix these, use google, and if you can't find anything helpful (unlikely), feel free to ask here again. Reexecute "osc build", as you fix errors. After you have fixed all errors, submit your files to the OBS. $> osc checkin Now you can watch OBS building your package. Add more repositories. For simple packages, e.g. other openSUSE versions and most likely Fedora will work without further changes. Unfortunately, Debian/Ubuntu and Arch will require more work. Kind regards, Stefan -- Stefan Brüns / Bergstraße 21 / 52062 Aachen home: +49 241 53809034 mobile: +49 151 50412019 work: +49 2405 49936-424-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Fri, Nov 7, 2014 at 12:07 PM, Tobias Lauterbach <tol@qosmotec.com> wrote:
Hello,
I´m quite new to building packages on Linux, so I have some very basic, probably stupid questions, but after hours of luckless research this is the last way I see to understand this.
I have the task to find out how to avoid the need of 3 build servers for supplying three different distributions.
All I read is the open build service would be able to build binary packages from sources for a wide range of operating systems and hardware architectures. But I don't understand what those sources are.
Would I be able to compile binaries for different distributions from C++ code with this? And then based on those binaries build the packages?
If yes, how can I do it? If not, what is the benefit from just building packages if I need to build the binaries on different distributions?
Thank you very much, tol
Tol, Stefan seems to have written a nice reply, but tl;dr. (sorry Stefan) open build service is software specifically designed to do what you want. There are 3 variations you need to choose from: osc - the command line version that runs on your local linux box. Minimal special code needed. With it you can tell your openSUSE distribution to create a chroot jail and fill it up with the needed Radhat packages to let you compile a Redhat set of binaries, then put the binaries into an RPM. Lots of different distros can be both the host linux OS and the target linux OS. "open build service" itself. It comes as a full openSUSE distro with all the software and tools to create a dedicated build farm. I believe the farm can be as small as one PC, or a couple hundred (or more). People like Dell and Oracle run private instances like that. Obviously lots of smaller companies as well. In general, open build service uses VMs in which to do the builds, not chroot jails. It also has a WebUI, so you can interact with it from other than the command line. OBS - "build.opensuse.org" - This is the reference instance of "open build service". You can create an account and use it to your hearts content. It has numerous linux distros in the various repositories so you can build packages for any of them. === For your initial famiarization, take a look at how "osc" itself is packaged and made available: https://build.opensuse.org/package/show/openSUSE:Tools/osc Look at all the distros it is being built for on OBS. You can download the various packages from: http://software.opensuse.org/download.html?project=openSUSE%3ATools&package=osc === Once you have a login, you can create a sub-project within your home project and do similar things. As an example, the "sleuthkit" project created a project login and tinkered with doing what you want to do. They never actually did much, but you can see their concept by looking at: https://build.opensuse.org/project/show/home:sleuthkit === OBS, the reference instance, by policy can only host opensource projects. If the code your need to work with is proprietary you need to look at osc, or your own private instance of open build service. ======= I hope that isn't overwhelming, Greg -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi Stefan, hi Greg, thank you very much for your reply. With your help I managed to compile a hello World example with success on a few distributions. But I have two problems: 1. The built packages are empty. I guess I have to somehow add an instruction how to build the package(define which binaries should be installed with the package)? My workspace: https://build.opensuse.org/package/view_file/home:tol_qosmotec/helloWorld/he... https://build.opensuse.org/source/home:tol_qosmotec/helloWorld/helloWorld-1.... 2. Only CentOS Builds are started at the moment. The Debian and Ubuntu builds are marked as "excluded". I haven´t figured out why yet. Do you have any idea? One last thing for my understanding: Do I have to create an archive of my sources every time? Cause on our build server we have a source folder. I think it seems a bit unnecessary to do that every time when it needs to be unpacked shortly after that in the build distribution. Kind regards, Tobias N�����r��y隊Z)z{.���Wlz��qﮞ˛���m�)z{.��+�: �{Zr�az�'z��j)h���Ǜ�)]���Ǿ� ޮ�^�ˬz��
Am Dienstag, 25. November 2014, 10:13:56 schrieb Tobias Lauterbach:
Hi Stefan, hi Greg,
thank you very much for your reply.
With your help I managed to compile a hello World example with success on a few distributions. But I have two problems:
1. The built packages are empty. I guess I have to somehow add an instruction how to build the package(define which binaries should be installed with the package)?
have you looked at any other packages, as inspiration, yet? You have to add an %install section, install (or copy) the built binary over to $RPM_BUILDROOT%{_bindir}/hello_world_1, make sure its x-bit is set, and list that in your %files section, with the path it will have when finally installed to your system, i.e. %{_bindir}/hello_world_1.
My workspace: https://build.opensuse.org/package/view_file/home:tol_qosmotec/helloWorld/he... https://build.opensuse.org/source/home:tol_qosmotec/helloWorld/helloWorld-1....
2. Only CentOS Builds are started at the moment. The Debian and Ubuntu builds are marked as "excluded". I haven´t figured out why yet. Do you have any idea?
Because Debian/Ubuntu packages are .deb packages that require different control files than the spec you already have, again look at other projects/packages, and watch out for additional debian files (usually there's a small tar.gz holding a folder with meta information, and a .dsc file, the information in the .spec file is spread across these)
One last thing for my understanding: Do I have to create an archive of my sources every time? Cause on our build server we have a source folder. I think it seems a bit unnecessary to do that every time when it needs to be unpacked shortly after that in the build distribution.
for a single .cpp file that might make sense, but i doubt you will continue working on just a hello world example - imagine your software source tree grows, has subdirs and hundreds of files in it... Even a small GUI application usually consists of dozens of sources and quickly takes up a few hundred kilobytes. also the tarball should have a versioned name, like HelloWorld-0.0.1.tar.bz2 and accordingly a directory HelloWorld-0.0.1 inside - a sort of convention to avoid future confusion, make this "release" unique and distinguishable, and keep the packager's workflow easy ;) 2€cents, Edgar -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On November 25, 2014 5:13:56 AM EST, Tobias Lauterbach <tol@qosmotec.com> wrote:
Hi Stefan, hi Greg,
thank you very much for your reply.
With your help I managed to compile a hello World example with success on a few distributions. But I have two problems:
1. The built packages are empty. I guess I have to somehow add an instruction how to build the package(define which binaries should be installed with the package)? My workspace: https://build.opensuse.org/package/view_file/home:tol_qosmotec/helloWorld/he... https://build.opensuse.org/source/home:tol_qosmotec/helloWorld/helloWorld-1....
2. Only CentOS Builds are started at the moment. The Debian and Ubuntu builds are marked as "excluded". I haven´t figured out why yet. Do you have any idea?
One last thing for my understanding: Do I have to create an archive of my sources every time? Cause on our build server we have a source folder. I think it seems a bit unnecessary to do that every time when it needs to be unpacked shortly after that in the build distribution.
Kind regards, Tobias
It looks like you are making progress. A couple high level comments: - your first post was OBS specific and this mailing list is one of the few resources that exist to answer it. Now your questions are generic packaging questions, so you have the entire universe of packaging resources available to you. In particular, every RPM based distro uses specfiles to build the RPMs. That means, if someone is packaging your code for fedora, then they have an existing specfile, so you can go grab a copy of that one and use it on OBS to build fedora RPMs here. - you will soon learn that a specfile for fedora rarely works perfectly for opensuse, or any other distro. Assuming the changes are minimal then "%if distro" logic can allow you to customize the specfile for different distros. - OBS predefines various macros, some of which might be OBS specific, I'm not sure. I only package via OBS. - debian style distros don't use specfiles. OBS is not a standalone packaging solution, it uses each distros native packaging tools so you have to learn how to package for every distro you support. In general that is RPMs and DEBs, but there may be others on OBS I don't know about. - this mailing list is for overall OBS discussion. Once you start asking about what goes into a specfile, etc you have made the transition to packaging and the opensuse-packaging list is the place that focuses on those issues. Not that you will get kicked out of here, but you will have a different set of subscribers there and I suspect your questions will be exposed to a broader range of packagers there. Hope that helps, Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tue, Nov 25, 2014 at 8:48 AM, Greg Freemyer <greg.freemyer@gmail.com> wrote:
On November 25, 2014 5:13:56 AM EST, Tobias Lauterbach <tol@qosmotec.com> wrote:
Hi Stefan, hi Greg,
thank you very much for your reply.
With your help I managed to compile a hello World example with success on a few distributions. But I have two problems:
1. The built packages are empty. I guess I have to somehow add an instruction how to build the package(define which binaries should be installed with the package)? My workspace: https://build.opensuse.org/package/view_file/home:tol_qosmotec/helloWorld/he... https://build.opensuse.org/source/home:tol_qosmotec/helloWorld/helloWorld-1....
2. Only CentOS Builds are started at the moment. The Debian and Ubuntu builds are marked as "excluded". I haven´t figured out why yet. Do you have any idea?
One last thing for my understanding: Do I have to create an archive of my sources every time? Cause on our build server we have a source folder. I think it seems a bit unnecessary to do that every time when it needs to be unpacked shortly after that in the build distribution.
Kind regards, Tobias
It looks like you are making progress.
A couple high level comments:
- your first post was OBS specific and this mailing list is one of the few resources that exist to answer it. Now your questions are generic packaging questions, so you have the entire universe of packaging resources available to you. In particular, every RPM based distro uses specfiles to build the RPMs.
That means, if someone is packaging your code for fedora, then they have an existing specfile, so you can go grab a copy of that one and use it on OBS to build fedora RPMs here.
- you will soon learn that a specfile for fedora rarely works perfectly for opensuse, or any other distro. Assuming the changes are minimal then "%if distro" logic can allow you to customize the specfile for different distros.
- OBS predefines various macros, some of which might be OBS specific, I'm not sure. I only package via OBS.
- debian style distros don't use specfiles. OBS is not a standalone packaging solution, it uses each distros native packaging tools so you have to learn how to package for every distro you support. In general that is RPMs and DEBs, but there may be others on OBS I don't know about.
- this mailing list is for overall OBS discussion. Once you start asking about what goes into a specfile, etc you have made the transition to packaging and the opensuse-packaging list is the place that focuses on those issues. Not that you will get kicked out of here, but you will have a different set of subscribers there and I suspect your questions will be exposed to a broader range of packagers there.
A couple more generic comments. - Both -buildservice and -packaging are open lists meaning you don't have to subscribe to post. If you don't subscribe I would make a point of asking to be kept in CC on replies. - Both lists are archived various places. My favorite search site for the archives is http://opensuse.markmail.org. Here's a sample search for "rpm": <http://opensuse.markmail.org/search/?q=rpm> - It gets a couple thousands hits, so not a great choice of search terms but it shows you the names of the opensuse lists discussing rpms on the left middle panel. - build.opensuse.org is a rather large build farm. On the bottom of every page is a link to "status monitor". There will be times that OBS seems slow. By looking at the status monitor you can see how well the overall farm is doing. I tend to scroll down to the bottom a look at the monthly load graphs. You will see the load is highly variable and a big set of builds (10's of thousands of projects to build at one time) can take days to work through. - osc is a command line tool that is great for interacting with OBS. It can do local builds on your own linux machine. It is available for lots of distros. I use it a lot. The typical workflow for me is: osc co <project> # checkout a copy of a project to my local linux box edit specfile or make other changes osc build # test out my changes repeat edit / build sequence as needed to get the build working install RPM to local machine and test repeat until the binaries, etc. in the RPM do what you want osc vc # update the changes file to explain why the project is being updated osc commit # push the changes back to OBS for compiling with multiple distros / versions By using osc like that to do local builds you are much less impacted if the farm itself is busy. Even at its fastest the farm is slower than a local build. Greg -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (4)
-
Brüns, Stefan -
Edgar Aichinger -
Greg Freemyer -
Tobias Lauterbach