Taking ruby packaging to the next level ========================================= Table of Content ------------------ 1. TL;DR 2. Where we started 3. The basics 4. One step at a time 5. Rocks on the road 6. "Job done right?" "Well almost." 7. Whats left? TL;DR ------- * we are going back to versioned ruby packages with a new naming scheme. * one spec file to rule them all. (one spec file to build a rubygem for all interpreters) * macro based buildrequires. %{rubygem rails:4.1 >= 4.1.4} * gem2rpm.yml as configuration file for gem2rpm. no more manual editing of spec files. * with the config spec files can always be regenerated without losing things. Where we started ------------------ A long time ago we started with the support for building for multiple ruby versions, actually MRI versions, at the same time. The ruby base package was in good shape in that regard. But we had one open issue - building rubygems for multiple ruby versions. This issue was hanging for awhile so we went and reverted to a single ruby version packaged for openSUSE again. While this might work for openSUSE, it can become really challenging for SLES with its long life cycle. Even in the openSUSE world we have Evergreen for 11.4 still alive. That was released in March 2011 and still no sign that Evergreen support for it is ending. We really want multiple ruby versions here. With that in mind we were wondering how much work would be left to do, so we can actually fully support multiple ruby versions or even multiple ruby interpreters. The steps we need to take --------------------------- When asking around for "If we support multiple versions again what would you expect in regard to rubygems packaging?", we got a few important points. * only 1 spec file for all. python currently uses 1 spec file for each version * avoid re-packaging in d:l:r:e [1] * system ruby should always be /usr/bin/ruby. The basics ------------ The first step was to restore the versioned ruby packaging and add at least one extra version. As a start we used a snapshot package for MRI 2.2. All the common bits needed for all ruby interpreters moved into ruby-common. The unversioned ruby package became a wrapper to pull in the system ruby. As mentioned above /usr/bin/ruby should always point to the system ruby. So in the new schema it is **not** a symlink handled by update-alternatives, but a hardlink. The same goes for %{_libdir}/libruby.so. It is actually a pity that we still need this file, as any good program linking ruby should just query RbConfig for the commands to link the libruby for the currently running interpreter. Sadly some tools ask for the CFLAGS but then hardcode -lruby. Ugh. Our naming scheme is ``` mainpackage = "#{interpreter}#{major}.#{minor}" jruby1.7 rubinius2.2 ruby2.1 ruby2.2 ``` The interperter name is more or less the same except for rubinius. Their binary is called rbx. So we get: ``` interpretername = "#{binaryname}#{major}.#{minor}" jruby1.7 rbx2.2 ruby2.1 ruby2.2 ``` As we wanted to support gems for more than one version, we also need to express that in their package names now. ``` gempackage = "#{interpretername}-rubygem-#{gemname}#{gemsuffix}" # gemsuffix is optional. we only need it for packages where we need more # than one version jruby1.7-rubygem-tzinfo rbx2.2-rubygem-tzinfo ruby2.1-rubygem-tzinfo ruby2.2-rubygem-tzinfo ``` The gem binary naming is a bit longer now as well. But most of you should not notice anything as update-alternatives covers this: gembinary = "#{binaryname}.#{interpretername}-#{gemversion}" ``` $ ls -l /usr/bin/bundler* lrwxrwxrwx 1 root root 25 25. Jul 19:04 /usr/bin/bundler -> /etc/alternatives/bundler lrwxrwxrwx 1 root root 31 25. Jul 19:04 /usr/bin/bundler-1.6.5 -> /etc/alternatives/bundler-1.6.5 lrwxrwxrwx 1 root root 38 5. Sep 13:00 /usr/bin/bundler.rbx2.2 -> ../lib64/rubinius/gems/2.2/bin/bundler lrwxrwxrwx 1 root root 33 25. Jul 19:04 /usr/bin/bundler.ruby2.1 -> /etc/alternatives/bundler.ruby2.1 -rwxr-xr-x 1 root root 504 25. Jul 18:53 /usr/bin/bundler.ruby2.1-1.6.5 lrwxrwxrwx 1 root root 33 8. Sep 19:19 /usr/bin/bundler.ruby2.2 -> /etc/alternatives/bundler.ruby2.2 -rwxr-xr-x 1 root root 504 5. Sep 15:57 /usr/bin/bundler.ruby2.2-1.6.5 ``` Last but not least we have the ruby(abi). In the past it used to be just the "version" number you had in your ruby paths. Now the define also contains the ruby interpreter. If we take our example list from above: ``` jruby:1.7 rubinius:2.2 ruby:2.1.0 ruby:2.2.0 ``` While this works nicely for our packaging needs. It is actually tricky for cases where a gem wants at least a certain ruby version. The gemspec only has spec.required_ruby_version which is a version number. While in MRI this number is compared with the ruby version number, in the case of rubinius/jruby it is compared with the version of ruby language standard that is implemented. One possible solution would be to add also mri(abi), jruby(abi), rubinius(abi), which just have a numerical abi comparison. ``` Provides: jruby(abi) = 20100 Provides: rubinius(abi) = 20100 Provides: mri(abi) = 20100 Provides: mri(abi) = 20200 ``` One step at a time -------------------- Many of our gems nowadays build without having a buildrequires to their runtime dependencies. We only generate the dependencies into the package meta. Though some packages still could require e.g. rspec so they could run their testsuite at build time. At this point generating multiple spec files seemed like an easier solution: we would generate build deps for each ruby interpreter/version into the spec file. But we were asked not to, so we stick with a single spec file. Fortunately rpm gives us macros, which on the other hand also need support in the buildservice. Good thing is the guy to make that happen for rpm and the OBS is the same person. :D ``` BuildRequires: %{ruby} BuildRequires: %{rubydevel} BuildRequires: %{rubygem somegem >= someversion} ``` Without going into too many details here [2], in home:darix:ruby this expands to: ``` BuildRequires: ruby2.1 ruby2.2 rubinius2.2 BuildRequires: ruby2.1-devel ruby2.2-devel rubinius2.2-devel BuildRequires: rubygem(ruby:2.1.0:somegem) >= someversion rubygem(ruby:2.2.0:somegem) >= someversion rubygem(rubinius:2.2:somegem) >= someversion ``` Most of our machinery for installing and cleaning up things was hidden in macros/shell scripts already so adding a loop in those places was trivial. Suddenly we have multiple ruby interpreter/versions in the same build root and our gems build with each of them. So far nothing that would break all that horribly when we push it onto d:l:r:e. Rocks on the road ------------------- The problems started when we came to the %files sections in the spec files. Until now we had been generating them into the spec file. That means for every new ruby interpreter/major branch we would need to regenerate all spec files. That is not really a viable solution going forward. So we replaced the static files section with a macro too. ``` %gem_packages ``` But this flexibility comes at a price. If building the gem fails for one ruby interpreter/version, it will break the build for all. But you can control which interpreters are even pulled into the build environment. ``` %define rb_build_versions %{rb_default_ruby} BuildRequires: %{rubydevel} BuildRequires: %{rubygem cheetah} ``` In home:darix:ruby this would expand to: ``` BuildRequires: ruby2.1-devel BuildRequires: rubygem(ruby:2.1.0:cheetah) ``` The valid values for %rb_build_versions can be found on the in the macros files of each interpreter package. We would have loved to use the package names as the macro values but those values are passed into macros again and macro names can not contain dots. For easier reading you can look at the prjconf of home:darix:ruby. [2] "Job done right?" "Well almost." ---------------------------------- There were all those little bits and pieces that creeped in now. We had gems with %pre/%post scriptlets, because the gem was actually a somewhat better tarball for a service. Rubygems also lacked a way to express native buildrequires. As it became clear that we will need to regenerate all the spec files in d:l:r:e, we aimed for a solution that allowed us regenerating the spec files at any time. **No more manually editing spec files** So we looked through a huge selection of spec files and checked what things, we actually modified. Once that list was compiled, we created a config file. The config file is named gem2rpm.yml. Each field in the config file [3] has a matching hook in the spec file template or files section template [4] (via %gem_packages). gem2rpm.yml was patched to support the config file. Right now you have to manually pass the config file, but there is a small shell wrapper [5] that checks if the config file exists and adds the option automatically. With that in place you can regenerate all spec files without worrying to lose manual edits. There wont be any. And yes ... in the development process this has been done multiple times after fixes to the spec file template. The final result? ``` # # spec file for package rubygem-tzinfo-0 <snip> # This file was generated with a gem2rpm.yml and not just plain gem2rpm. # All sections marked as MANUAL, license headers, summaries and descriptions # can be maintained in that file. Please consult this file before editing any # of those fields # Name: rubygem-tzinfo-0 Version: 0.3.37 Release: 0 %define mod_name tzinfo %define mod_full_name %{mod_name}-%{version} %define mod_version_suffix -0 BuildRoot: %{_tmppath}/%{name}-%{version}-build BuildRequires: ruby-macros >= 5 BuildRequires: %{ruby} BuildRequires: %{rubygem gem2rpm} BuildRequires: %{rubygem rdoc > 3.10} Url: http://tzinfo.rubyforge.org/ Source: http://rubygems.org/gems/%{mod_full_name}.gem Source1: gem2rpm.yml Summary: Daylight-savings aware timezone library License: MIT Group: Development/Languages/Ruby %description TZInfo is a Ruby library that uses the standard tz (Olson) database to provide daylight savings aware transformations between times in different time zones. %prep %build %install %gem_install \ --doc-files="CHANGES LICENSE README" \ -f %gem_packages %changelog ``` As you see this spec uses a gem2rpm.yml, which looks like this: ``` --- :version_suffix: '-0' ``` With that spec file the build generates for me: ``` rbx2.2-rubygem-tzinfo-0-0.3.37-9.4.x86_64.rpm rbx2.2-rubygem-tzinfo-doc-0-0.3.37-9.4.x86_64.rpm rbx2.2-rubygem-tzinfo-testsuite-0-0.3.37-9.4.x86_64.rpm ruby2.1-rubygem-tzinfo-0-0.3.37-9.4.x86_64.rpm ruby2.1-rubygem-tzinfo-doc-0-0.3.37-9.4.x86_64.rpm ruby2.1-rubygem-tzinfo-testsuite-0-0.3.37-9.4.x86_64.rpm ruby2.2-rubygem-tzinfo-0-0.3.37-9.4.x86_64.rpm ruby2.2-rubygem-tzinfo-doc-0-0.3.37-9.4.x86_64.rpm ruby2.2-rubygem-tzinfo-testsuite-0-0.3.37-9.4.x86_64.rpm ``` This has already been used extensively to build [Discourse](http://discourse.org) and [GitLab](https://about.gitlab.com/) for SLE 12. What is left? --------------- If you look just at the things that are packaged using the new way, we are done. But it would be nice to also support the versioning pattern that we have used up to 13.1. Maybe even support building against unversioned ruby versions. (Yes, in theory that is possible.) Once we have taken those steps, we have to recreate all spec files in d:l:r:e. Yes, you read correctly ... recreate all spec files. While we wrote all macros in a way to work as a drop in replacement, as soon as we want to build for more than one ruby version, we need the %gem_packages usage. I really hope enough people step up to the effort so that the number of packages each person has to touch is kept in the low double digits. Most of the work will be extracting the manual bits into gem2rpm.yml files and then regenerate the spec file with the new template. A more detailed mail will be sent later. Last but not least we can improve the templates (move duplicated code into gem2rpm e.g.) Building non gem based ruby libraries is not covered by this new packaging at all. For once many of the non rubygem based libraries are bindings build within a larger build process. We would need to manually do the loop for all ruby interpreter in those. I think in many cases we only need those libraries for our system ruby. (like yast e.g.) For the other cases we should work with upstream to switch the ruby bindings to rubygems. Also not covered yet, but it would be really useful: All the ruby scripts shipped as part of the distro should use a shebang line pointing to the versioned binary. You might ask "why? /usr/bin/ruby is a hardlink!" While this is true ... people can still replace it. Our scripts/programs should **not** break in that situation. E.g. yast is very critical in that regard. Again a gem based distribution makes it easier as rubygems does it for us in that case. But it shouldn't be much work to add a small script to fix shebang lines and an rpmlint check that finds all shebang lines that are unversioned. Footnotes ----------- [1]: https://build.opensuse.org/project/show/devel:languages:ruby:extensions [2]: The interested reader can check https://build.opensuse.org/project/prjconf/home:darix:ruby It involves recursive macro calls and other fun things. You have been warned. [3]: http://files.nordisch.org/ruby-packaging-next/gem2rpm.yml [4]: http://files.nordisch.org/ruby-packaging-next/sles12.spec.erb http://files.nordisch.org/ruby-packaging-next/sles12.gem_packages.spec.erb [5]: the little wrapper called "g2r" ``` #!/bin/sh if [ -e gem2rpm.yml ] ; then cfg="--config gem2rpm.yml" fi exec gem2rpm $cfg -t /usr/share/doc/packages/rubygem-gem2rpm/sles12.spec.erb -o *spec *gem ``` -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org