[opensuse-ruby] new wiki page for Ruby gem packaging strategies
[Reply-To: set to opensuse-ruby] Hi all, After many discussions on this list, and also many internal discussions within SUSE, I have attempted to summarise a collective understanding about the pros and cons of the various strategies for packaging Ruby applications. This is not a complete understanding since it was not possible to involve all experts in the area, nor is it even at this point a set of guidelines or recommendations for which strategy to adopt. However hopefully it should help Ruby application architects make an informed decision about which approach best suits their project. Feedback is warmly encouraged - please feel free to edit the wiki page directly with your additions / amendments! Thanks, Adam -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
Adam Spiers (aspiers@suse.com) wrote:
[Reply-To: set to opensuse-ruby]
Hi all,
After many discussions on this list, and also many internal discussions within SUSE, I have attempted to summarise a collective understanding about the pros and cons of the various strategies for packaging Ruby applications.
[snipped] Ahem. It might help to include the link ;-) https://en.opensuse.org/openSUSE:Ruby_Gem_Strategies
This is not a complete understanding since it was not possible to involve all experts in the area, nor is it even at this point a set of guidelines or recommendations for which strategy to adopt. However hopefully it should help Ruby application architects make an informed decision about which approach best suits their project.
Feedback is warmly encouraged - please feel free to edit the wiki page directly with your additions / amendments!
Thanks, Adam -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
On Thu, Jan 24, 2013 at 07:32:11PM +0100, Adam Spiers wrote:
Adam Spiers (aspiers@suse.com) wrote:
[Reply-To: set to opensuse-ruby]
Hi all,
After many discussions on this list, and also many internal discussions within SUSE, I have attempted to summarise a collective understanding about the pros and cons of the various strategies for packaging Ruby applications.
[snipped]
Ahem. It might help to include the link ;-)
Hi Adam, interesting reading - but this is not just about Ruby. Wearing my Java packager's hat, with just changes (s/Ruby/Java/; s/gem/jar/; s/Gemfile.lock/maven/) it 100% apply there as well. The reason why it appears more in environment such Ruby community is that it is much easier to download gem, or jar or python's egg to some dir and make it a de-facto a part of your code, than to make the same with a native library. And (I'll discuss it later) the native packaging is better researched and even solved problem. And this is even the reason why packaging of this s*** (let say stuff) is so hard, because no-one excluding few poor souls even think about component dependencies, API stability and so. Extend zypper to support installing gems directly ================================================= Note the "we" means a distributor, so someone, who will be responsible for the result. * we rely on external service - things can dissappear, or be silently changed without our control * we can't make changes or fix a bugs * we even don't know if the code have a proper license * in Java world it is common jars includes a bytecode, not a source code - but afaik nothing prevents you to publish a gem with a Ruby bytecode only So yes, this is an easy path to make a project done, but a nightmare for the maintenance. IOW: if you want to throw a result and never look back, this is the best approach. zypper doesn't support multiple versions of same gem concurrently installed by default. ==================================================================== I completely disagree with above sentence! Sorry to say it, but if you make a bad package, you'll get a bad results. Just don't call it a zypper fault, because it is not. Really, there is nothing in rpm/zypper prevents you to do so. Rpm just does not let you to install two packages at the same location. And this is very reasonable constraint, otherwise, you'll end with two packages installed according rpmdb, but only the later will be there. To make an example, I have libncurses5 and libncurses6 installed on one system and it works - zypper, neither rpm complaints and every program use the correct version of a library. And this is achieved because 1.) shared library packaging policy [1], results in packages installable in a parallel 2.) linker (ld) and loader (ld.so) and autoreqprov magic in rpmbuild. They all ensures the needed packages are installed and correct libraries are loaded. Number one is easy to achieve if you already have an automation. But writing an alternative to ld.so is tricky part, because it is not solved upstream. For python it can be probably achieved by virtualenv, but don't know Ruby in such big details - Google says there is RVM [2]. [1] http://en.opensuse.org/openSUSE:Shared_library_packaging_policy [2] https://rvm.io/ BTW: even this does not saves you from inter-dependencies, like gem A needs gem B-v1.0, where gem C needs B-v2.0 and in some time, you'll need to use A and C together. This is to some extent solved in ld.so and ld, but I'm afraid unsolvable without a big hacking in rest of the world. Fortunately this is probably very rare issue. Regards Michal Vyskocil
Hi Michal, Firstly thanks a lot for sharing your thoughts! Michal Vyskocil (mvyskocil@suse.cz) wrote:
On Thu, Jan 24, 2013 at 07:32:11PM +0100, Adam Spiers wrote:
interesting reading - but this is not just about Ruby. Wearing my Java packager's hat, with just changes (s/Ruby/Java/; s/gem/jar/; s/Gemfile.lock/maven/) it 100% apply there as well.
Yeah I see a lot of parallels too, that's why I added a section at the end of the page for comparisons with other languages. I think 100% is maybe a bit high though ;-) e.g. I guess Java does not have the ~> "pessimistic" version operator, and Bundler works differently to maven. [snipped]
Extend zypper to support installing gems directly =================================================
Note the "we" means a distributor, so someone, who will be responsible for the result.
* we rely on external service - things can dissappear, or be silently changed without our control * we can't make changes or fix a bugs * we even don't know if the code have a proper license
These are good points - please add them to the disadvantages list on the wiki page!
* in Java world it is common jars includes a bytecode, not a source code - but afaik nothing prevents you to publish a gem with a Ruby bytecode only
That sounds like a strange possibility but I can't say you're wrong.
So yes, this is an easy path to make a project done, but a nightmare for the maintenance. IOW: if you want to throw a result and never look back, this is the best approach.
zypper doesn't support multiple versions of same gem concurrently installed by default. ====================================================================
I completely disagree with above sentence! Sorry to say it, but if you make a bad package, you'll get a bad results.
I think you misunderstood the statement:
Just don't call it a zypper fault, because it is not.
I didn't call it a fault ;-) But it is a limitation. It's a deliberate feature of zypper (see below).
Really, there is nothing in rpm/zypper prevents you to do so.
In rpm there is nothing. In zypper there definitely is. I am not talking about overlapping files, I am talking about two versions of the same package, e.g. rubygem-rails-2_3-2.3.9 rubygem-rails-2_3-2.3.11 Unless you change multiversion settings inside /etc/zypp, there is no way to install these together using zypper. That's why I said "by default". grep for 'multiversion' in /etc/zypp/zypp.conf and you will see what I mean. My suggestion in the wiki page was to add an entry to /etc/zypp/multiversion.d just for rubygems, but some people don't like that because there is no known solution for clean removal of unused old versions of gems. So different versions of the same gem could accumulate over time on a system. The kernel packages have the same problem, but it is a much bigger problem for them because kernels are a lot bigger than gems and /boot can easily fill up.
Rpm just does not let you to install two packages at the same location. And this is very reasonable constraint, otherwise, you'll end with two packages installed according rpmdb, but only the later will be there.
Right - but the limitation is in zypper not rpm. [snipped]
BTW: even this does not saves you from inter-dependencies, like gem A needs gem B-v1.0, where gem C needs B-v2.0 and in some time, you'll need to use A and C together. This is to some extent solved in ld.so and ld, but I'm afraid unsolvable without a big hacking in rest of the world.
This is currently solved in Ruby by using "-1" or "-1_2" style suffixes in the Name: field of the spec file: https://en.opensuse.org/openSUSE:Packaging_Ruby#Updating_gem_packages_to_a_n... but it has some known disadvantages which the multiversion approach could possibly solve. However they are corner cases so it's not currently clear whether it's worth switching to multiversion. -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
* in Java world it is common jars includes a bytecode, not a source code - but afaik nothing prevents you to publish a gem with a Ruby bytecode only
That sounds like a strange possibility but I can't say you're wrong.
There is the possibility to have built native gems. It is not common and I just have seen it on some docs regarding speeding up deployment. That usually means building yourself the native gems and then installing them on production. -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
On 24.01.2013 19:31, Adam Spiers wrote:
[Reply-To: set to opensuse-ruby]
Hi all,
After many discussions on this list, and also many internal discussions within SUSE, I have attempted to summarise a collective understanding about the pros and cons of the various strategies for packaging Ruby applications.
This is not a complete understanding since it was not possible to involve all experts in the area, nor is it even at this point a set of guidelines or recommendations for which strategy to adopt. However hopefully it should help Ruby application architects make an informed decision about which approach best suits their project.
Feedback is warmly encouraged - please feel free to edit the wiki page directly with your additions / amendments!
Using bundler/gem just dropped from that list: http://blog.newrelic.com/2013/01/30/new-relic-and-rubygems-security/ Greetings, Stephan -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
Stephan Kulow (coolo@suse.de) wrote:
On 24.01.2013 19:31, Adam Spiers wrote:
[Reply-To: set to opensuse-ruby]
Hi all,
After many discussions on this list, and also many internal discussions within SUSE, I have attempted to summarise a collective understanding about the pros and cons of the various strategies for packaging Ruby applications.
This is not a complete understanding since it was not possible to involve all experts in the area, nor is it even at this point a set of guidelines or recommendations for which strategy to adopt. However hopefully it should help Ruby application architects make an informed decision about which approach best suits their project.
Feedback is warmly encouraged - please feel free to edit the wiki page directly with your additions / amendments!
Using bundler/gem just dropped from that list:
http://blog.newrelic.com/2013/01/30/new-relic-and-rubygems-security/
I added the link to the wiki. -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
On Thursday 31 January 2013 15:13:12 Stephan Kulow wrote:
Using bundler/gem just dropped from that list:
http://blog.newrelic.com/2013/01/30/new-relic-and-rubygems-security/
How is this related to the question how gems are packaged? In the end they are all coming from rubygems in any case. -- Cornelius Schumacher <cschum@suse.de> -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
On Thu, 31 Jan 2013 17:41:15 +0100 Cornelius Schumacher <cschum@suse.de> wrote:
On Thursday 31 January 2013 15:13:12 Stephan Kulow wrote:
Using bundler/gem just dropped from that list:
http://blog.newrelic.com/2013/01/30/new-relic-and-rubygems-security/
How is this related to the question how gems are packaged? In the end they are all coming from rubygems in any case.
Well, there is one big difference. We control it. So e.g. if OBS is extended to provide also difs for gems, you can review changes from last submit and review it. If you use directly rubygems.org, then you depend on external service, where is no guaranty and as last case show no review. Of course own gem server is different case, but there is other problems like that we must maintain it. It must be public so also we must secure it etc. Josef -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
Quoting Josef Reidinger <jreidinger@suse.cz>:
On Thu, 31 Jan 2013 17:41:15 +0100 Cornelius Schumacher <cschum@suse.de> wrote:
On Thursday 31 January 2013 15:13:12 Stephan Kulow wrote:
Using bundler/gem just dropped from that list:
http://blog.newrelic.com/2013/01/30/new-relic-and-rubygems-security/
How is this related to the question how gems are packaged? In the end they are all coming from rubygems in any case.
Well, there is one big difference. We control it. So e.g. if OBS is extended to provide also difs for
"if it were extended" ....
gems, you can review changes from last submit and review it. If you use directly rubygems.org, then you depend on external service, where is no guaranty and as last case show no review. Of course own gem server is different case, but there is other problems like that we must maintain it. It must be public so also we must secure it etc.
Josef -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
Quoting Josef Reidinger <jreidinger@suse.cz>:
On Thu, 31 Jan 2013 17:41:15 +0100 Cornelius Schumacher <cschum@suse.de> wrote:
On Thursday 31 January 2013 15:13:12 Stephan Kulow wrote:
Using bundler/gem just dropped from that list:
http://blog.newrelic.com/2013/01/30/new-relic-and-rubygems-security/
How is this related to the question how gems are packaged? In the end they are all coming from rubygems in any case.
Well, there is one big difference. We control it. So e.g. if OBS is extended to provide also difs for gems, you can review changes from last submit and review it. If you use directly rubygems.org, then you depend on external service, where is no guaranty and as last case show no review. Of course own gem server is different case, but there is other problems like that we must maintain it. It must be public so also we must secure it etc.
"must be public" ?? I do not see why.
Josef -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
On Thu, 31 Jan 2013 18:57:34 +0100 Jordi Massaguer Pla <jmassaguerpla@suse.de> wrote:
Quoting Josef Reidinger <jreidinger@suse.cz>:
On Thu, 31 Jan 2013 17:41:15 +0100 Cornelius Schumacher <cschum@suse.de> wrote:
On Thursday 31 January 2013 15:13:12 Stephan Kulow wrote:
Using bundler/gem just dropped from that list:
http://blog.newrelic.com/2013/01/30/new-relic-and-rubygems-security/
How is this related to the question how gems are packaged? In the end they are all coming from rubygems in any case.
Well, there is one big difference. We control it. So e.g. if OBS is extended to provide also difs for gems, you can review changes from last submit and review it. If you use directly rubygems.org, then you depend on external service, where is no guaranty and as last case show no review. Of course own gem server is different case, but there is other problems like that we must maintain it. It must be public so also we must secure it etc.
"must be public" ?? I do not see why.
Well, maybe my fault. I think that this disqualify solution that zypper can work with rubygems. Because in this case all customer should be able to download from rubygems org or from our gemserver. It is also valid for any solution that do not pack all required gems during build on internal server ( because OBS is in public network, so only IBS can do it ). Josef
Josef -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
Quoting Josef Reidinger <jreidinger@suse.cz>:
On Thu, 31 Jan 2013 18:57:34 +0100 Jordi Massaguer Pla <jmassaguerpla@suse.de> wrote:
Quoting Josef Reidinger <jreidinger@suse.cz>:
On Thu, 31 Jan 2013 17:41:15 +0100 Cornelius Schumacher <cschum@suse.de> wrote:
On Thursday 31 January 2013 15:13:12 Stephan Kulow wrote:
Using bundler/gem just dropped from that list:
http://blog.newrelic.com/2013/01/30/new-relic-and-rubygems-security/
How is this related to the question how gems are packaged? In the end they are all coming from rubygems in any case.
Well, there is one big difference. We control it. So e.g. if OBS is extended to provide also difs for gems, you can review changes from last submit and review it. If you use directly rubygems.org, then you depend on external service, where is no guaranty and as last case show no review. Of course own gem server is different case, but there is other problems like that we must maintain it. It must be public so also we must secure it etc.
"must be public" ?? I do not see why.
Well, maybe my fault. I think that this disqualify solution that zypper can work with rubygems. Because in this case all customer should be able to download from rubygems org or from our gemserver. It is also valid for any solution that do not pack all required gems during build on internal server ( because OBS is in public network, so only IBS can do it ). Josef
Josef, I understand your point. I just want to clarify that we do package gems in an RPM. The difference between my approach and yours is the amount of RPMs. You have one RPM per gem and I have all gems into one RPM, with all the advantages and disadvantages we have discussed.
Josef -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org
participants (6)
-
Adam Spiers -
Cornelius Schumacher -
Jordi Massaguer Pla -
Josef Reidinger -
Michal Vyskocil -
Stephan Kulow