[opensuse-ruby] Howto package a Rails App?

newer
[opensuse-ruby] new wiki page for...

Ralf Haferkamp

9 Jan 2013 9 Jan '13

12:25

Hi, I currently try to package a rails app as RPM and ran into some issues that don't know howto solve. The current state of that package can be seen here: https://build.opensuse.org/package/show?package=crowbar-barclamp-crowbar&project=home%3Arhafer%3Abranches%3Asystemsmanagement%3Acrowbar%3A2.0 The app ships with a Gemfile but does not include the Gemfile.lock (upstream has some Good Reason(TM) for that). One of my problems is now that in order to be able to use the app I need to call "bundle install" at some point either during buildtime or after installing package. (Ideally with --local so it will not talk to any remote servers and just use the gem install locally). That means however that all gems listed in Gemfile will become BuildRequires of the rails-app. Even when calling "gem install local --without developement test" the gems in the development and test groups need to be present. Apart from the fact that this will add quite a few useless build dependencies to the package I also ran some other problems. E.g. the Gemfile contains this: gem 'rcov', :platforms => :ruby_18, :group => [:development, :test] gem 'simplecov', :platforms => :ruby_19, :group => :test, :require => false Even when "bundle install" runs on a :ruby_19 platform it needs the gems for the :ruby_18 to be present (AFAIK is needs access to some metadata of all gems mentioned in Gemfile). That means even when building for Factory I would need to add rubygem-rcov as a BuildRequires, which doesn't work as rubygem-rcov depends on ruby-1.8 which we don't have on Factory. Options to solve this I can think of: 1. Patch the gemfile to not require rcov 2. Avoid all dependencies on rubygem-*rpms, by calling "bundle install/package" on my local machine and vendor all dependencies into the app's rpm package. If possible I would like to avoid both of the above options. Does anybody here have an idea if that is possible? What is the general approach to packaging rails apps? (Avoid it? ;)) Is there a way to avoid bloating the BuildRequires and still create a working RPM without the need to call bundler in the installed system? -- regards, Ralf -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Show replies by thread

Josef Reidinger

9 Jan 9 Jan

12:37

On Wed, 9 Jan 2013 13:25:22 +0100 Ralf Haferkamp wrote:

...

Hi,

I currently try to package a rails app as RPM and ran into some issues that don't know howto solve.

The current state of that package can be seen here: https://build.opensuse.org/package/show?package=crowbar-barclamp-crowbar&project=home%3Arhafer%3Abranches%3Asystemsmanagement%3Acrowbar%3A2.0

The app ships with a Gemfile but does not include the Gemfile.lock (upstream has some Good Reason(TM) for that). One of my problems is now that in order to be able to use the app I need to call "bundle install" at some point either during buildtime or after installing package. (Ideally with --local so it will not talk to any remote servers and just use the gem install locally).

Hi Ralf, I describe how webyast and SLMS ( both are rails servers ) do it. if you have all packages available as RPMS, you do not need to call bundle install. But for resolution you need to move away other kinds. SLMS use groups and do for i in $RPM_BUILD_ROOT/usr/share/slms/*/Gemfile; do sed -i '/^group/,/^end/s/^/#/' $i; done That comment out all groups, of course you can adapt script to your needs. I already complain about it, but it looks like bundler upstream is not interested in such topic. Webyast have some task, that allows to install specific gemfile for given environment - see https://github.com/webyast/webyast/tree/master/webyast part in spec file. Also it is interesting that Gemfile can interpret ruby code, so you can comment it out quite easy. Also do not forget to remove Gemfile.lock after update of any gem. We solve it with removing Gemfile.lock in init script and during everytime is created new one with fresh versions.

...

That means however that all gems listed in Gemfile will become BuildRequires of the rails-app. Even when calling "gem install local --without developement test" the gems in the development and test groups need to be present. Apart from the fact that this will add quite a few useless build dependencies to the package I also ran some other problems. E.g. the Gemfile contains this:

gem 'rcov', :platforms => :ruby_18, :group => [:development, :test] gem 'simplecov', :platforms => :ruby_19, :group => :test, :require => false

Even when "bundle install" runs on a :ruby_19 platform it needs the gems for the :ruby_18 to be present (AFAIK is needs access to some metadata of all gems mentioned in Gemfile). That means even when building for Factory I would need to add rubygem-rcov as a BuildRequires, which doesn't work as rubygem-rcov depends on ruby-1.8 which we don't have on Factory.

Options to solve this I can think of: 1. Patch the gemfile to not require rcov 2. Avoid all dependencies on rubygem-*rpms, by calling "bundle install/package" on my local machine and vendor all dependencies into the app's rpm package.

If possible I would like to avoid both of the above options. Does anybody here have an idea if that is possible?

Sorry, I don't see it. I can point you to some internal discussion about this topic and also to my patch for bundler, that is still pending for easier excluding of groups.

...

What is the general approach to packaging rails apps? (Avoid it? ;))

Is there a way to avoid bloating the BuildRequires and still create a working RPM without the need to call bundler in the installed system?

you need to call bundler in installed system ( not directly, but indirectly via starting rails server ) to properly load gems during rails start. Josef

...

-- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Ralf Haferkamp

15:40

On Wed, Jan 09, 2013 at 01:37:18PM +0100, Josef Reidinger wrote:

...

On Wed, 9 Jan 2013 13:25:22 +0100 Ralf Haferkamp wrote:

...
Hi,

I currently try to package a rails app as RPM and ran into some issues that don't know howto solve.

The current state of that package can be seen here: https://build.opensuse.org/package/show?package=crowbar-barclamp-crowbar&project=home%3Arhafer%3Abranches%3Asystemsmanagement%3Acrowbar%3A2.0

The app ships with a Gemfile but does not include the Gemfile.lock (upstream has some Good Reason(TM) for that). One of my problems is now that in order to be able to use the app I need to call "bundle install" at some point either during buildtime or after installing package. (Ideally with --local so it will not talk to any remote servers and just use the gem install locally).

Hi Ralf, I describe how webyast and SLMS ( both are rails servers ) do it. Yeah, thanks. That helps a lot.

...

if you have all packages available as RPMS, you do not need to call bundle install. But for resolution you need to move away other kinds. SLMS use groups and do for i in $RPM_BUILD_ROOT/usr/share/slms/*/Gemfile; do sed -i '/^group/,/^end/s/^/#/' $i; done

That comment out all groups, of course you can adapt script to your needs. Yeah, this is the approach (or at least a similar one) I am currently trying as well. I created a small script that parses the Gemfile an filters out unneeded groups and platforms and generates a new Gemfile from that.

...

I already complain about it, but it looks like bundler upstream is not interested in such topic. Yeah I have seen a couple of similar complaints when looking around for solutions to this. Most people just seem to include all the dependend gems into

As a bonus it is also able to write out the apps RPM dependencies similar to what rubygemdeps (in ruby-common) does for rubygems. It is however not integrated with rpm fileattrs and I don't see a good way of achieving that yet. So currently I ran that manually. And given the fact that I only use it for a single package currently I guess that's ok. This is the script: https://build.opensuse.org/package/view_file?expand=1&file=gemfile_filter.rb&package=crowbar-barclamp-crowbar&project=home%3Arhafer%3Abranches%3Asystemsmanagement%3Acrowbar%3A2.0 the app's packages.

...

Webyast have some task, that allows to install specific gemfile for given environment - see https://github.com/webyast/webyast/tree/master/webyast part in spec file. Also it is interesting that Gemfile can interpret ruby code, so you can comment it out quite easy.

Also do not forget to remove Gemfile.lock after update of any gem. We solve it with removing Gemfile.lock in init script and during everytime is created new one with fresh versions. Oh, good hint. I didn't think about that.

[..] -- Ralf -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Adam Spiers

10 Jan 10 Jan

10:34

Ralf Haferkamp (rhafer@suse.de) wrote:

...

On Wed, Jan 09, 2013 at 01:37:18PM +0100, Josef Reidinger wrote:

...
On Wed, 9 Jan 2013 13:25:22 +0100 Ralf Haferkamp wrote:

...
Hi,

I currently try to package a rails app as RPM and ran into some issues that don't know howto solve.

The current state of that package can be seen here: https://build.opensuse.org/package/show?package=crowbar-barclamp-crowbar&project=home%3Arhafer%3Abranches%3Asystemsmanagement%3Acrowbar%3A2.0

The app ships with a Gemfile but does not include the Gemfile.lock (upstream has some Good Reason(TM) for that). One of my problems is now that in order to be able to use the app I need to call "bundle install" at some point either during buildtime or after installing package. (Ideally with --local so it will not talk to any remote servers and just use the gem install locally).

Hi Ralf, I describe how webyast and SLMS ( both are rails servers ) do it. Yeah, thanks. That helps a lot.

...
if you have all packages available as RPMS, you do not need to call bundle install. But for resolution you need to move away other kinds. SLMS use groups and do for i in $RPM_BUILD_ROOT/usr/share/slms/*/Gemfile; do sed -i '/^group/,/^end/s/^/#/' $i; done

That comment out all groups, of course you can adapt script to your needs.

Yeah, this is the approach (or at least a similar one) I am currently trying as well. I created a small script that parses the Gemfile an filters out unneeded groups and platforms and generates a new Gemfile from that.

I think upstream have a good reason for not doing this. "bundle help install" says: This is so that installing a different set of groups on another machine (such as a production server) will not change the gems and versions that you have already developed and tested against. Bundler offers a rock-solid guarantee that the third-party code you are running in development and testing is also the third-party code you are running in production. You can choose to exclude some of that code in different environments, but you will never be caught flat-footed by different versions of third-party code being used in different environ- ments. Would it be a lot more pain/effort to ensure that there are gem packages for all the environments? Even if they were covered by BuildRequires for crowbar-barclamp-crowbar, they wouldn't have to go in the final ISO. Although I admit this is getting ugly, and is another argument in favour of ditching the "one gem per rpm" approach in favour of "bundle package" :-/ -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Ralf Haferkamp

11:02

On Thu, Jan 10, 2013 at 10:34:19AM +0000, Adam Spiers wrote:

...

Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Wed, Jan 09, 2013 at 01:37:18PM +0100, Josef Reidinger wrote:

...
On Wed, 9 Jan 2013 13:25:22 +0100 Ralf Haferkamp wrote:

[..]

Yeah, this is the approach (or at least a similar one) I am currently trying as well. I created a small script that parses the Gemfile an filters out unneeded groups and platforms and generates a new Gemfile from that.

I think upstream have a good reason for not doing this. "bundle help install" says:

This is so that installing a different set of groups on another machine (such as a production server) will not change the gems and versions that you have already developed and tested against.

Bundler offers a rock-solid guarantee that the third-party code you are running in development and testing is also the third-party code you are running in production. You can choose to exclude some of that code in different environments, but you will never be caught flat-footed by different versions of third-party code being used in different environ- ments.

Would it be a lot more pain/effort to ensure that there are gem packages for all the environments? Even if they were covered by BuildRequires for crowbar-barclamp-crowbar, they wouldn't have to go in the final ISO. I stopped adding all the gems to BuildRequires, as they are not needed for building the package. As Josef noted bundle install does not need to be called during the build. (It will be implicitly called, when starting the app for the first time though.)

It should however be possible to create "-devel" and "-test" subpackages that just add the dependencies for the respective Gemfile groups. BTW, I created rpm for most of the devel and test dependencies in home:rhafer:branches:systemsmanagement:crowbar:2.0 while experimenting with the packaging.

...

Although I admit this is getting ugly, and is another argument in favour of ditching the "one gem per rpm" approach in favour of "bundle package" :-/ True

-- Ralf -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Adam Spiers

11:52

Ralf Haferkamp (rhafer@suse.de) wrote:

...

On Thu, Jan 10, 2013 at 10:34:19AM +0000, Adam Spiers wrote:

...
Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Wed, Jan 09, 2013 at 01:37:18PM +0100, Josef Reidinger wrote:

...
On Wed, 9 Jan 2013 13:25:22 +0100 Ralf Haferkamp wrote:

[..]

Yeah, this is the approach (or at least a similar one) I am currently trying as well. I created a small script that parses the Gemfile an filters out unneeded groups and platforms and generates a new Gemfile from that.

I think upstream have a good reason for not doing this. "bundle help install" says:

This is so that installing a different set of groups on another machine (such as a production server) will not change the gems and versions that you have already developed and tested against.

Bundler offers a rock-solid guarantee that the third-party code you are running in development and testing is also the third-party code you are running in production. You can choose to exclude some of that code in different environments, but you will never be caught flat-footed by different versions of third-party code being used in different environ- ments.

Would it be a lot more pain/effort to ensure that there are gem packages for all the environments? Even if they were covered by BuildRequires for crowbar-barclamp-crowbar, they wouldn't have to go in the final ISO.

I stopped adding all the gems to BuildRequires, as they are not needed for building the package. As Josef noted bundle install does not need to be called during the build. (It will be implicitly called, when starting the app for the first time though.)

Yeah, but doesn't that break the guarantee described above? which slightly worries me.

...

It should however be possible to create "-devel" and "-test" subpackages that just add the dependencies for the respective Gemfile groups. BTW, I created rpm for most of the devel and test dependencies in home:rhafer:branches:systemsmanagement:crowbar:2.0 while experimenting with the packaging.

Cool, I like that subpackage idea a lot. If it's not a lot of work to finish packaging all the dev/test gems, then Gemfile.lock could be created at build-time and we could preserve the bundler guarantee, which IMHO would be better. I'm not sure where the break-even point is on the cost:reward ratio though. -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Ralf Haferkamp

12:42

On Thu, Jan 10, 2013 at 11:52:24AM +0000, Adam Spiers wrote:

...

Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Thu, Jan 10, 2013 at 10:34:19AM +0000, Adam Spiers wrote:

...
Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Wed, Jan 09, 2013 at 01:37:18PM +0100, Josef Reidinger wrote:

...
On Wed, 9 Jan 2013 13:25:22 +0100 Ralf Haferkamp wrote:

[..]

Yeah, this is the approach (or at least a similar one) I am currently trying as well. I created a small script that parses the Gemfile an filters out unneeded groups and platforms and generates a new Gemfile from that.

I think upstream have a good reason for not doing this. "bundle help install" says:

This is so that installing a different set of groups on another machine (such as a production server) will not change the gems and versions that you have already developed and tested against.

Bundler offers a rock-solid guarantee that the third-party code you are running in development and testing is also the third-party code you are running in production. You can choose to exclude some of that code in different environments, but you will never be caught flat-footed by different versions of third-party code being used in different environ- ments.

Would it be a lot more pain/effort to ensure that there are gem packages for all the environments? Even if they were covered by BuildRequires for crowbar-barclamp-crowbar, they wouldn't have to go in the final ISO.

I stopped adding all the gems to BuildRequires, as they are not needed for building the package. As Josef noted bundle install does not need to be called during the build. (It will be implicitly called, when starting the app for the first time though.)

Yeah, but doesn't that break the guarantee described above? which slightly worries me. Don't we run our tests based on the packages from the ISO (+ probably the update channels). I think that provides a similar guarantee, right?

...

...
It should however be possible to create "-devel" and "-test" subpackages that just add the dependencies for the respective Gemfile groups. BTW, I created rpm for most of the devel and test dependencies in home:rhafer:branches:systemsmanagement:crowbar:2.0 while experimenting with the packaging.

Cool, I like that subpackage idea a lot. If it's not a lot of work to finish packaging all the dev/test gems, then Gemfile.lock could be created at build-time and we could preserve the bundler guarantee, which IMHO would be better. That would bring back a pretty long list of BuildRequires, with all it's drawbacks. (much larger buildroot, longer build times, more rebuilts). Addtionally I think we would still need to fiddle around with the Gemfile, because of the ruby18 (rcov) vs. ruby19 (scov) problems I mentioned above.

...

I'm not sure where the break-even point is on the cost:reward ratio though. Me neither. I would rather not spent much more time on this currently.

-- Ralf -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Ladislav Slezak

12:54

Dne 9.1.2013 16:40, Ralf Haferkamp napsal(a):

...

On Wed, Jan 09, 2013 at 01:37:18PM +0100, Josef Reidinger wrote:

...
On Wed, 9 Jan 2013 13:25:22 +0100 Ralf Haferkamp wrote: [...] Yeah, this is the approach (or at least a similar one) I am currently trying as well. I created a small script that parses the Gemfile an filters out unneeded groups and platforms and generates a new Gemfile from that.

I have a simple rake task for this in Webyast: https://github.com/webyast/webyast/blob/master/webyast/lib/tasks/gemfile.rak... The Gemfiles are created during build in .spec: https://github.com/webyast/webyast/blob/master/webyast/package/webyast-base....

...

...
Webyast have some task, that allows to install specific gemfile for given environment - see https://github.com/webyast/webyast/tree/master/webyast part in spec file.

The additional Gemfiles are used for specific tasks, like this: BUNDLE_GEMFILE=Gemfile.test RAILS_ENV=test rake test

...

...
Also it is interesting that Gemfile can interpret ruby code, so you can comment it out quite easy.

Gemfile is basically a Ruby file which is eval()ed, so you can write any Ruby code there. WebYast uses this feature for finding the installed webyast modules (gems) at runtime. It loads the gems dynamically depending on what's present in the system. https://github.com/webyast/webyast/blob/master/webyast/package/webyast-base.... WebYast also loads some more additional development gems when running from Git checkout (a nice trick, it's convenient for developers and does not affect deployment/RPM build): https://github.com/webyast/webyast/blob/master/webyast/Gemfile#L63 -- Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Ladislav Slezak

13:02

Dne 10.1.2013 12:52, Adam Spiers napsal(a): [...]

...

Cool, I like that subpackage idea a lot. If it's not a lot of work to finish packaging all the dev/test gems, then Gemfile.lock could be created at build-time and we could preserve the bundler guarantee,

Um, you would still need to update Gemfile.lock at runtime. Imagine that we released a maintenance update for a gem with a newer version, like the recent Rails security update... After installing such update your application would not find the version specified in Gemfile.lock and would not start. -- Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Ralf Haferkamp

15:13

On Thu, Jan 10, 2013 at 01:42:22PM +0100, Ralf Haferkamp wrote:

...

On Thu, Jan 10, 2013 at 11:52:24AM +0000, Adam Spiers wrote:

...
Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Thu, Jan 10, 2013 at 10:34:19AM +0000, Adam Spiers wrote:

...
Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Wed, Jan 09, 2013 at 01:37:18PM +0100, Josef Reidinger wrote:

...
On Wed, 9 Jan 2013 13:25:22 +0100 Ralf Haferkamp wrote:

[..] Cool, I like that subpackage idea a lot. If it's not a lot of work to finish packaging all the dev/test gems, then Gemfile.lock could be created at build-time and we could preserve the bundler guarantee, which IMHO would be better. That would bring back a pretty long list of BuildRequires, with all it's drawbacks. (much larger buildroot, longer build times, more rebuilts). Addtionally I think we would still need to fiddle around with the Gemfile, because of the ruby18 (rcov) vs. ruby19 (scov) problems I mentioned above.

...
I'm not sure where the break-even point is on the cost:reward ratio though. Me neither. I would rather not spent much more time on this currently. I've now created -devel and -test subpackages an added the required gem dependencies as "Requires" to them. Currently those packages only contain a specific Gemfile.(test|development). I guess it would make sense to move at least all the test related files into the -test subpackage in the future.

-- Ralf -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Adam Spiers

18:55

Ralf Haferkamp (rhafer@suse.de) wrote:

...

On Thu, Jan 10, 2013 at 11:52:24AM +0000, Adam Spiers wrote:

...
Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Thu, Jan 10, 2013 at 10:34:19AM +0000, Adam Spiers wrote:

...
Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Wed, Jan 09, 2013 at 01:37:18PM +0100, Josef Reidinger wrote:

...
On Wed, 9 Jan 2013 13:25:22 +0100 Ralf Haferkamp wrote: [..]

Yeah, this is the approach (or at least a similar one) I am currently trying as well. I created a small script that parses the Gemfile an filters out unneeded groups and platforms and generates a new Gemfile from that.

I think upstream have a good reason for not doing this. "bundle help install" says:

This is so that installing a different set of groups on another machine (such as a production server) will not change the gems and versions that you have already developed and tested against.

Bundler offers a rock-solid guarantee that the third-party code you are running in development and testing is also the third-party code you are running in production. You can choose to exclude some of that code in different environments, but you will never be caught flat-footed by different versions of third-party code being used in different environ- ments.

Would it be a lot more pain/effort to ensure that there are gem packages for all the environments? Even if they were covered by BuildRequires for crowbar-barclamp-crowbar, they wouldn't have to go in the final ISO.

I stopped adding all the gems to BuildRequires, as they are not needed for building the package. As Josef noted bundle install does not need to be called during the build. (It will be implicitly called, when starting the app for the first time though.)

Yeah, but doesn't that break the guarantee described above? which slightly worries me.

Don't we run our tests based on the packages from the ISO (+ probably the update channels). I think that provides a similar guarantee, right?

Only if we did every single bit of development and testing against packages from ISOs. Enforcing that sounds like a bad idea to me. I'm pretty sure James would agree, at least :)

...

...
...
It should however be possible to create "-devel" and "-test" subpackages that just add the dependencies for the respective Gemfile groups. BTW, I created rpm for most of the devel and test dependencies in home:rhafer:branches:systemsmanagement:crowbar:2.0 while experimenting with the packaging.

Cool, I like that subpackage idea a lot. If it's not a lot of work to finish packaging all the dev/test gems, then Gemfile.lock could be created at build-time and we could preserve the bundler guarantee, which IMHO would be better.

That would bring back a pretty long list of BuildRequires, with all it's drawbacks. (much larger buildroot, longer build times, more rebuilts).

... but only for a single package: crowbar-barclamp-crowbar, which is already a pretty lightweight.

...

Addtionally I think we would still need to fiddle around with the Gemfile, because of the ruby18 (rcov) vs. ruby19 (scov) problems I mentioned above.

Yeah, that gets ugly.

...

...
I'm not sure where the break-even point is on the cost:reward ratio though.

Me neither. I would rather not spent much more time on this currently.

Right, I think we have already wasted a lot of time shoe-horning gems into rpms when they don't naturally fit :-/ -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Adam Spiers

19:11

Ladislav Slezak (lslezak@suse.cz) wrote:

...

Dne 9.1.2013 16:40, Ralf Haferkamp napsal(a):

...
On Wed, Jan 09, 2013 at 01:37:18PM +0100, Josef Reidinger wrote:

...
On Wed, 9 Jan 2013 13:25:22 +0100 Ralf Haferkamp wrote: [...] Yeah, this is the approach (or at least a similar one) I am currently trying as well. I created a small script that parses the Gemfile an filters out unneeded groups and platforms and generates a new Gemfile from that.

I have a simple rake task for this in Webyast:

https://github.com/webyast/webyast/blob/master/webyast/lib/tasks/gemfile.rak...

The Gemfiles are created during build in .spec: https://github.com/webyast/webyast/blob/master/webyast/package/webyast-base....

Thanks for the info!

...

...
...
Also it is interesting that Gemfile can interpret ruby code, so you can comment it out quite easy.

Gemfile is basically a Ruby file which is eval()ed, so you can write any Ruby code there. WebYast uses this feature for finding the installed webyast modules (gems) at runtime. It loads the gems dynamically depending on what's present in the system.

https://github.com/webyast/webyast/blob/master/webyast/package/webyast-base....

Is this link right? It's the same as above.

...

WebYast also loads some more additional development gems when running from Git checkout (a nice trick, it's convenient for developers and does not affect deployment/RPM build):

https://github.com/webyast/webyast/blob/master/webyast/Gemfile#L63

Yeah, that's a cute trick. It still breaks bundler's guarantee which worries me a bit, but I don't know in practice how bad that is. -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Adam Spiers

19:12

Ralf Haferkamp (rhafer@suse.de) wrote:

...

I guess it would make sense to move at least all the test related files into the -test subpackage in the future.

+1 -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Ralf Haferkamp

11 Jan 11 Jan

08:00

Hi, On Thu, Jan 10, 2013 at 06:55:47PM +0000, Adam Spiers wrote:

...

Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Thu, Jan 10, 2013 at 11:52:24AM +0000, Adam Spiers wrote:

...
Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Thu, Jan 10, 2013 at 10:34:19AM +0000, Adam Spiers wrote: [..] I stopped adding all the gems to BuildRequires, as they are not needed for building the package. As Josef noted bundle install does not need to be called during the build. (It will be implicitly called, when starting the app for the first time though.)

Yeah, but doesn't that break the guarantee described above? which slightly worries me.

Don't we run our tests based on the packages from the ISO (+ probably the update channels). I think that provides a similar guarantee, right?

Only if we did every single bit of development and testing against packages from ISOs. Enforcing that sounds like a bad idea to me. I'm pretty sure James would agree, at least :) Not from the ISOs, but probably from the systemsmanagement:crowbar:2.0 channels. If you install all the require gems (not the crowbar-* packages) on your development machine from there before running gem install, you'll get very close (close enough?). Adding those packages to James' development image in Studio should be easy enough. And packaging-wise we are now pretty close to have everything we need in s:c:2, I think.

This is probably getting a little off-topic here now. Should we switch to somewhere else?

...

...
...
...
It should however be possible to create "-devel" and "-test" subpackages that just add the dependencies for the respective Gemfile groups. BTW, I created rpm for most of the devel and test dependencies in home:rhafer:branches:systemsmanagement:crowbar:2.0 while experimenting with the packaging.

Cool, I like that subpackage idea a lot. If it's not a lot of work to finish packaging all the dev/test gems, then Gemfile.lock could be created at build-time and we could preserve the bundler guarantee, which IMHO would be better.

That would bring back a pretty long list of BuildRequires, with all it's drawbacks. (much larger buildroot, longer build times, more rebuilts).

... but only for a single package: crowbar-barclamp-crowbar, which is already a pretty lightweight. But as Ladislav and Josef already mentioned we'd have to regenerate Gemfile.lock on the installed system everytime a dependency get's updated. So generating it during buildtime is pretty much a wasted effort. IMO the only real alternative would be to go down the "bundle package" path.

...

...
Addtionally I think we would still need to fiddle around with the Gemfile, because of the ruby18 (rcov) vs. ruby19 (scov) problems I mentioned above.

Yeah, that gets ugly.

...
...
I'm not sure where the break-even point is on the cost:reward ratio though.

Me neither. I would rather not spent much more time on this currently.

Right, I think we have already wasted a lot of time shoe-horning gems into rpms when they don't naturally fit :-/

-- Ralf -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Josef Reidinger

08:18

On Thu, 10 Jan 2013 19:11:32 +0000 Adam Spiers wrote:

...

Ladislav Slezak (lslezak@suse.cz) wrote:

...
Dne 9.1.2013 16:40, Ralf Haferkamp napsal(a):

...
On Wed, Jan 09, 2013 at 01:37:18PM +0100, Josef Reidinger wrote:

...
On Wed, 9 Jan 2013 13:25:22 +0100 Ralf Haferkamp wrote: [...] Yeah, this is the approach (or at least a similar one) I am currently trying as well. I created a small script that parses the Gemfile an filters out unneeded groups and platforms and generates a new Gemfile from that.

I have a simple rake task for this in Webyast:

https://github.com/webyast/webyast/blob/master/webyast/lib/tasks/gemfile.rak...

The Gemfiles are created during build in .spec: https://github.com/webyast/webyast/blob/master/webyast/package/webyast-base....

Thanks for the info!

...
...
...
Also it is interesting that Gemfile can interpret ruby code, so you can comment it out quite easy.

Gemfile is basically a Ruby file which is eval()ed, so you can write any Ruby code there. WebYast uses this feature for finding the installed webyast modules (gems) at runtime. It loads the gems dynamically depending on what's present in the system.

https://github.com/webyast/webyast/blob/master/webyast/package/webyast-base....

Is this link right? It's the same as above.

...
WebYast also loads some more additional development gems when running from Git checkout (a nice trick, it's convenient for developers and does not affect deployment/RPM build):

https://github.com/webyast/webyast/blob/master/webyast/Gemfile#L63

Yeah, that's a cute trick. It still breaks bundler's guarantee which worries me a bit, but I don't know in practice how bad that is.

Well, there is a lot of notice about this guarantee. I think it is not bad idea, but what I want is ability to switch it off as you need metadata of all gems even those in other groups ( and thats what my patch to bundler does). Reason why I want this is that there is controlled environment where you don't need such guarantee. First example is bundled gems. You control which gems goes there, you just need to load it. Second example is our rpms in enterprise products where we release only versions after testing if it works with products. So what I want is only ability to switch this feature off. Josef -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Adam Spiers

12:32

Josef Reidinger (jreidinger@suse.cz) wrote:

...

On Thu, 10 Jan 2013 19:11:32 +0000 Adam Spiers wrote:

...
Ladislav Slezak (lslezak@suse.cz) wrote:

...
WebYast also loads some more additional development gems when running from Git checkout (a nice trick, it's convenient for developers and does not affect deployment/RPM build):

https://github.com/webyast/webyast/blob/master/webyast/Gemfile#L63

Yeah, that's a cute trick. It still breaks bundler's guarantee which worries me a bit, but I don't know in practice how bad that is.

Well, there is a lot of notice about this guarantee. I think it is not bad idea, but what I want is ability to switch it off as you need metadata of all gems even those in other groups ( and thats what my patch to bundler does). Reason why I want this is that there is controlled environment where you don't need such guarantee.

First example is bundled gems. You control which gems goes there, you just need to load it.

You mean via "bundle package"? In this case, you already have Gemfile.lock, right? If so, that should already have bundler's "seal of approval", so I agree it should be safe to disable the check.

...

Second example is our rpms in enterprise products where we release only versions after testing if it works with products. So what I want is only ability to switch this feature off.

This one seems more dangerous to me, since it doesn't prevent us from developing and testing from source with a different set of gem versions to those used when QA is done later in the cycle. -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Adam Spiers

13:09

Ralf Haferkamp (rhafer@suse.de) wrote:

...

This is probably getting a little off-topic here now. Should we switch to somewhere else?

Yeah, this probably belongs in the ruby-devel internal list, since it's more oriented around SUSE products. I doubt many Rails developers in the openSUSE community rely on gem rpms (but if anyone does, now would be a really good time to speak up!) Cross-posting with Reply-To set ...

...

On Thu, Jan 10, 2013 at 06:55:47PM +0000, Adam Spiers wrote:

...
Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Thu, Jan 10, 2013 at 11:52:24AM +0000, Adam Spiers wrote:

...
Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Thu, Jan 10, 2013 at 10:34:19AM +0000, Adam Spiers wrote: [..] I stopped adding all the gems to BuildRequires, as they are not needed for building the package. As Josef noted bundle install does not need to be called during the build. (It will be implicitly called, when starting the app for the first time though.)

Yeah, but doesn't that break the guarantee described above? which slightly worries me.

Don't we run our tests based on the packages from the ISO (+ probably the update channels). I think that provides a similar guarantee, right?

Only if we did every single bit of development and testing against packages from ISOs. Enforcing that sounds like a bad idea to me. I'm pretty sure James would agree, at least :)

Not from the ISOs, but probably from the systemsmanagement:crowbar:2.0 channels.

I don't like that either - it should be possible to develop and test from source without using packages at all. (If it wasn't, then none of the great work James is currently doing would be possible.) Sure, this limits what you can work on, but it's still important IMHO.

...

...
...
...
...
It should however be possible to create "-devel" and "-test" subpackages that just add the dependencies for the respective Gemfile groups. BTW, I created rpm for most of the devel and test dependencies in home:rhafer:branches:systemsmanagement:crowbar:2.0 while experimenting with the packaging.

Cool, I like that subpackage idea a lot. If it's not a lot of work to finish packaging all the dev/test gems, then Gemfile.lock could be created at build-time and we could preserve the bundler guarantee, which IMHO would be better.

That would bring back a pretty long list of BuildRequires, with all it's drawbacks. (much larger buildroot, longer build times, more rebuilts).

... but only for a single package: crowbar-barclamp-crowbar, which is already a pretty lightweight.

But as Ladislav and Josef already mentioned we'd have to regenerate Gemfile.lock on the installed system everytime a dependency get's updated. So generating it during buildtime is pretty much a wasted effort.

If a dependency gets updated then OBS should automatically rebuild the barclamp anyway, surely? And that would automatically regenerate Gemfile.lock at build-time, assuming that the updated dependency didn't break any of the constraints in Gemfile. And if it did, then we would find out at build-time not install-time, which is good.

...

IMO the only real alternative would be to go down the "bundle package" path.

This looks more and more appealing for every extra minute I spend on this debate :-/ I have a feeling it might eliminate most of the problems we are currently having to worry about. -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Josef Reidinger

13:26

On Fri, 11 Jan 2013 12:32:46 +0000 Adam Spiers wrote:

...

Josef Reidinger (jreidinger@suse.cz) wrote:

...
On Thu, 10 Jan 2013 19:11:32 +0000 Adam Spiers wrote:

...
Ladislav Slezak (lslezak@suse.cz) wrote:

...
WebYast also loads some more additional development gems when running from Git checkout (a nice trick, it's convenient for developers and does not affect deployment/RPM build):

https://github.com/webyast/webyast/blob/master/webyast/Gemfile#L63

Yeah, that's a cute trick. It still breaks bundler's guarantee which worries me a bit, but I don't know in practice how bad that is.

Well, there is a lot of notice about this guarantee. I think it is not bad idea, but what I want is ability to switch it off as you need metadata of all gems even those in other groups ( and thats what my patch to bundler does). Reason why I want this is that there is controlled environment where you don't need such guarantee.

First example is bundled gems. You control which gems goes there, you just need to load it.

You mean via "bundle package"? In this case, you already have Gemfile.lock, right? If so, that should already have bundler's "seal of approval", so I agree it should be safe to disable the check.

...
Second example is our rpms in enterprise products where we release only versions after testing if it works with products. So what I want is only ability to switch this feature off.

This one seems more dangerous to me, since it doesn't prevent us from developing and testing from source with a different set of gem versions to those used when QA is done later in the cycle.

It depends, for webyast and SLMS which I know we run automatic tests in build environment, so it is rebuild and retested automatic with right version of gems. So I don't see much danger here, but of course it require to run tests during build ( that actually for SLMS takes most of its build ). Josef -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Sascha Peilicke

13:30

On 01/11/2013 02:09 PM, Adam Spiers wrote:

...

Ralf Haferkamp (rhafer@suse.de) wrote:

...
This is probably getting a little off-topic here now. Should we switch to somewhere else?

Yeah, this probably belongs in the ruby-devel internal list, since it's more oriented around SUSE products. I doubt many Rails developers in the openSUSE community rely on gem rpms (but if anyone does, now would be a really good time to speak up!)

Well, the OBS developers do at least. Also, openSUSE has countless (generic) tools that build upon those RPMs (like vagrant, chef, puppet). Even as an administrator of a Rails app I can benefit. Instead of constantly checking all upstream gems and see if they fixed a security issue (happened twice recently with Rails), potentially applying patches by hand and having to bundle new gems if that happens plus running "zypper ref" + "zypper up" all the time, I only have to do the latter an be done. Admins appreciate this. As a developer I'm usually fine to create my dev env as messy as I want it. But if I'm sloppy with that, the big hammer will hit once I try to deploy something.

...

Cross-posting with Reply-To set ...

...
On Thu, Jan 10, 2013 at 06:55:47PM +0000, Adam Spiers wrote:

...
Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Thu, Jan 10, 2013 at 11:52:24AM +0000, Adam Spiers wrote:

...
Ralf Haferkamp (rhafer@suse.de) wrote:

...
On Thu, Jan 10, 2013 at 10:34:19AM +0000, Adam Spiers wrote: [..] I stopped adding all the gems to BuildRequires, as they are not needed for building the package. As Josef noted bundle install does not need to be called during the build. (It will be implicitly called, when starting the app for the first time though.)

Yeah, but doesn't that break the guarantee described above? which slightly worries me.

Don't we run our tests based on the packages from the ISO (+ probably the update channels). I think that provides a similar guarantee, right?

Only if we did every single bit of development and testing against packages from ISOs. Enforcing that sounds like a bad idea to me. I'm pretty sure James would agree, at least :)

Not from the ISOs, but probably from the systemsmanagement:crowbar:2.0 channels.

I don't like that either - it should be possible to develop and test from source without using packages at all. (If it wasn't, then none of the great work James is currently doing would be possible.) Sure, this limits what you can work on, but it's still important IMHO.

...
...
...
...
...
It should however be possible to create "-devel" and "-test" subpackages that just add the dependencies for the respective Gemfile groups. BTW, I created rpm for most of the devel and test dependencies in home:rhafer:branches:systemsmanagement:crowbar:2.0 while experimenting with the packaging.

Cool, I like that subpackage idea a lot. If it's not a lot of work to finish packaging all the dev/test gems, then Gemfile.lock could be created at build-time and we could preserve the bundler guarantee, which IMHO would be better.

That would bring back a pretty long list of BuildRequires, with all it's drawbacks. (much larger buildroot, longer build times, more rebuilts).

... but only for a single package: crowbar-barclamp-crowbar, which is already a pretty lightweight.

But as Ladislav and Josef already mentioned we'd have to regenerate Gemfile.lock on the installed system everytime a dependency get's updated. So generating it during buildtime is pretty much a wasted effort.

If a dependency gets updated then OBS should automatically rebuild the barclamp anyway, surely? And that would automatically regenerate Gemfile.lock at build-time, assuming that the updated dependency didn't break any of the constraints in Gemfile. And if it did, then we would find out at build-time not install-time, which is good.

...
IMO the only real alternative would be to go down the "bundle package" path.

This looks more and more appealing for every extra minute I spend on this debate :-/ I have a feeling it might eliminate most of the problems we are currently having to worry about.

You'll get different ones, ask Jordi :-) -- With kind regards, Sascha Peilicke SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)

4130

Age (days ago)

4132

Last active (days ago)

List overview

Download

18 comments

5 participants

participants (5)

Adam Spiers
Josef Reidinger
Ladislav Slezak
Ralf Haferkamp
Sascha Peilicke

[opensuse-ruby] Howto package a Rails App?

tags

participants (5)