Re: [opensuse-ruby] Fwd: [obs submit-request 127033] openSUSE:Factory/puppet-dashboard:, declined by coolo

Sascha Peilicke

17 Jul 2012 17 Jul '12

09:53

Hi guys, From what I got from the discussion so far is that shipping bundled gems in-tree instead of using packages seems to help mostly the packager in that he doesn't have to fiddle with all the masses of ever-changing / don't-look-backwards gems du jour. Studio seems to have made their experiences with that, altough I think ~200 bleeding-edge-n-monkey-patching-everything gems are just a maintenance nightmare in any form, be it packages or bundles gems. OBS broke due to behavior changes even in patch-level gem updates. As we don't have the time to look at every commit in every gem, we are rather conservative with introducing new gem dependencies (or updating old ones). As long as we can upstream gem fixes, I see no real problem with bundled gems (besides our packaging policies maybe). But how about SUSE-specific gem patches that are not upstreamable / rejected? Do you branch gems on github and use those instead of upstream ones? What about SUSE-confidential changes, e.g. for SLE (if there are any)? I think this would remain an issue even if zypper would support gems (which would be great). Using packages certainly has an advantage here. @James: How does Studio handle such situations? However, I don't really see anybody except SUSE products actually using (or more precisely 'caring for') packaged gems for the same reasons we struggle with them. The Ruby community just doesn't care and probably never will, enterprise (or 'stable') isn't a fancy word in the webby world. So we should either bribe Ryan Bates to not present fancy new gems anymore or we do just what suits ourselves. Actually, we (as in coolo, adrian & me) only touch gems/packages if OBS (and maybe software-o-o) needs it. Studio seems to use the same approach when it comes to devel:languages:ruby:extensions. Therefore, openSUSE mostly went the Ruby-1.9 / Rails-3.2 road with packaged gems and doesn't care for anything else anymore. As Studio won't be shipped with / supported by openSUSE any time soon, openSUSE would just gain nothing by shipping Studio's gem dependencies (or even caring for compatibility). Thus openSUSE should retain its current policies. As long as the security and maintenance teams are ok, the Studio team should continue doing whatever suits it and it's customers best. But again, you should really try to keep the fancyness level low. OBS already depends on far (far far far) to many gems (hey, it's Rails and it's supposed to be cool), but 200+ for Studio just seems like screaming madness to me. Linking dynamically or statically to 200 C libraries would be calling for trouble too! BTW. sorry for not replying to the thread, I just subscribed and Thunderbird doesnt' allow to inject "In-Reply-To" in the compose view :-) -- Viele Grüße, Sascha Peilicke

Attachments:

signature.asc (application/pgp-signature — 262 bytes)

Show replies by date

James Tan

17 Jul 17 Jul

12:25

New subject: [opensuse-ruby] Fwd: [obs submit-request 127033] openSUSE:Factory/puppet-dashboard:, declined by coolo

Hi Sascha, Thanks for joining the mailing list and contributing to this discussion! On 07/17/2012 11:53 AM, Sascha Peilicke wrote:

...

From what I got from the discussion so far is that shipping bundled gems in-tree instead of using packages seems to help mostly the packager in that he doesn't have to fiddle with all the masses of ever-changing / don't-look-backwards gems du jour.

Yes this is exactly my (and other Ruby packagers) pain point. The effort of repackaging GEMs into RPMs does not justify benefits that it brings. There are exceptions but we can find solutions to that.

...

Studio seems to have made their experiences with that, altough I think ~200 bleeding-edge-n-monkey-patching-everything gems are just a maintenance nightmare in any form, be it packages or bundles gems.

We don't use a lot of "bleeding-edge-n-monkey-patching-everything gems" in Studio. The Gemfile of ui-server, Studio's largest Rails app, has about 70 gems including those for tests and development. They are all stable versions, and not even necessarily the latest upstream. We only update them when necessary - eg. when there are security or bug fixes which affect us. Or new features that we need. The other Rails apps in the Studio project (there are 5 in total) plus their dependencies brings the total number to 154 (at least on my dev system).

...

As long as we can upstream gem fixes, I see no real problem with bundled gems (besides our packaging policies maybe).

We should always try to upstream gem fixes anyway.

...

But how about SUSE-specific gem patches that are not upstreamable / rejected? Do you branch gems on github and use those instead of upstream ones?

Yes we do this as the worst case scenario. There's unfortunately only a handful.

...

What about SUSE-confidential changes, e.g. for SLE (if there are any)?

I can't think of any examples for this and don't think that it is a problem. If there are, we can run our own GEM server. We do this for Studio Online, but only to make deployments more reliable not because of confidential changes. All our GEM patches are upstream or open-sourced (if we can't get it upstream for whatever reason).

...

I think this would remain an issue even if zypper would support gems (which would be great). Using packages certainly has an advantage here.

But GEMs are also packages.

...

@James: How does Studio handle such situations?

As mentioned above. Let me know if I missed anything.

...

However, I don't really see anybody except SUSE products actually using (or more precisely 'caring for') packaged gems for the same reasons we struggle with them. The Ruby community just doesn't care and probably never will, enterprise (or 'stable') isn't a fancy word in the webby world. So we should either bribe Ryan Bates to not present fancy new gems anymore or we do just what suits ourselves.

Actually, we (as in coolo, adrian& me) only touch gems/packages if OBS (and maybe software-o-o) needs it. Studio seems to use the same approach when it comes to devel:languages:ruby:extensions. Therefore, openSUSE mostly went the Ruby-1.9 / Rails-3.2 road with packaged gems and doesn't care for anything else anymore. As Studio won't be shipped with / supported by openSUSE any time soon, openSUSE would just gain nothing by shipping Studio's gem dependencies (or even caring for compatibility). Thus openSUSE should retain its current policies.

I only used Studio as an example of the general rubygem packaging problem. I'm not interested in getting openSUSE to ship Studio's gem dependencies. As indicated in the email subject and the original emails, the request is to allow Ruby applications to bundle gems in openSUSE so that it is easy to package apps like puppet-dashboard (nothing to do with Studio). Puppet upstream is distributing their apps this way (bundling), same with Chef. So why do we want to create extra work by forcing packagers to extract these and repackage them? I did this for Chef and ended up with ~50 rubygem packages [1]. Is it worth the time and effort? What do we gain from doing this? This is the discussion that I would like to have. [1] https://build.opensuse.org/project/monitor?project=systemsmanagement%3Achef

...

As long as the security and maintenance teams are ok, the Studio team should continue doing whatever suits it and it's customers best.

Sure and as mentioned above, this discussion has nothing to do that.

...

But again, you should really try to keep the fancyness level low. OBS already depends on far (far far far) to many gems (hey, it's Rails and it's supposed to be cool), but 200+ for Studio just seems like screaming madness to me. Linking dynamically or statically to 200 C libraries would be calling for trouble too!

Also explained above, we do try to keep the dependencies low but it's a big project that touches many areas. I can run through the Gemfiles with you and maybe you can help us identify what we can remove. And once again, this discussion is about packaging Ruby applications in general, specifically Puppet and Chef so let's stick to that. Cheers, James T. -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Sascha Peilicke

12:59

New subject: [opensuse-ruby] Fwd: [obs submit-request 127033] openSUSE:Factory/puppet-dashboard:, declined by coolo

On 07/17/2012 02:25 PM, James Tan wrote:

...

Hi Sascha,

[snip'e'ty'snip]

I only used Studio as an example of the general rubygem packaging problem. I'm not interested in getting openSUSE to ship Studio's gem dependencies.

As indicated in the email subject and the original emails, the request is to allow Ruby applications to bundle gems in openSUSE so that it is easy to package apps like puppet-dashboard (nothing to do with Studio). Ok, I was just (ab)using it as an example too. Let's talk about puppet and chef instead ;-)

...

Puppet upstream is distributing their apps this way (bundling), same with Chef. Same with OBS and other Rails apps.

...

So why do we want to create extra work by forcing packagers to extract these and repackage them? I did this for Chef and ended up with ~50 rubygem packages [1].

...

Is it worth the time and effort? Usually gems are packaged with gem2rpm, which takes really little time and effort (unless you have to fix the gem somehow), so I don't think

From what I get from the project is that actually all of these are already found in devel:languages:ruby:extensions, so instead of aggregating each and every gem there, you could also simply add d:l:r:e as an additional repository path (ugly naming) for the repositories you build against. For instance, check your project _meta XML (https://build.opensuse.org/project/meta?project=systemsmanagement%3Achef) and modifie each repository entry accordingly: <repository name="$REPO"> ... <path project="devel:languages:ruby:extensions" repository="$REPO"/> ... </repository> This way you _also_ build against d:l:r:e when you build against, say, openSUSE:Factory. BTW. this is what you get when you setup a branch. So the easy way would have been to just submit IBS: Devel:Cloud / rubygem-chef (and co) to OBS: d:l:r:e and be done. that's an issue. Also, your friendly OBS team was really busy keeping d:l:r:e in shape, so most of those gems are already there.

...

What do we gain from doing this? This is the discussion that I would like to have.

The ability to add suse-specific patches (i.e. that are not upstreamable). Like stuff shipping it's own SSL certificates but we want it to use the distro ones. The ability to fix an issue in one place. Consider a security issue in any gem and that gem is bundled in $COUNT Ruby apps we ship in openSUSE. With bundled gems we would have to identify those apps, check the versions that are shipped and see if they're affected. If so, you would have to fix the gem and upstream it, fix each app and submit there, then fix each Ruby app package and submit to openSUSE. A packaged gem is the only place where a fix would have to be done. Sort of a trade-off between the initial packaging work and the maintenance afterwards. The only remaining issue with packaged gems are incompatibilities between the required gem versions of various software components. The packaging answer to that is indeed a bit more burdensome. It includes postfixing package names with versions and possibly the use of update-alternatives. But most of this work has been done for the more important gems in d:l:r:e already. So if puppet or chef would need different gem versions than those in d:l:r:e, I'm sure we can fix that easily.

...

[1] https://build.opensuse.org/project/monitor?project=systemsmanagement%3Achef

...
As long as the security and maintenance teams are ok, the Studio team should continue doing whatever suits it and it's customers best.

Sure and as mentioned above, this discussion has nothing to do that. Well, sort of. There's a lot of talented Ruby developers in the Studio team of which only some appear to contribute to d:l:r:e. It would be more guys would be willing to help out. As a first step to get you issue solved, you could prepare a list of gems that puppet/chef/studio need and are not or only found in different versions in d:l:r:e. -- Viele Grüße, Sascha Peilicke

James Tan

16:39

New subject: [opensuse-ruby] Fwd: [obs submit-request 127033] openSUSE:Factory/puppet-dashboard:, declined by coolo

On 07/17/2012 02:59 PM, Sascha Peilicke wrote:

...

On 07/17/2012 02:25 PM, James Tan wrote:

...
So why do we want to create extra work by forcing packagers to extract these and repackage them? I did this for Chef and ended up with ~50 rubygem packages [1]. From what I get from the project is that actually all of these are already found in devel:languages:ruby:extensions,

All the rubygem packages in systemsmanagement:chef and their dependencies (like gecode) are aggregated from devel:languages:ruby:extensions because I wanted it to be easy for users to install Chef - eg. just add the Chef repo and install. It also makes the list of dependencies obvious. I had to add and modify 22 packages in d:l:r:e to get Chef to build.

...

so instead of aggregating each and every gem there, you could also simply add d:l:r:e as an additional repository path (ugly naming) for the repositories you build against. For instance, check your project _meta XML (https://build.opensuse.org/project/meta?project=systemsmanagement%3Achef) and modifie each repository entry accordingly:

<repository name="$REPO"> ... <path project="devel:languages:ruby:extensions" repository="$REPO"/> ... </repository>

This way you _also_ build against d:l:r:e when you build against, say, openSUSE:Factory. BTW. this is what you get when you setup a branch. So the easy way would have been to just submit IBS: Devel:Cloud / rubygem-chef (and co) to OBS: d:l:r:e and be done.

Thanks for the suggestion - I might just switch to that, at least for the devel project (home:jatan11:chef-devel).

...

...
Is it worth the time and effort? Usually gems are packaged with gem2rpm, which takes really little time and effort (unless you have to fix the gem somehow), so I don't think that's an issue. Also, your friendly OBS team was really busy keeping d:l:r:e in shape, so most of those gems are already there.

Yup I used that script a lot and also thanks guys for keeping d:lr:e in shape. But it's still a lot of repetitive work made worse by the "~>" operator in the gemspec. I had to add multiple packages (eg. rubygem-json, rubygem-json-1_5), update project configuration to tell it which one to prefer, update rubygem spec files to add missing "Provides" symbols, etc.

...

...
What do we gain from doing this? This is the discussion that I would like to have. The ability to add suse-specific patches (i.e. that are not upstreamable). Like stuff shipping it's own SSL certificates but we want it to use the distro ones.

Do you have some concrete examples? In general I think we should avoid patching gems at all. It makes maintenance much harder down the road.

...

The ability to fix an issue in one place. Consider a security issue in any gem and that gem is bundled in $COUNT Ruby apps we ship in openSUSE. With bundled gems we would have to identify those apps, check the versions that are shipped and see if they're affected. If so, you would have to fix the gem and upstream it, fix each app and submit there, then fix each Ruby app package and submit to openSUSE. A packaged gem is the only place where a fix would have to be done. Sort of a trade-off between the initial packaging work and the maintenance afterwards.

I agree this is one of the trade-offs, but overall I'm pretty sure it's less work to bundle the gems. How many Rails apps do we have in openSUSE now? You need to test the application anyway with the updated gems, rather than just update the dependencies and hope that things still work.

...

The only remaining issue with packaged gems are incompatibilities between the required gem versions of various software components. The packaging answer to that is indeed a bit more burdensome. It includes postfixing package names with versions and possibly the use of update-alternatives. But most of this work has been done for the more important gems in d:l:r:e already.

So if puppet or chef would need different gem versions than those in d:l:r:e, I'm sure we can fix that easily.

Thanks for the hard work in getting things to work. And yes we can fix these issues but it is just a lot of unnecessary work from my perspective.

...

...
...
As long as the security and maintenance teams are ok, the Studio team should continue doing whatever suits it and it's customers best. Sure and as mentioned above, this discussion has nothing to do that. Well, sort of. There's a lot of talented Ruby developers in the Studio team of which only some appear to contribute to d:l:r:e. It would be more guys would be willing to help out.

Few contribute because they don't see the value of doing it. I don't either that's why I'm hoping to understand why we are doing this in the first place and if there is some way to improve things. Currently I think that adding gem support to zypper would be a nice way to solve it, but of course there are some issues (eg. need to install devel packages for gems that have C code) that we have to figure out. That might not fit all use cases but IMO it does make many much nicer.

...

As a first step to get you issue solved, you could prepare a list of gems that puppet/chef/studio need and are not or only found in different versions in d:l:r:e.

As mentioned above, I've already added and fixed those that I need in the past week or so. Cheers, James T. -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Stephan Kulow

13:35

New subject: [opensuse-ruby] Fwd: [obs submit-request 127033] openSUSE:Factory/puppet-dashboard:, declined by coolo

On 17.07.2012 14:25, James Tan wrote:

...

Puppet upstream is distributing their apps this way (bundling), same with Chef. So why do we want to create extra work by forcing packagers to extract these and repackage them? I did this for Chef and ended up with ~50 rubygem packages [1]. Is it worth the time and effort? What do we gain from doing this? This is the discussion that I would like to have.

It makes sense for upstream to do that, so you can get it up and running quickly. But as a distribution, it does not make sense to bundle everything with everything. After all, it's appliances makers that cry for "make it smaller, share more" all the time. And after that we defined our policies. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

James Tan

16:21

New subject: [opensuse-ruby] Fwd: [obs submit-request 127033] openSUSE:Factory/puppet-dashboard:, declined by coolo

On 07/17/2012 03:35 PM, Stephan Kulow wrote:

...

On 17.07.2012 14:25, James Tan wrote:

...
Puppet upstream is distributing their apps this way (bundling), same with Chef. So why do we want to create extra work by forcing packagers to extract these and repackage them? I did this for Chef and ended up with ~50 rubygem packages [1]. Is it worth the time and effort? What do we gain from doing this? This is the discussion that I would like to have.

It makes sense for upstream to do that, so you can get it up and running quickly. But as a distribution, it does not make sense to bundle everything with everything. After all, it's appliances makers that cry for "make it smaller, share more" all the time. And after that we defined our policies.

Fair point about the size, but from my perspective the extra 10s of MB is negligible in this context. Disk is cheap. Rails application deployment best practices usually involve running "bundle install --deployment" so that all the gem dependencies are cached in vendor/bundle for each deployment. This trades disk space for self-contained applications and easier roll-back. Cheers, James T. -- To unsubscribe, e-mail: opensuse-ruby+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-ruby+owner@opensuse.org

Sascha Peilicke

13:40

New subject: [opensuse-ruby] Fwd: [obs submit-request 127033] openSUSE:Factory/puppet-dashboard:, declined by coolo

On 07/17/2012 02:25 PM, James Tan wrote:

...

But GEMs are also packages.

But shipping bundled gems breaks when they have arch-dependent (speak C code) in it. You would either have to build/compile the gem for all architectures you target (with some %if/%else mumbo-jumbo in the spec file of the app using/shipping the gem). You could also do the gem fetching/building in the %post section of your app's spec file to get around that, but that's also tedious. -- Viele Grüße, Sascha Peilicke

Josef Reidinger

13:47

New subject: [opensuse-ruby] Fwd: [obs submit-request 127033] openSUSE:Factory/puppet-dashboard:, declined by coolo

On Tue, 17 Jul 2012 15:40:46 +0200 Sascha Peilicke wrote:

...

On 07/17/2012 02:25 PM, James Tan wrote:

...
But GEMs are also packages.

But shipping bundled gems breaks when they have arch-dependent (speak C code) in it.

You would either have to build/compile the gem for all architectures you target (with some %if/%else mumbo-jumbo in the spec file of the app using/shipping the gem).

You could also do the gem fetching/building in the %post section of your app's spec file to get around that, but that's also tedious.

Thats not so horrible. Just your product is arch dependent and you compile bindings during build. Usually appliance has attached gems and it is installed during deploy ( build phase in OBS ). Josef

Sascha Peilicke

14:31

New subject: [opensuse-ruby] Fwd: [obs submit-request 127033] openSUSE:Factory/puppet-dashboard:, declined by coolo

On 07/17/2012 03:47 PM, Josef Reidinger wrote:

...

On Tue, 17 Jul 2012 15:40:46 +0200 Sascha Peilicke wrote:

...
On 07/17/2012 02:25 PM, James Tan wrote:

...
But GEMs are also packages.

But shipping bundled gems breaks when they have arch-dependent (speak C code) in it.

You would either have to build/compile the gem for all architectures you target (with some %if/%else mumbo-jumbo in the spec file of the app using/shipping the gem).

You could also do the gem fetching/building in the %post section of your app's spec file to get around that, but that's also tedious.

Thats not so horrible. Just your product is arch dependent and you compile bindings during build. That makes sense, yes.

I was more refering to Ruby applications that already ship bundled gems as part of their source tree. But I may get something wrong here and that this isn't the case.

...

Usually appliance has attached gems and it is installed during deploy ( build phase in OBS ). Josef

-- Viele Grüße, Sascha Peilicke

4311

Age (days ago)

4311

Last active (days ago)

List overview

Download

8 comments

4 participants

participants (4)

James Tan
Josef Reidinger
Sascha Peilicke
Stephan Kulow