New packaging guidelines for Ruby

Description

Hi,
we would like to have have our new Ruby packaging guidelines reviewed and approved as a part of migration to Ruby 1.9.3 [1], which will incorporate some changes in the way of Ruby packaging. The proposed draft is at [2].

Change History

I think you have a mistake in the definition of %{gem_instdir}, it says "%{gem_dir}/gems/%{gem_name}-%{version}", but that would eval out to "/usr/share/gems/gems/%{gem_name}-%{version}". Perhaps it should be "%{gem_dir}/%{gem_name}-%{version}" ?

To give some background info:
RubyGems? are generally placed under a directory with following subdirectories:

cache - a directory with cached versions of Gems (a single file named %{gem_name}-%{version}.gem for each Gem) - these are only used for "gem pristine" command, which restores the Gem to its original state (if user modifies it and does something wrong, he can use this) - obviously, this can be replaced by "yum reinstall" in RPM versions of Gems.

doc - directory with documentation, each Gem has its own subdirectory called %{gem_name}-%{version}

The top directory of the Gem structure is /usr/share/gems - this is done because we are planning to introduce multiple Ruby runtimes (JRuby, Rubinius) and the Gems should be available independently on the Ruby implementation installed on user's computer (therefore /usr/share/ruby-1.9.3/ would not be a good name for the top level directory).

Since the top directory of the Gem structure is /usr/share/gems and the directory structure under it reflects a standard Gem directory (as explained above), the source code of each Gem is placed under /usr/share/gems/gems/%{gem_name}-%{version} == %{gem_dir}/gems/%{gem_name}-%{version}.

Also, as explained above, the %{gem_cache} expands into /usr/share/gems/cache/%{gem_name}-%{version}.gem, which is a single file - the cached Gem.

Why are we deprecating packaging gems for non-gem use? One of the things we looked at originally was how to not have to force packagers and end users to patch upstream ruby application and script code in order to use libraries which we had only packaged as gems. Does ruby no longer have the former methods of importing libraries? Everything must be done via gems now?

I don't see %{ruby_vendorlibdir} defined anywhere in the draft. Is that a %{_libdir} directory or a /usr/lib directory?

It looks like you have some instances of %{gemdir} ; are those supposed to be %{gem_dir} ?

A good addition to this guideline would be how to deal with having to apply patches to a gem. If I'm reading this right, for the pure ruby case, the patch can be applied directly to the source that was created with gem install in %prep. It's unclear what to do when patching something that has C extensions.

One more item: It looks like the gem_install block (for both noarch and arch gems) should be fine to call in %build (where it would be more appropriate) instead of %prep. Is there any reason that this is impossible or impractical?

One more item: It looks like the gem_install block (for both noarch and arch gems) should be fine to call in %build (where it would be more appropriate) instead of %prep. Is there any reason that this is impossible or impractical?

We also analyzed this a bit further and realized that the %setup line isn't extracting the sources. The source isn't extracted until the "gem install" command is run. However, we may come back to this once we know what gem commands need to be run to patch rubygems that have C extensions/ if there's some alternate commands that allow us to extract the gem in %prep and then build it/install it to the proper looking hierarchy in a later step.

Why are we deprecating packaging gems for non-gem use? One of the things we looked at originally was how to not have to force packagers and end users to patch upstream ruby application and script code in order to use libraries which we had only packaged as gems. Does ruby no longer have the former methods of importing libraries? Everything must be done via gems now?

Since Ruby now requires Rubygems library automatically, there is no need for non-gem subpackages. Moreover, Ruby core programmers want to gemify standard library [1], so the Gems are pretty much a standard in Ruby world and no one uses Ruby without Rubygems.

I don't see %{ruby_vendorlibdir} defined anywhere in the draft. Is that a %{_libdir} directory or a /usr/lib directory?

It looks like you have some instances of %{gemdir} ; are those supposed to be %{gem_dir} ?

You are right, I fixed this.

A good addition to this guideline would be how to deal with having to apply patches to a gem. If I'm reading this right, for the pure ruby case, the patch can be applied directly to the source that was created with gem install in %prep. It's unclear what to do when patching something that has C extensions.

For the pure Ruby case, you are right, for binary extensions, you need to use gem install, apply the patch and then recompile. You are probably right, that we should create an example that would show how to do this.

One more item: It looks like the gem_install block (for both noarch and arch gems) should be fine to call in %build (where it would be more appropriate) instead of %prep. Is there any reason that this is impossible or impractical?

AFAIK the %patch macros only work in the %prep section, so we would need to apply the patches manually in %build - so it would be impractical.

One more item: It looks like the gem_install block (for both noarch and arch gems) should be fine to call in %build (where it would be more appropriate) instead of %prep. Is there any reason that this is impossible or impractical?

We also analyzed this a bit further and realized that the %setup line isn't extracting the sources. The source isn't extracted until the "gem install" command is run.However, we may come back to this once we know what gem commands need to be run to patch rubygems that have C extensions/ if there's some alternate commands that allow us to extract the gem in %prep and then build it/install it to the proper looking hierarchy in a later step.

One more item: It looks like the gem_install block (for both noarch and arch gems) should be fine to call in %build (where it would be more appropriate) instead of %prep. Is there any reason that this is impossible or impractical?

AFAIK the %patch macros only work in the %prep section, so we would need to apply the patches manually in %build - so it would be impractical.

Or you can call it in %install section ;). We chosen %prep, so in the case you need to apply patch later, you don't have to change "whole" spec.

It makes much sense, so we added it later on your request and that might confused you.

To be honest, there is still missing one scenario: if you cannot install the gem, because the binary extension is incompatible with the Ruby. And that situation is not well supported. I tried to ask upstream once and I got this response [1]. On the other hand, after rebuild of almost all RubyGems? because of Ruby 1.9.3, I am not aware yet of any gem which would need this treatment (although there are signs that rubygem-idn might be first candidate).

It makes much sense, so we added it later on your request and that might confused you.

To be honest, there is still missing one scenario: if you cannot install the gem, because the binary extension is incompatible with the Ruby. And that situation is not well supported. I tried to ask upstream once and I got this response [1]. On the other hand, after rebuild of almost all RubyGems? because of Ruby 1.9.3, I am not aware yet of any gem which would need this treatment (although there are signs that rubygem-idn might be first candidate).

Unfortunately, this is not correct in that the syntax require ('rubygem') does not work. Also, I'm not sure from some of what I've read in the above comments if what's being said in the comments above is that rubygem is being shipped with ruby now or rubygem is being require'd now (and thus, the ruby path is being setup transparently).

I'm going to go through and reorganize the draft now to match other guidelines.

It's a shame that we couldn't have looked at normalizing the site/vendor macros with other languages as well. python, perl, etc have LANG_sitearch -- only ruby has LANG_sitearchdir.

Is there a reason to include instructions for Packaging for Gem and non-Gem use at all? If you can do a simple require 'ruby package that may or may not be a gem' now then it seems like we don't have any reason to package old packages in non-gem format now. We can strip that section to just the disclaimer.

This simplifies the guidelines for packagers and reviewers as there's only one path to follow (no decisions for binary vs noarch; patching vs non-patching.)

This properly breaks up the steps for building the package into %prep, %build, and %install which has been a longstanding issue with the guidelines. One benefit of breaking it up like this is rpmbuild [-bi | -bc] --short-circuit will finally work as expected with rubygems

Why was the rubygems package moved from {site,vendor}libdir into /usr/share/rubygems?

Okay... I see the note in the spec file for this which says that you want the rubygem module to be shared between different implementations of the ruby interpreter (for instance, jruby -- which currently bundles its own copy of rubygems since rubygems is also in the ruby stdlib). IIUC, though, with the new rubygem guidelines, the idea is that rubygems can be shared between different ruby implementations. If that's correct, then shouldn't the same be done with vendorlib rather than special-casing just rubygems?

I've been working on the guideline this morning -- structural changes, I just went ahead and did. Many of the changes I mentioned as being possibly desirable here, I marked in the new page with an admon/question template to either confirm or to think about replacing some section with another that I've written.

I've only gotten to the gem building section so far. Need to merge the rest of the gem building and patching sections with it and then see if we can replace all those other sections with the one I wrote.

I'm abadger1999 on irc.freenode.net if anyone wants to give me guidance on this stuff. I'll be in and out this weekend (I'm out of time for today) but the packaging meeting is on Wednesday so it would be good to catch me before the meeting next week.

Packaging for Gem and non-Gem use - There were suggestions to completely remove the section. However, there are a few packages which provides also nongem packages for Gems. Therefore, I think we should have it in guidelines mentioned as deprecated practice and prohibit it for new reviewed packages, but if you believe that it is just confusing, then I am fine with removal of this chapter.

Site/vendor macros - Ruby itself is using sitelibdir, sitearchdir and similarly for vendor, also, in previous guidelines there were typically used gemdir and quite common, there was used macro gemsintdir. That is why we went with the dir suffix. Not sure if and how to normalize it with other languages. Please note that we also prefer vendor instead of site, which I believe is preferred by other languages, although it is wrong IMO.

Guidelines for R1.9 - You are right that the new guidelines target just the Ruby 1.9. If we could keep copy of the old guidelines somewhere around, this would be nice. On the other hand, for example the macros we are using now, which are provided by rubygems-devel were backported by mtasaka also for Fedoras < F17, so they may be used also in older Fedoras. You can take look on rubygem-nokogiri's spec file [1] for example.

Binary Extension Fails to Build - I strongly disagree to follow this path always. As I said, from more than 200 gems, I know just about one which needs this procedure. Moreover, it is not as straight forward as it might look. Just for the one gem I know, I needed to tweak this approach [2]. So suggest to do something what doesn't work universally and is rarely used makes no sense to me. Previously, a lot of gems were using just %install section, now we suggest to always use also the %prep section, because it was widely used and if not in time of review, then some time later, but the %build section is typically useless.

/usr/share/rubygems - Firs of all, you are right that the idea is to share RubyGems? library and the Gems between implementation. On the other hand, it is in theoretical level yet. Nevertheless JRuby currently breaks packaging guidelines and I doubt there were approved exception. Regarding the vendorlib, there are several things I'd like to point out: (1) We try to limit usage of vendorlib as much as possible. I don't like to see any new packages to be reviewed which goes to vendor lib. That is not the way which Ruby development goes, at least since the RubyGems? become standard part of Ruby. There is just a few, old and typically unmaintained libraries which are not provided as a Gem. (2) Rubygems provides mechanisms, how to install more Gems of some version for different platforms in parallel, e.g. if you install nokogiri, it ends up in nokogiri-1.5.0 for x86 and nokogiri-1.5.0-java for jruby. There is no mechanism how to achieve it for vendorlib. This might work for noarch packages, but will bring us issues for arch dependent libraries.

Site/vendor macros - Ruby itself is using sitelibdir, sitearchdir and similarly for vendor>

python doesn't have the concept of vendorlib. perl is using perl_vendorlib. What do you mean by RUby itself is using sitelibdir and sitearchdir? Are those variables available as public API to ruby somehow? If so, I would agree that that would be a valid thing to sync against rather than other language's macros.

Guidelines for R1.9

I've added an admonition link to the top of the new guidelines for older Fedora and RHEL. If you also want to work on updating the old guidelines, you can... but my recommendation is that there'd be little benefit for the work and you'd have to check whether the changes work on both older Fedora and RHEL. So I probably wouldn;t invest the time myself.

Binary Extension Fails to Build - I strongly disagree to follow this path always. As I said, from more than 200 gems, I know just about one which needs this procedure. Moreover, it is not as straight forward as it might look. Just for the one gem I know, I needed to tweak this approach [2].

/usr/share/rubygems - Firs of all, you are right that the idea is to share RubyGems? library and the Gems between implementation. On the other hand, it is in theoretical level yet. Nevertheless JRuby currently breaks packaging guidelines and I doubt there were approved exception. Regarding the vendorlib, there are several things I'd like to point out: (1) We try to limit usage of vendorlib as much as possible. I don't like to see any new packages to be reviewed which goes to vendor lib. That is not the way which Ruby development goes, at least since the RubyGems? become standard part of Ruby. There is just a few, old and typically unmaintained libraries which are not provided as a Gem. (2) Rubygems provides mechanisms, how to install more Gems of some version for different platforms in parallel, e.g. if you install nokogiri, it ends up in nokogiri-1.5.0 for x86 and nokogiri-1.5.0-java for jruby. There is no mechanism how to achieve it for vendorlib. This might work for noarch packages, but will bring us issues for arch dependent libraries.

I don't see an actual argument for why vendorlib/sitelib shouldn't be interpreter independent if gems are independent in the first half (through #1). For #2, I'm not sure what you're driving at. This is what I'm getting out of it: nokogiri builds on all archs. It creates a nokogiri-1.5.0.x86 w/ nokogiri-jruby-1.5.0.noarch and a nokogiri-1.5.0.x86_64 w/ nokogiri-jruby-1.5.0.noarch. I'm assuming that the next step is the problem?

The problem that I'm seeing is that the rubygems package itself is currently being special-cased for no apparent reason. It doesn't seem like there's any reason that all of vendorlib, not just rubygems, can't be shared between ruby interpreters if gems are already shared and rubygems itself can be shared.

Site/vendor macros - Ruby itself is using sitelibdir, sitearchdir and similarly for vendor>

python doesn't have the concept of vendorlib. perl is using perl_vendorlib. What do you mean by RUby itself is using sitelibdir and sitearchdir? Are those variables available as public API to ruby somehow? If so, I would agree that that would be a valid thing to sync against rather than other language's macros.

See the previous version of guidelines [1], e.g. "puts Config::CONFIG[ 'sitelibdir' ]"

Guidelines for R1.9

I've added an admonition link to the top of the new guidelines for older Fedora and RHEL. If you also want to work on updating the old guidelines, you can... but my recommendation is that there'd be little benefit for the work and you'd have to check whether the changes work on both older Fedora and RHEL. So I probably wouldn;t invest the time myself.

No, I don't want to :)

Binary Extension Fails to Build - I strongly disagree to follow this path always. As I said, from more than 200 gems, I know just about one which needs this procedure. Moreover, it is not as straight forward as it might look. Just for the one gem I know, I needed to tweak this approach [2].

You are missing the "sed" line. This is what may bite somebody. I had to find solution how to make the "gem build" actually work. I really don't like to fight with this kind of issues. Moreover, if you are packaging such gem, it should warn you, that something is really bad. Why it does not work? The rubygem-idn does not work, because it is not compatible with Ruby 1.9.3 nor maintained by upstream. Do we really want to support such unmaintained libraries? My answer is NO, with some exceptions. You are asking for more troubles than you solve. This is workaround if you are in troubles, not a standard supported solution.

/usr/share/rubygems - Firs of all, you are right that the idea is to share RubyGems? library and the Gems between implementation. On the other hand, it is in theoretical level yet. Nevertheless JRuby currently breaks packaging guidelines and I doubt there were approved exception. Regarding the vendorlib, there are several things I'd like to point out: (1) We try to limit usage of vendorlib as much as possible. I don't like to see any new packages to be reviewed which goes to vendor lib. That is not the way which Ruby development goes, at least since the RubyGems? become standard part of Ruby. There is just a few, old and typically unmaintained libraries which are not provided as a Gem. (2) Rubygems provides mechanisms, how to install more Gems of some version for different platforms in parallel, e.g. if you install nokogiri, it ends up in nokogiri-1.5.0 for x86 and nokogiri-1.5.0-java for jruby. There is no mechanism how to achieve it for vendorlib. This might work for noarch packages, but will bring us issues for arch dependent libraries.

I don't see an actual argument for why vendorlib/sitelib shouldn't be interpreter independent if gems are independent in the first half (through #1). For #2, I'm not sure what you're driving at. This is what I'm getting out of it: nokogiri builds on all archs. It creates a nokogiri-1.5.0.x86 w/ nokogiri-jruby-1.5.0.noarch and a nokogiri-1.5.0.x86_64 w/ nokogiri-jruby-1.5.0.noarch. I'm assuming that the next step is the problem?

The problem that I'm seeing is that the rubygems package itself is currently being special-cased for no apparent reason. It doesn't seem like there's any reason that all of vendorlib, not just rubygems, can't be shared between ruby interpreters if gems are already shared and rubygems itself can be shared.

Example might help here. If you are using plain gems, you can use "--platform" parameter to specify the gem for which platform you want to install. In this case RubyGems? will use following structure to prevent some collisions:

Therefore you can install jruby packages together with mri packages into one gems tree. RubyGems? later pickup the proper gem for the interpreter, depending on the platform. You are not able to achieve this in vendorlib, because it was never thought of.

Moreover, at these days, there really is not plethora of libraries which are not provided as gems. Take a look on ruby- packages in Fedora and you'll see that there is typically available its gem equivalent. I am working with Ruby more than 6 years now and I don't remember any Ruby library I would ever need to install into site/vendor libdir (except RubyGems? itsefl).

ruby vs rubygems provides - I like the idea to have just ruby() provide, however I am afraid that it is too late for F17 and I am not sure about all possible implications. There are some for sure (1) There is no way how to prioritize rubygem- over ruby- packages. (2) There are other ruby libraries, which can take your gems and vendorize them into application you are developing, which is definitely not possible if the you have ruby- library instead of rubygem- (3) Rubygems has metadata, if you require one gem, it loads also the gem's dependencies. (4) You proposed to have provide like ruby(requiredfilename), however there are libraries, where you need to require them using "require 'foo/bar'". Typical example is the rubygem-net-* family. There might be others (may ber rubygem-rack-* ?). And I could continue. I propose to revert the provides to ruby(libraryname)/rubygems(gem_name) and postpone it, lets say for F19?

Site/vendor macros - Ruby itself is using sitelibdir, sitearchdir and similarly for vendor>

See the previous version of guidelines [1], e.g. "puts Config::CONFIG[ 'sitelibdir' ]"

Okay, taken out.

Binary Extension Fails to Build - I strongly disagree to follow this path always. As I said, from more than 200 gems, I know just about one which needs this procedure. Moreover, it is not as straight forward as it might look. Just for the one gem I know, I needed to tweak this approach [2].

You are missing the "sed" line. This is what may bite somebody. I had to find solution how to make the "gem build" actually work. I really don't like to fight with this kind of issues. Moreover, if you are packaging such gem, it should warn you, that something is really bad. Why it does not work? The rubygem-idn does not work, because it is not compatible with Ruby 1.9.3 nor maintained by upstream. Do we really want to support such unmaintained libraries? My answer is NO, with some exceptions. You are asking for more troubles than you solve. This is workaround if you are in troubles, not a standard supported solution.

Okay -- looking at this, it seems like a case where upstream's code is wrong. So fixing it from our spec file and submitting a bug report upstream should upstream be active looks like exactly the right thing to do.

/usr/share/rubygems

Your example tree is making me even more cautious... You're thinking that that tree is going to show up in %gem_dir, not in %gem_extdir correct? And it's only going to show up for ruby gems that have extensions (not pure ruby)?

Moreover, at these days, there really is not plethora of libraries which are not provided as gems. Take a look on ruby- packages in Fedora and you'll see that there is typically available its gem equivalent. I am working with Ruby more than 6 years now and I don't remember any Ruby library I would ever need to install into site/vendor libdir (except RubyGems? itsefl).

This portion doesn't really have any bearing on the discussion. Or if it does... maybe it's that vendorlib should be fine to treat this way since the only thing that hsould be in it is rubygems itself.

ruby vs rubygems provides - I like the idea to have just ruby() provide, however I am afraid that it is too late for F17

First: are you talking below about my proposal for changing the usage of ruby() vs rubygem() or about having ruby() only? I'm assuming the former so some of these won't apply if you mean the latter.

If it's too late for F17 then you'll need to push back your ruby-1.9.3 Feature until F18 as this is one of the things that the Feature says needs to be done for the Feature to go ahead. I'd rather see this finished but there does seem to be a lot of work.

and I am not sure about all possible implications. There are some for sure (1) There is no way how to prioritize rubygem- over ruby- packages.

Requires: rubygem(rubylibrary)

(2) There are other ruby libraries, which can take your gems and vendorize them into application you are developing, which is definitely not possible if the you have ruby- library instead of rubygem-

Not that we should be encouraging it, Requires: rubygem(rubylibrary) in that package would work as well.

(3) Rubygems has metadata, if you require one gem, it loads also the gem's dependencies.

and that won't change no matter what the spec file Requires: look like.

(4) You proposed to have provide like ruby(requiredfilename), however there are libraries, where you need to require them using "require 'foo/bar'". Typical example is the rubygem-net-* family. There might be others (may ber rubygem-rack-* ?).

I don't see how changing when rubygem() and when ruby() is used would cause these issues. Which makes me think that maybe when I changed when ruby() and when rubygem() is used, it introduced something unintended that's not clear. The thrust of the change to requires and provides that I want is that:

Provides:
rubygems provide both a ruby() and a rubygem() symbol.
non-gems provide only a ruby()

Requires:
Requires can always use ruby() if the package does not require anything that can only be found via gem packaging (which as far as I know means, require'rubylibrary')

Requires may use rubygem() if the package needs to use rubygem metadata (for instance, gem 'rubylibrary' -- since there's more metadata here, there's likely other things that a package could be doing that requires that the rub library has gem metadata. Those would also fall here).

And I could continue. I propose to revert the provides to ruby(libraryname)/rubygems(gem_name) and postpone it, lets say for F19?

The ruby provides and requires are one of the three things (building and file placement are the others) that I think need to be fixed in this draft before it can be approved. So you can't push them back without pushing back the entire guidelines which also means pushing back the Ruby-1.9.3 feature.

Your example tree is making me even more cautious... You're thinking that that tree is going to show up in %gem_dir, not in %gem_extdir correct? And it's only going to show up for ruby gems that have extensions (not pure ruby)?

The tree is going to show up also in %gem_extdir, but only for gems which has binary extension.

Moreover, at these days, there really is not plethora of libraries which are not provided as gems. Take a look on ruby- packages in Fedora and you'll see that there is typically available its gem equivalent. I am working with Ruby more than 6 years now and I don't remember any Ruby library I would ever need to install into site/vendor libdir (except RubyGems? itsefl).

This portion doesn't really have any bearing on the discussion. Or if it does... maybe it's that vendorlib should be fine to treat this way since the only thing that hsould be in it is rubygems itself.

My point is that ideally, (1) vendorlib will not be used anymore, therefore it does not matter too much where it is placed (2) how would you treat library that has different implementation for JRuby and MRI when it is in single vendorlib? If you are using one vendorlib for all implementations, you have to assume that every library will run with every implementation, but that is true just as long as you are speaking about noarch libraries (and also there might be exceptions, since JRuby does not support fork for example). How will you deal with implementation specific libraries?

However, back to RubyGems?. If we would not care and go with upstream, the RubyGems? would be placed into /usr/share/ruby/rubygems, as they are originally bundled. We though also about moving them into vendorlib, but based on the (2), we decided to move them completely out of the Ruby's tree and also we decided to place the gems into separate tree.

ruby vs rubygems provides - I like the idea to have just ruby() provide, however I am afraid that it is too late for F17

First: are you talking below about my proposal for changing the usage of ruby() vs rubygem() or about having ruby() only? I'm assuming the former so some of these won't apply if you mean the latter.
If it's too late for F17 then you'll need to push back your ruby-1.9.3 Feature until F18 as this is one of the things that the Feature says needs to be done for the Feature to go ahead. I'd rather see this finished but there does seem to be a lot of work.

and I am not sure about all possible implications. There are some for sure (1) There is no way how to prioritize rubygem- over ruby- packages.

Requires: rubygem(rubylibrary)

(2) There are other ruby libraries, which can take your gems and vendorize them into application you are developing, which is definitely not possible if the you have ruby- library instead of rubygem-

Not that we should be encouraging it, Requires: rubygem(rubylibrary) in that package would work as well.

(3) Rubygems has metadata, if you require one gem, it loads also the gem's dependencies.

and that won't change no matter what the spec file Requires: look like.

(4) You proposed to have provide like ruby(requiredfilename), however there are libraries, where you need to require them using "require 'foo/bar'". Typical example is the rubygem-net-* family. There might be others (may ber rubygem-rack-* ?).

I don't see how changing when rubygem() and when ruby() is used would cause these issues. Which makes me think that maybe when I changed when ruby() and when rubygem() is used, it introduced something unintended that's not clear. The thrust of the change to requires and provides that I want is that:

Provides:
rubygems provide both a ruby() and a rubygem() symbol.
non-gems provide only a ruby()

Requires:
Requires can always use ruby() if the package does not require anything that can only be found via gem packaging (which as far as I know means, require'rubylibrary')

Requires may use rubygem() if the package needs to use rubygem metadata (for instance, gem 'rubylibrary' -- since there's more metadata here, there's likely other things that a package could be doing that requires that the rub library has gem metadata. Those would also fall here).

And I could continue. I propose to revert the provides to ruby(libraryname)/rubygems(gem_name) and postpone it, lets say for F19?

The ruby provides and requires are one of the three things (building and file placement are the others) that I think need to be fixed in this draft before it can be approved. So you can't push them back without pushing back the entire guidelines which also means pushing back the Ruby-1.9.3 feature.

Sorry, re-reading it once more and reading your comment, I was wrong. You proposed that Gems should have ruby() as well as rubygems() provides and I did not get that originally. However, there are implications I'd like to point out.

(1) If you require ruby(), it means that library is directly accessible in Ruby's load path and RubyGems? are out of the game.

(2) If you are requiring rubygem(), it means that RubyGems? are in play and therefore the library is not in Ruby's load path by default and also that RubyGems? might pick up different version of library which you have installed locally.

Binary Extension Fails to Build - I strongly disagree to follow this path always. As I said, from more than 200 gems, I know just about one which needs this procedure. Moreover, it is not as straight forward as it might look. Just for the one gem I know, I needed to tweak this approach [2].

You are missing the "sed" line. This is what may bite somebody. I had to find solution how to make the "gem build" actually work. I really don't like to fight with this kind of issues. Moreover, if you are packaging such gem, it should warn you, that something is really bad. Why it does not work? The rubygem-idn does not work, because it is not compatible with Ruby 1.9.3 nor maintained by upstream. Do we really want to support such unmaintained libraries? My answer is NO, with some exceptions. You are asking for more troubles than you solve. This is workaround if you are in troubles, not a standard supported solution.

Okay -- looking at this, it seems like a case where upstream's code is wrong. So fixing it from our spec file and submitting a bug report upstream should upstream be active looks like exactly the right thing to do.

Toshio, I really want to convince you, that repackaging gem in %prep is bad practice, workaround, unsupported black magic etc. It introduces more problems then problems solves. Please don't make it mandatory. It's like requesting Fedora's users to always rebuild each RPM locally, because there might be sometimes broken dependency or other error.

I'm willing to propose upstream some nice "gem install" flag, which would allow to install gem even though the binary extension build was not successful, to solve this issue, but repackaging gems is real overkill.

Toshio, I really want to convince you, that repackaging gem in %prep is bad practice, workaround, unsupported black magic etc. It introduces more problems then problems solves. Please don't make it mandatory. It's like requesting Fedora's users to always rebuild each RPM locally, because there might be sometimes broken dependency or other error.

I'm willing to propose upstream some nice "gem install" flag, which would allow to install gem even though the binary extension build was not successful, to solve this issue, but repackaging gems is real overkill.

Exactly. Please never think (currently) that repackaging gems like this way is supported and will work always.

Your example tree is making me even more cautious... You're thinking that that tree is going to show up in %gem_dir, not in %gem_extdir correct? And it's only going to show up for ruby gems that have extensions (not pure ruby)?

The tree is going to show up also in %gem_extdir, but only for gems which has binary extension.

So... that leaves a lot of questions about just what's in those trees.

There's an FHS concern: The directories themselves have platform names in them (nokogiri-1.5.0-x86-mingw32) which leads me to believe that they contain platform specific code. I think that what that means is that the code in there is only used by some platforms but that the rubygem mechanism is able to select the appropriate directory of code to use for the platform that it's currently running on. I think that this is contrary to the letter of the FHS ("/usr/share is for read-only architecture independent data files") but I'm not sure that it violates the spirit of the FHS or if that will matter. The full FPC will need to talk about that. Could you give more information about all the files that end up in these trees?

The second one is about having gems for multiple interpreters. But I'll keep it merged with the next section:

This portion doesn't really have any bearing on the discussion. Or if it does... maybe it's that vendorlib should be fine to treat this way since the only thing that hsould be in it is rubygems itself.

My point is that ideally, (1) vendorlib will not be used anymore, therefore it does not matter too much where it is placed

This is my point too. If no one's using vendorlib except for rubygems, then it seems like it's fine to put rubygems in it and move vendorlib to an interpreter-neutral location if your goal is to share the rubygems module among different interpreters.

(2) how would you treat library that has different implementation for JRuby and MRI when it is in single vendorlib? If you are using one vendorlib for all implementations, you have to assume that every library will run with every implementation, but that is true just as long as you are speaking about noarch libraries (and also there might be exceptions, since JRuby does not support fork for example). How will you deal with implementation specific libraries?

Right. So this is where it ties into the above section's question. How are gems and gem rpm packaging handling this? When does having the location be the same for different interpreters and platforms make sense and when does it not make sense (How does a shared location benefit gems when there will be different versions of modules anyway [I'm thinking the answer would be -- there's different versions only in a few instances but I'm seeking confirmation])?

However, back to RubyGems?. If we would not care and go with upstream, the RubyGems? would be placed into /usr/share/ruby/rubygems, as they are originally bundled. We though also about moving them into vendorlib, but based on the (2), we decided to move them completely out of the Ruby's tree and also we decided to place the gems into separate tree.

Is /usr/share/ruby interpreter independent or interpreter dependent?

ruby vs rubygems provides - [...]

Sorry, re-reading it once more and reading your comment, I was wrong. You proposed that Gems should have ruby() as well as rubygems() provides and I did not get that originally. However, there are implications I'd like to point out.

(1) If you require ruby(), it means that library is directly accessible in Ruby's load path and RubyGems? are out of the game.

The packaging as Gem/Non?-gem portion of the guidelines existed so that require 'json' worked. The argument that the draft guidelines seemed to be making was that separate non-gem subpackages were no longer needed to make that work as long as the ubiquitous requirement, rubygems, was required by this particular script/application/library. Adding a patch, if necessary, that simply did <code>require 'rubygems'</code> seems fine.

Is there anything I'm missing?

(2) If you are requiring rubygem(), it means that RubyGems? are in play and therefore the library is not in Ruby's load path by default and also that RubyGems? might pick up different version of library which you have installed locally.

The first part of this is related to the above explanation about Requires: ruby(). The second part is true. Requires: rubygem() can be seen as a superset of the functionality provided by Requires: ruby().

Requires: ruby() -- The module will be available when require 'rubylibrary' is used.

Requires: rubygem() -- In addition, the module may have specific version requirements or be used in other ways that require the extra metadata that rubygems provide.

Toshio, I really want to convince you, that repackaging gem in %prep is bad practice, workaround, unsupported black magic etc. It introduces more problems then problems solves. Please don't make it mandatory. It's like requesting Fedora's users to always rebuild each RPM locally, because there might be sometimes broken dependency or other error.

You'll need to actually provide us with examples of it not working that we can analyze then. At the moment, there's only one example and that one has a problem that it is entirely appropriate for Fedora to fix and submit upstream if Fedora wants to be giving back to the upstream communities.

The current guidelines about gems are a *very* ugly hack that lutter came up with for the FPC because there seemed to be no other way to get gems into packages. FPC was reluctant to pass them but with the assurance that no better way existed, we did. I believe we even asked lutter to keep looking into other ways to make it work after we passed them. Even at the last FPC meeting where we discussed the Guidelines in general, the way that gems apparently required non-standard methods was contentious. Now that a way exists that actually conforms to the way that rpms are supposed to be built, we're very likely to switch to it even if there are minor difficulties with it. You'll need to show us examples of all the varieties of what can go wrong for us to evaluate whether there is, in fact, unaddressable problems with the strategy.

I'm willing to propose upstream some nice "gem install" flag, which would allow to install gem even though the binary extension build was not successful, to solve this issue, but repackaging gems is real overkill.

So, here's pseudo-code for the general way gem commands should look in order to be properly built as rpms:

The thing is, we now have non-optimal forms of those commands. UNARCHIVE is roughly equivalent to gem unpack. BUILDONLY can be simulated with gem install and the proper flags and environment variables. INSTALLONLY can be made to work by copying the directories created in the previous step to the proper places in the %{buildroot}.

Your proposed gem install flag totally misses the point of the changes and wouldn't be any closer to achieving this version of the spec file instructions than what we're building now. In fact, a flag that does not fail if a C extension fails to build is more likely to be something that FPC says SHOULD or MUST not be used than something that would be useful.

Since no one was willing to give us packages that don't work with the new guidelines, I just selected a few packages and updated them to use the gem unpack method of building: ​http://toshio.fedorapeople.org/rubygems/.

The common case is to add three lines, modify two others, and move what gets done in %build vs %prep. I had to further modify one package's spec file which had a find that was catching directories when it should only be operating on files (added -type f). Thanks to macros for gem_name and other package info, these changes don't even vary from package to package. The one package that had patches applied went from 11 lines of commands down to 9 for the build step. Reorganizing that section also made it obvious that the unittests were being installed in the binary rpm under the old code, a problem that the new code did not share. The application example also became simpler by two lines.

I did not modify the Requires/Provides? with the test builds I did. Those changes don't seem to be controversial as to whether they break building; only if they break code on the user's systems.

In giving the guidelines the last bit of testing and polish I did find some more unanswered questions with interpreter independence.

It seems that the ruby(abi) Provides is only in ruby-libs; not in jruby{,-libs}. That means that an end user wanting to use jruby and having only rubygems installed still needs to have ruby-libs installed (because of the dep on ruby(abi).) Is this intentional?

There needs to be more information about jruby and other alternate interpreters. In particular, the path macros for the jruby non-gem packages need to be added.

I took a guess that non-gem packages should have separate subpackages for each interpreter since the thought is that rubygems can handle multiple interpreters but non-gem requires cannot. Further, I guessed that they should be named INTERPRETER-module, for instance, jruby-rpm.

Since no one was willing to give us packages that don't work with the new guidelines, I just selected a few packages and updated them to use the gem unpack method of building: ​http://toshio.fedorapeople.org/rubygems/.

The common case is to add three lines, modify two others, and move what gets done in %build vs %prep. I had to further modify one package's spec file which had a find that was catching directories when it should only be operating on files (added -type f). Thanks to macros for gem_name and other package info, these changes don't even vary from package to package. The one package that had patches applied went from 11 lines of commands down to 9 for the build step. Reorganizing that section also made it obvious that the unittests were being installed in the binary rpm under the old code, a problem that the new code did not share. The application example also became simpler by two lines.

Well, it's not that they wouldn't work. It's just that you have to actually patch the spec (that you get from gem spec foo.gem) with some sedding (sometimes), which doesn't seem to be a very clean way. Also, there are 250 gems, so it would be quite hard to apply you approach to all of them to see if everything works. I tried handful of them and your approach worked with some sedding, as I said. I still don't like the new approach very much, because it will make it harder to get Ruby people to RPM packaging, but I understand your strict packaging point of view. The Ruby SIG has always looked at that from a half packaging/half Ruby perspective, which is the main cause of our reasoning. So from my point of view, ok, do it this way.

I did not modify the Requires/Provides? with the test builds I did. Those changes don't seem to be controversial as to whether they break building; only if they break code on the user's systems.

As for the Provides, I still have to insist on rubygems providing only rubygem(foo) and not ruby(foo). If I looked correctly, you deleted the section about mandatory creating non-gem subpackages - thanks for that. Now, we have to distinguish between non-gem library and rubygem.
Here is my opinion:

The non-gem package ruby-foo should have Provides: ruby(foo)

The Gem package rubygem-bar should have Provides: rubygem(bar), but shouldn't have Provides: ruby(bar).

It even makes more sense now, that the non-gem subpackages of gems won't exist. The provide will let you know, if the package is requirable (in the Ruby terms, not packaging terms) without having rubygems enabled, which is an important feature. (E.g. if the user runs "yum install 'ruby(foo)'", then he knows, that this package will be requirable even with rubygems disabled.
Also, we can get to a situation (however unlikely) of having a library with a same name both as a gem and as a non-gem, but with different functionality. Then having Provides: ruby(foo) in the gem version would obviously break things.

In giving the guidelines the last bit of testing and polish I did find some more unanswered questions with interpreter independence.

It seems that the ruby(abi) Provides is only in ruby-libs; not in jruby{,-libs}. That means that an end user wanting to use jruby and having only rubygems installed still needs to have ruby-libs installed (because of the dep on ruby(abi).) Is this intentional?

The ultimate plan is for JRuby to provide ruby(abi) and use the system rubygems, but we are still facing some issues:
JRuby ships with it's own modified version of rubygems (not currently compatible with the Rubygems upstream; please note that this is not a case of bundling, but rather a fork), and we are currently trying to solve this situation. I am working intensively with the JRuby upstream trying to help them find a way to push their alterations to Rubygems upstream, so that we can really have only one universal system Rubygems (this involves some hooks here and there, alterations of gem building mechanism etc.). It seems doable, but it will probably get to F18 (if everything goes well). So we are currently sticking with the modified rubygems inside JRuby, that operate on their own via jgem command.

There needs to be more information about jruby and other alternate interpreters. In particular, the path macros for the jruby non-gem packages need to be added.

For the time being, we only have one alternative interpreter - JRuby. It's path's are gonna be a little problem, but I am discussing this with the upstream as well, so it should get fixed and macroed to F18. For F17, there are no JRuby-specific packages anyway, and everyone can still use the old approach that we used in the previous ruby guidelines (e.g. running jruby interactively to get the path from them). JRuby has been working like this since it's initial packaging into Fedora, so I think we can postpone it for one more release.

I took a guess that non-gem packages should have separate subpackages for each interpreter since the thought is that rubygems can handle multiple interpreters but non-gem requires cannot. Further, I guessed that they should be named INTERPRETER-module, for instance, jruby-rpm.

Yep, that should probably be the ideal state - but there should also be an explicit note, that the package _must_ be tested with JRuby too, because these packages are often only created for the MRI Ruby and might not work with JRuby, so the packager should pay a special attention to this.