From my peanut gallery position here, I strongly prefer /usr/lib/scl to anything under /var. Putting it under /var would also be a clear FHS violation. This packages are really not all that different from standard RPMs -- they are not variable data.

In fact, it's a stronger violation than /opt, which says

/opt is reserved for the installation of add-on application software packages.

And arguably that's what these are -- that term is poorly defined.

However, opt is problematic because the FHS also says that while distributions may install software in /opt, they must not delete or overwrite existing software, so if local installation has /opt/rh for whatever other reason, RPM doesn't have a good mechanism for dealing with that. That's a reasonable argument for staying clear, although one could also argue that the FHS should evolve to allow vendor prefixes within /opt to belong to a certain vendor.

If you think about scl installed in /opt as a add-on to Fedora, then it's according to FHS. I don't like idea of /var or /usr/lib in Fedora and /opt in all other scls. It's confusing for users who already packaged their stuff.

SCL was not intended to be part of standard Fedora. I mean part of Ring0 and Ring1 in meaning of Fedora.Next. So it will be add-on in the same sense as FHS have for /opt.

Additionally we will provide some basic collections, but it is expected that users will build collections on their own (mostly they will depend on those basic collection). And these guidelines are receipt to standardize it. And it really does not make sense that Fedora will have SCL under /usr, while users will put them in /opt. And it does not have sense to me that user will put their own SCL under /usr.

1) As Remi says - I think having compatible versions of the Red Hat Software Collection product available for EPEL is a good thing, and since EPEL feeds from Fedora, we should allow that to be built.

2) In the same way, having those compatbile versions for Fedora can be useful as well, for apps that want portability.

3) I'm fine with /opt/<vendor> - it's the de-facto usage already. Even the FHS says "Distributions may install software in /opt, but must not modify or delete software installed by the local system administrator without the assent of the local system administrator.", and, well, choosing a package to install via yum does require the assent of the local system administrator, so it still follows the standard.

The scl location issue strikes me as something that wants to have separate directories but I don't know if scls are flexible enough to do that (or if they could be enhanced to do it). For Fedora Commons, scls probably belong under %{_libdir} because Fedora Commons is being provided by the distribution. For other things in Ring 2/3 it could make sense to have a vendor-location. Local users will want to have a third location (the reason being that once they start creating scls if they're installed to the same location as either of the other two they start to run the risk of conflicts between their packages and their vendor's packages.) Third parties providing things packaged as scls would also benefit tremendously from a separate install location (this is just a generalization of the split between Fedora Commons and Fedora Addons in Ring 2/3).

Some of this becomes even more relevant when we think about how to apply this to EPEL. There we have several distinct vendors -- RHEL itself, RH SCLs, EPEL SCLs, CENTOS SCLs, and local user SCLs.

There are other possibilities to solve this than separate directory locations (although separate directories also allow us to address the FHS concerns). One way would be for scl packages to bear vendor prefixes (I heard from mmaslano and langdon that this was under discussion for perhaps different reasons as well). The vendor prefixes would carry over to what is installed on the filesystem. So you'd have something like /opt/scl/centos-ruby2.0 /opt/scl/rht-ruby2.0 /opt/scl/fdr-ruby2.0 ) I haven't yet thought these alternate strategies through to figure out if they're better or worse than the others (I have a feeling that deps may be tricky but it's very possible they are not trickier than inter-scl deps in general.)

The scl location issue strikes me as something that wants to have separate directories but I don't know if scls are flexible enough to do that (or if they could be enhanced to do it). For Fedora Commons, scls probably belong under %{_libdir} because Fedora Commons is being provided by the distribution. For other things in Ring 2/3 it could make sense to have a vendor-location. Local users will want to have a third location (the reason being that once they start creating scls if they're installed to the same location as either of the other two they start to run the risk of conflicts between their packages and their vendor's packages.) Third parties providing things packaged as scls would also benefit tremendously from a separate install location (this is just a generalization of the split between Fedora Commons and Fedora Addons in Ring 2/3).

It seems to me like lot of space for packaging errors. Even current packages are not packaged properly and ask people to decide what should go into %{_libdir} and what should go into /opt seems like another weak spot.

It seems to me like lot of space for packaging errors. Even current packages are not packaged properly and ask people to decide what should go into %{_libdir} and what should go into /opt seems like another weak spot.

If you're building a package for Fedora Commons => %{_libdir} If you're building for one of the Rings, => /opt. That part should be easy. The question of whether scl is flexible enough to handle this and in what way is not so easy (do the spec files look the same and just the macros get adjusted? Do the spec files look different?)

If you're building a package for Fedora Commons => %{_libdir} If you're building for one of the Rings, => /opt. That part should be easy. The question of whether scl is flexible enough to handle this and in what way is not so easy (do the spec files look the same and just the macros get adjusted? Do the spec files look different?)

IMO placing collections in two different locations will just make mess of things. It is possible, but I'd be strongly against it.

I know that people have various interpretations of FHS. But if you consider %_libdir (be it /usr/lib or /usr/lib64), FHS statest [1]: "/usr/lib includes object files, libraries, and internal binaries that are not intended to be executed directly by users or shell scripts." So IMO placing SCLs there would violate FHS.

Under /opt, on the other hand [2] "/opt/bin, /opt/doc, /opt/include, /opt/info, /opt/lib, and /opt/man are reserved for local system administrator use". Also, "Programs to be invoked by users must be located in the directory /opt/<package>/bin or under the /opt/<provider>" - so using /opt/fedora (assuming "fedora" is the LANANA name) seems to be the proper place to place SCLs.

@corsepiu could you please explain the glass clear violation? It's not glass clear for me.

I can't attend FPC meetings, so I'd like to sum up issues from your last meeting and receive guidance for next development of my draft.

1/ Installation path

I'd like to see /opt as installation path. If you disagree than please propose better place for installation. It shouldn't be problem to set macros in scl-utils differently, but it will confuse downstreams using the opt directory.

2/ core+addons discussion

This discussion is premature. I do not know what will Fedora become. I'd like to provide stable versions of some languages badly needed but some projects. SCL are not trying to define addons, because they'd existed before I first heard about whole Fedora.next initiative.

3/ not defining restrictions

I was hoping FPC can give me a list of missing parts in the proposal. I can propose a restriction policy on SCLs, but I'd like to know first if FPC is interested in such work. I already stated in the proposal SCL SIG should approve SCLs. Could you give me directions if it's not enough? For example should new SCL be approved by FESCo as a new Change?

I can't attend FPC meetings, so I'd like to sum up issues from your last meeting and receive guidance for next development of my draft.

I've been talking with langdon. I hope that is okay? I don't want to get biased ideas but at the same time IRC is the best way to move this forwards quickly. (It would be even more helpful if you and I or bkabrda could have IRC meetings about this so we could directly work on the draft but I'll take what I can get :-)

1/ Installation path

I'd like to see /opt as installation path. If you disagree than please propose better place for installation. It shouldn't be problem to set macros in scl-utils differently, but it will confuse downstreams using the opt directory.

There were several questions about this in the message I just sent. I think that FPC isn't completely ready to vote on this yet -- there's several ideas for where the installation could be and I think we can narrow it down to fewer options before we vote on it.

2/ core+addons discussion

This discussion is premature. I do not know what will Fedora become. I'd like to provide stable versions of some languages badly needed but some projects. SCL are not trying to define addons, because they'd existed before I first heard about whole Fedora.next initiative.

Addons provide something useful to the discussion because SCLs have very limited use within Fedora. Certain policies and guidelines that we could implement would, IMHO, mean that scls had no use within Fedora. By figuring out the complete picture we can see what possibility exists. For instance, you might want to have the language runtime in Fedora. But a language is of limited use without the stack of third party libraries that go with it. So then we have to figure out whether those also go in the same SCL, in separate SCLs, or in separate SCLs provided by the Addons/third parties. If we decided to ignore addons as an eventual part of the Fedora ecosystem, we'd be in a different position than we are if we count on them to be an important part of making things more useful later.

3/ not defining restrictions

I was hoping FPC can give me a list of missing parts in the proposal. I can propose a restriction policy on SCLs, but I'd like to know first if FPC is interested in such work.

Yes.

I already stated in the proposal SCL SIG should approve SCLs.

yeah, we didn't get to discussing that page. This particular part is something that I think I disagree with. Why treat SCLs different than other package reviews? If we have good criteria that can be applied to any package of SCLs to decide whether the SCL should be approved or not there's no reason for a separate process. If there aren't good criteria then the body that approves SCLs will be stuck with a constant burden of navigating what should be approved and not approved trying not to be unfair to their precedent, and at the same time being able to adapt to what they want to do in the future. The bundled library exception process is a similar thing in this respect -- and it's one that I think all FPC members would be happy to get rid of except that we haven't been smart enough to lay out the criteria for when bundling is allowed precisely enough that people could follow it yet.

I think SCL criteria (at least from talking with langdon) could be much simpler so we should attempt to nail that down. We can whittle out small corner cases as time progresses (and have wholly different criteria in addons if we want) but that will be a much easier process than if you end up with a morass like the bundled library exception approval process.

IMHO, the whole concept of SCLs is a violation of the FHS, because the FHS clearly says where in /usr packages should install what kind of files, having a competing prefix to /usr (or as a subdirectory of /usr) is against the FHS. (We have an exception in Fedora to allow /usr/target as a prefix for cross compilers, but that's an exception, it's NOT part of the FHS.)

In addition, SCLs open a whole bunch of practical problems, e.g. having to set up a special environment before being able to build things against them, or in some cases even before being able to run things from the SCL. There are also all the other caveats of conflicting libraries, e.g. the dreaded symbol conflicts (so you'd need to be careful all the time what you build against what SCLs).

So, I don't think SCLs should be allowed in Fedora. Instead, we should spend the effort on making it unnecessary to install multiple versions of stuff in parallel, instead always shipping the latest version and making sure the stuff we ship works against that. Or in those few cases where there are multiple major versions of libraries which need to coexist, we should make sure they can coexist in /usr (see e.g. our kdelibs packaging). I can see how this is not possible in a RHEL environment (due to the slow release cycle), but in Fedora, it has always worked so far.

IMHO, the whole concept of SCLs is a violation of the FHS, because the FHS clearly says where in /usr packages should install what kind of files, having a competing prefix to /usr (or as a subdirectory of /usr) is against the FHS. (We have an exception in Fedora to allow /usr/target as a prefix for cross compilers, but that's an exception, it's NOT part of the FHS.)

Do you have any suggestions, where to put SCLs?

In addition, SCLs open a whole bunch of practical problems, e.g. having to set up a special environment before being able to build things against them, or in some cases even before being able to run things from the SCL. There are also all the other caveats of conflicting libraries, e.g. the dreaded symbol conflicts (so you'd need to be careful all the time what you build against what SCLs).

Libraries won't conflict. People are usually prefixing name of libraries (did you mean *.so). I'm aware of users who switched whole Qt stack into SCL and they didn't run to any issues.

So, I don't think SCLs should be allowed in Fedora. Instead, we should spend the effort on making it unnecessary to install multiple versions of stuff in parallel, instead always shipping the latest version and making sure the stuff we ship works against that. Or in those few cases where there are multiple major versions of libraries which need to coexist, we should make sure they can coexist in /usr (see e.g. our kdelibs packaging). I can see how this is not possible in a RHEL environment (due to the slow release cycle), but in Fedora, it has always worked so far.

I don't want to see everything in SCL and I agree with your point that everything in Fedora should be in latest version and working well together. SCL in Fedora should help projects running on various distributions namely cloud projects, who don't have time to move to latest Rails every Fedora release. I'm aware of many projects, which would like to have stable set of packages. Currently is development in Fedora not very good.
In EPEL might be SCL more interesting for different kind of developers who want new php and similar stuff.

Besides Fedora being upstream for derived enterprise distros, I see use cases for SCL in Fedora itself.

Fedora has a really short release cycle and upgrades in a released version are forbidden. On the other side, Fedora wants to be first and to be used for development. When updating a component, which has a lot of dependencies, it's likely to break a dep. Until now, the way has been to build parallel installable versions, where it's possible. Still, there are cases, where it's not possible, and we can not expect upstreams to upgrade from one version to another version within days after a component has been released.

OpenStack? as example, is a fast moving target, and we want to have it on top of another fast moving target. Release cycles are not coupled together, like GNOME and Fedora. So there might be a OpenStack? release shortly after Fedora's feature freeze. On the other side, if you have a OpenStack? installation, there might be a good reason not to upgrade to a later version, but there might be a compelling reason to upgrade Fedora. SCL just decouple both components.

There are so many use cases, which of course don't exist in a perfect world. Just arguing against SCL because the path looks ugly, *not right* or else is just too simple.

I talk to a lot of people about Fedora and why they do or do not run Fedora -- specifically in the cloud, but in other environments as well. The overwhelming feedback I get is that Fedora is great, but that it's hard to manage change between releases. I want to combat that.

I don't know if you have seen this blog post ​http://groveronline.com/2013/06/fedora-for-servers/ about using Fedora for short-lived servers, where long lifespan isn't important but the ability to re-deploy quickly is. That's great, but a missing piece is helping users' code actually still run after the redeploy. SCLs are one way to address that problem. (Specifically, SCLs of language/platform stacks.)

I don't think they're the only way to address the problem. I'm not even sure they're the best way. But of the ideas I've heard, they're the one that is "shovel ready". I'd really, really like to see that happen, because it addresses a very real need.

To give a concrete example, Rails 4 is a Fedora 20 change. This is good, and in line with our fast-moving mandate. But it means that people using Fedora must either upgrade their code on our schedule, or else leave for a different platform. I'd like them to stay. Having a Ruby 1.9.3 / Rails 3.2 SCL in Fedora will allow these users to use our platform and migrate on their own schedule.

On a separate note, I am working to combat the perception that Fedora is an unfriendly place for people who are currently outside of Fedora to collaborate and to eventually become part of Fedora. Fedora's mission is broad: to lead the advancement of free and open source software and content as a collaborative community. When we refuse to give a home to projects -- from Red Hat or any other place -- we are not fulfilling that mission.

We are also not making the world a better place in general, because these projects must have some sort of home, and without Fedora, they must find it elsewhere. I saw some discussion in the meeting logs about SCLs being something solely from Red Hat with no benefit for Fedora. Although I don't think that's true (see crucial Fedora use case in previous comment), I also don't think it matters. A lot of software we package is there to benefit various users of the project, not just a tight core of self-reflection. When we say "no", all we are doing is restricting our ability to influence the project in question in a positive way, bringing openness, transparency, and community to areas where it might not exist. If we can't find a way to do that, we're really failing ourselves.

Where there are technical problems, let's identify them and ask for or offer solutions. Where there are non-technical problems, let's exercise Fedora's "friends" value and find a way to be inclusive and positive.

Talked with one of the LSB members after yesterday's LSB meeting regarding the /opt patch I proposed. Here's what he said:

<abadger1999> orc_fedo: If I might ask a question -- the last comment on my FHS /opt ticket is that it was discussed in triage... was there any feedback that I should act upon?
<abadger1999> https://bugs.linuxfoundation.org/show_bug.cgi?id=1164#c9
<orc_fedo> abadger1999: it is triaged .. fix will follow 5 issuance most prolly
<abadger1999> orc_fedo: Okay -- so that means the patch is acceptable; just won't be in an LSB/FHS release until later?
<orc_fedo> abadger1999: the patch has some verbiage that was not standards like, but rather behavioual 'best practices' that needed to be cleaned to match FHS style as I recall .. functionally it is correct
<orc_fedo> the bug as filed really covers two things - the /opt path stuff as to fedora, and some opt path stuff as a general rule
<orc_fedo> we are not so good about splitting bugs , esp as it is in the old tracker ;)
<orc_fedo> s/filed/developed/
<orc_fedo> I think comment 2, 3 and 4 in the bug basically says it is fine
<abadger1999> orc_fedo: Cool, thanks. If there's any rewriting that you need me to do, just let me know.

So this isn't a guarantee that the change will go in but it's a good sign that it will likely be accepted in some form. Actually getting it in will likely be looked at after the release of LSB 5 (that's their priority right now).

FPC members (limburgher, I remember you specifically were willing to vote for /opt if this change went into FHS), is this sufficient for you? We'd probably note it as an exception to the FHS with the justification that the FHS is going to be updated on this point (with a link to the LSB bugzilla ticket).

I've pinged the LSB people and got a reply from Jeff Licquia. @spot: He says he's going to process his backlog of LANANA requests this week. If you don't get an email from him about it being done, feel free to ping him next week (IRC: licquia) and he'll dig out our specific request.

Apologies, I saw the original email but not your followup on Wednesday. I had already been in touch with Jeff on IRC by that time.

As for whether there should be a ticket -- the LANANA web page says that the way to get a new vendor entry is to send email rather than filing a ticket. It's certainly nicer to check for status on a ticket so perhaps the instructions should be changed.

I was trying to stay away from this discussion for a long time, but seeing log from yesterday with proposals such as "SCL packages must be separate packages, not separate branches in the git repos" I really wonder who will go and maintain any SCL in Fedora. It seems you try to make it as hard as possible, without thinking about sustainable evolution and maintenance of collections etc.

Let me remind you current guideline about SCL macros [1]. In that light I'll try to give you an example. Lets assume I have Ruby 2.0.0 in F20, which already contains every possible SCL macro and can build as a collection package. Now I'll prepare Ruby 2.1 for F21. So it would be natural to branch the Ruby 2.0.0 and keep them in repository and put Ruby 2.1 into the master for F21.

But now, it is not possible. Although the package was fine for F20, you try to make a re-review for making it F21 collection package. What is the reason? Why don't you keep people on focusing on important issues? This will lead just to formal reviews, where somebody click on some button, maybe run fedora-review above the package but that is it. It is just bureaucracy.

Also, if there should be some security update, or some packaging update, it is quite common, that the patch needs to be applied into all versions of the package, but keeping the packages in separate repositories makes it much harder. We are using Git but this approach of several repositories reminds me CVS era.

Finally, I understand our tooling is not good enough, you cannot delete private branches [2, 3], etc, but it seems that instead of improving the tooling, it is used against any progress.

As of now, I can't see how ruby193 SCL [4, 5] could get into Fedora ever, since you have not removed any obstacle in a way so far, but build others. You more or less ignore every knowledge and expertise we gained during development of SCLs in Red Hat and release of RHSCL for RHEL. The only think you achieved so far is that you pushed us forward to build softwarecollections.org. Well done guys.

I really admire Marcela, that she haven't completely gave up yet, but you are close to that point as far as I can tell.

Hopefully that will stop us from having to do even more work in the future to revert changes to mainstream versions of packages.

We also started voting on whether to recommend to releng that the SCL packages need to be placed in separate package/git repos or in the same branch. Since more votes are needed for that one, I've created a new ticket to track those votes. I'll send the results to releng once completed:

@vondruch - the FPC discussion about making SCL Guidelines quickly settled on having separation of SCL and mainstream spec files similar to how mingw packages are managed. We officially voted on that at last week's meeting and changed the guidelines you're pointing to.

It's odd that you talk about people being unwilling to refuse to improve tooling. I've been feeling the same way about the people pushing for SCL inclusion. FPC has talked about the things we'd like and the things we'd need in order to have SCLs in Fedora. But instead of working on improving rel-eng and infra code and to a lesser degree working on scl-utils (We've been blocked on this bug for months: ​https://bugzilla.redhat.com/show_bug.cgi?id=1066665 ) to make those things possible we've just heard the same arguments denigrating FPC's efforts over and over.

@vondruch - the FPC discussion about making SCL Guidelines quickly settled on having separation of SCL and mainstream spec files similar to how mingw packages are managed. We officially voted on that at last week's meeting and changed the guidelines you're pointing to.

That decision is against basic design goal of SCLs! The first assumption of any SCL package is that it should be as simple as possible to build as regular package or SCL package. If you treat SCL package as different package, it is not that simple anymore. You assumptions makes the packages diverge much easier than necessary.

This is *your* opinion. The only other person explicitly in favor of reviews in that thread is mattdm.

Actually, you might be surprised that I like reviews, but as long as you don't mandate review of every rebased package, or even every change in package, then this is just bureaucracy. If you take into account, that one RoR collection is more then 50 package, then I can't imagine, that somebody will spend his time and do the reviews carefully enough to be worthwhile. These reviews just decrease quality of the reviews and morale of reviewers.

It's odd that you talk about people being unwilling to refuse to improve tooling. I've been feeling the same way about the people pushing for SCL inclusion. FPC has talked about the things we'd like and the things we'd need in order to have SCLs in Fedora. But instead of working on improving rel-eng and infra code and to a lesser degree working on scl-utils (We've been blocked on this bug for months: ​https://bugzilla.redhat.com/show_bug.cgi?id=1066665 ) to make those things possible we've just heard the same arguments denigrating FPC's efforts over and over.

I agree that scl-utils are not evolving as fast as they should and as we'd like. There is not much I can do about that.

You might call my arguments denigrating, but this [1] is first ticket I remember about SCLs in Fedora. It was opened almost 2.5 years ago. Where we moved since that time? Nowhere. I just see basic disagreement between SCL proponents and FPC.

If you take into account, that one RoR collection is more then 50 package, then I can't imagine, that somebody will spend his time and do the reviews carefully enough to be worthwhile. These reviews just decrease quality of the reviews and morale of reviewers.

This has been sitting around forever and nothing is changing. I'm just going to close this ticket so it's off of our agenda, but I do hope that if someone wants to push this forward they would feel free to reopen or file a new ticket.