Starting with a summary: ​Let's 1) use something like dependency-report
scripts [1] to get coordinated, and 2) make the initial builds against
bootstrap so we get a working thing fast and can iterate.
I've realized that developing the initial set of modules for F27 could be a
bit tricky in the beginning as some things might require many basic
dependencies on top of Platform to run and build. These dependencies, like
autotools or dynamic languages for example, need to get modularized first.
In F26 Boltron we had a 'shared-userspace' and
'shared-build-dependencies'
modules that kind of randomly included many of the shared dependencies of
other modules. I don't think that's necessarily terrible (although I'm sure
contyk would argue it is!), but it's not ideal. We should make modules that
represent a thing, like autotools, python2, python3, ruby, gtk2, gtk3, git,
etc. But making all of these modules on a greenfield without a solid base
could be tricky. How do we help people to know what they should include,
and what's going to be somewhere else?
For this, I have written a set of scripts, and already named them badly
'dependency-report' [1]. We have tested it with FreeIPA maintainers to
generate a dependency report [2] for them, and we have already identified
many modules that FreeIPA need as dependencies. I think we could use this
to help us with the initial design.
This brings me to my second thought... to build what we want, we need to
modularize a lot of other stuff. And when we are ready with the initial
design - that means we have identified all the modules we want to have and
all their dependencies, we need to build them. But, for example, a pretty
commonly needed thing like autotools [3] has pretty crazy build
dependencies [4] including Java, gtk2, gtk3, erlang, X11, python2, python3,
etc. And many of these need autotools, of course. We need to do
bootstraping.
So instead of trying to make everything perfect from the begining, we could
build everything we need against the Bootstrap module - a module that is
used as a buildroot for Host and Platform and contains mostly everything we
need. To compare, there is a report without using bootstrap [5] and with
using bootstrap [6]. This way we'll have a pretty decent set of module very
soon.
The next step would be lookin at the build dependencies of these initial
modules, building them against bootstrap, and rebuilding the initial
modules against these new ones. And then do it for the new ones as well,
until we have everything in place.
Also, with this top-down approach, we'll be flexible with the frequent
changes of the Platform module as it's getting stabilized which also makes
the design of the low-level modules nearly impossible right now.
With this approach, we:
1) get coordinated and make modules without conflicts
2) have a working thing very soon
3) can iterate on expanding the base set over time
If you'd like to participate, please feel free to propose your module on
the F27 content plan [7], I'll make you a repository under the
modularity-modules space [8] which serves as an input to the scripts, and
include your module in there.
Cheers!
Adam
[1] https://github.com/fedora-modularity/dependency-report
[2]
https://github.com/fedora-modularity/dependency-report/tree/master/module...
[3]
https://github.com/fedora-modularity/dependency-report/tree/original-plan...
[4]
https://github.com/fedora-modularity/dependency-report/blob/original-plan...
[5]
https://github.com/fedora-modularity/dependency-report/tree/original-plan...
[6]
https://github.com/fedora-modularity/dependency-report/tree/bootstrap/glo...
[7] https://github.com/fedora-modularity/f27-content-tracking
[8] https://github.com/modularity-modules
--
Adam Šamalík
---------------------------
Software Engineer
Red Hat

So instead of trying to make everything perfect from the begining, we
could
build everything we need against the Bootstrap module - a module that is
used as a buildroot for Host and Platform and contains mostly everything we
need. To compare, there is a report without using bootstrap [5] and with
using bootstrap [6]. This way we'll have a pretty decent set of module very
soon.

I am +1 to a plan which yields a decent set of modules very soon. :)
Also, I'm continuing to feel good about this picture I drew several
years ago — https://mattdm.org/fedora/2015rings/ :)
--
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

This is exactly why the separate Core and Extras were such a PITA and why
the Core-Extras Merge was done. Doing the opposite now is a BAD idea.
Modularity may sound great on paper, but as soon as you actually try to
implement it, it falls apart like a house of cards.
Kevin Kofler

Adam Samalik wrote:
> all their dependencies, we need to build them. But, for example, a pretty
> commonly needed thing like autotools [3] has pretty crazy build
> dependencies [4] including Java, gtk2, gtk3, erlang, X11, python2,
> python3, etc.
This is exactly why the separate Core and Extras were such a PITA and why
the Core-Extras Merge was done. Doing the opposite now is a BAD idea.

I have similar feelings, unfortunately.
Dependencies go here and there, new software version means different dependancy
set; that's never-ending iteration in the distro ecosystem. Trying to draw a
thick border line (called "circle") in Fedora Everything to separate the
working
ecosystem into smaller ecosystem (where packages in one set don't see the other
sets) is IMO impossible without loosing some part of Fedora functionality (which
is not acceptable as we spent too much power on it so far).
Why I'm writing is the mentioned "autotools" keyword (for some reason it is
privileged soon-to-be module, dunno why). We should first define "what belongs
to autotools". But anyways... Making it a self-standing module is not easily
possible. All the (build)deps are there for reason. We really don't want to
have "language foo" support in Automake without having that tested during
build-time (since that's the only testsuite the upstream can maintain). It
means that build (and probably runtime) of Automake will depend on "lang foo"
forever (and that's noarch, with library deps it is even worse). Why we are
making problems from non-problem [4]?
Ok, I agree that Fedora needs modules for life-cycle separation. But I don't
understand why we start "bottom up" and not the other way around. Why we we
start with the base build toolset like autotools, instead of with leaf
packages on which (almost) nobody depends on? Good examples are
{ RHSCL } / { languages } ... Subtract language-collections as modularity
doesn't solve "parallel install-ability" so multiple versions of Python
versions won't be possible in module world (rhscl allows that).
Pavel

I don't. I consider what you call "life-cycle separation" (I'd rather
call
it "inconsistent EOLs") a bug rather than a feature.
This is yet another of those "features" that sound great on paper, but lead
to a horrible user experience in practice. Right now, it is easy to know
when you have to upgrade, as there is one EOL for the entire distro. With
inconsistent per-module EOLs, as soon as you have a non-trivial amount of
modules installed, it is impossible to track down when you need to upgrade
what. So either you end up with an unsupported version of the module without
even noticing, or you get forcefully upgraded at what will often be the
worst possible time.
Kevin Kofler

Pavel Raiskup wrote:
> Ok, I agree that Fedora needs modules for life-cycle separation.
I don't. I consider what you call "life-cycle separation" (I'd rather
call
it "inconsistent EOLs") a bug rather than a feature.
This is yet another of those "features" that sound great on paper, but lead
to a horrible user experience in practice. Right now, it is easy to know
when you have to upgrade, as there is one EOL for the entire distro. With
inconsistent per-module EOLs, as soon as you have a non-trivial amount of
modules installed, it is impossible to track down when you need to upgrade
what. So either you end up with an unsupported version of the module without
even noticing, or you get forcefully upgraded at what will often be the
worst possible time.

But you get upgraded even now. Firefox gets major-version upgrades
even within the life of the Fedora version, as do other packages.
Yes, it might become a mess if the tooling is not right or clear. But
it is also an opportunity to potentially get a choice between stay on
the old, stable, vs. get the latest greatest.
--
Jan Pazdziora
Senior Principal Software Engineer, OpenShift Security Team, Red Hat

Yes, it might become a mess if the tooling is not right or clear.
But
it is also an opportunity to potentially get a choice between stay on
the old, stable, vs. get the latest greatest.

But it seems to be the wrong way to do this; trying to shoehorn
multiple version choice onto an operating system structure that is
designed around having only a single choice. I think it would be
better to re-design the OS to allow multiple versions of everything to
exist in parallel. Would that be a lot of work? Sure. But so is
modularization, and if the job is done properly by modifying the OS, the
end result is a lot more stable and extensible, as well as maintainable.
To take an example, let's think about libraries. Currently, the system
is designed to use a so number to mean that we are locked into some api
version, and restricting choices of applications based on that.
Imagine if, instead of so numbers, we use signatures of the api calls (a
la c++ overloading) to version libraries and calls within those
libraries. When the OS tries to load an application, it looks at the
signatures it needs for the libraries the application calls, checks the
call signatures of the functions it is using on those libraries, and
decides which version of the library to use. The package manager does
the same thing during install, to determine whether it can use existing
libraries, or has to install a new library with the correct signature
for the application it is installing.
The whole tool chain will have to change for this. Compilers will have
to advertise signatures for all library calls in binaries. The OS will
have to match signatures between applications and libraries when loading
programs to run. Package managers will have to run garbage collection
on packages that no longer are used, especially libraries, and ensure
that a spanning set of library calls for all applications is
always present. Unfortunately, the library api function call signature
might be a false flag because the functionality could change without a
change in signature, but that should be rare.
However, after this is done, there is no need for modularity, or
rather, the entire system is modular. And that's the proper way to do
modularity. And it is pretty much invisible to the end user. They
just install a package, and the package manager installs whatever is
needed, if necessary. Of course, there has to be some mechanism to
determine which version of the application to run if there are multiple
versions of it installed.
And there are issues with the structure of the file system
(does /usr/bin or /usr/lib64 still make sense?), but I won't go into
that.

On Thu, 7 Sep 2017 13:27:18 +0200
Jan Pazdziora <jpazdziora(a)redhat.com&gt; wrote:
> Yes, it might become a mess if the tooling is not right or clear. But
> it is also an opportunity to potentially get a choice between stay on
> the old, stable, vs. get the latest greatest.
But it seems to be the wrong way to do this; trying to shoehorn
multiple version choice onto an operating system structure that is
designed around having only a single choice. I think it would be
better to re-design the OS to allow multiple versions of everything to
exist in parallel. Would that be a lot of work? Sure. But so is
modularization, and if the job is done properly by modifying the OS, the
end result is a lot more stable and extensible, as well as maintainable.
To take an example, let's think about libraries. Currently, the system
is designed to use a so number to mean that we are locked into some api
version, and restricting choices of applications based on that.
Imagine if, instead of so numbers, we use signatures of the api calls (a
la c++ overloading) to version libraries and calls within those
libraries. When the OS tries to load an application, it looks at the
signatures it needs for the libraries the application calls, checks the
call signatures of the functions it is using on those libraries, and
decides which version of the library to use. The package manager does
the same thing during install, to determine whether it can use existing
libraries, or has to install a new library with the correct signature
for the application it is installing.

The per-symbol API versioning in RPM was
proposed five years ago by
ALT Linux people. It actually works well in their RPM fork.
An isolated version of that code is available at https://github.com/svpv/rpmss
You may want to read http://rpm5.org/community/rpm-devel/5356.html and
associated thread discussion to understand how it all works, but
overall it is much better suited to cover per-symbol export and import
between ELF objects. The ABI requirements become explicit and ability to
automatically detect which applications are using which symbols from a
library affected by a CVE is a good example where this could be used.

The whole tool chain will have to change for this. Compilers will
have
to advertise signatures for all library calls in binaries. The OS will
have to match signatures between applications and libraries when loading
programs to run. Package managers will have to run garbage collection
on packages that no longer are used, especially libraries, and ensure
that a spanning set of library calls for all applications is
always present. Unfortunately, the library api function call signature
might be a false flag because the functionality could change without a
change in signature, but that should be rare.
However, after this is done, there is no need for modularity, or
rather, the entire system is modular. And that's the proper way to do
modularity. And it is pretty much invisible to the end user. They
just install a package, and the package manager installs whatever is
needed, if necessary. Of course, there has to be some mechanism to
determine which version of the application to run if there are multiple
versions of it installed.
And there are issues with the structure of the file system
(does /usr/bin or /usr/lib64 still make sense?), but I won't go into
that.
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org

The per-symbol API versioning in RPM was proposed five years ago by
ALT Linux people. It actually works well in their RPM fork.
An isolated version of that code is available at
https://github.com/svpv/rpmss
You may want to read http://rpm5.org/community/rpm-devel/5356.html and
associated thread discussion to understand how it all works, but
overall it is much better suited to cover per-symbol export and import
between ELF objects. The ABI requirements become explicit and ability
to automatically detect which applications are using which symbols
from a library affected by a CVE is a good example where this could
be used.

The per-symbol API versioning in RPM was proposed five years ago by
ALT Linux people. It actually works well in their RPM fork.
An isolated version of that code is available at https://github.com/svpv/rpmss
You may want to read http://rpm5.org/community/rpm-devel/5356.html and
associated thread discussion to understand how it all works, but
overall it is much better suited to cover per-symbol export and import
between ELF objects. The ABI requirements become explicit and ability to
automatically detect which applications are using which symbols from a
library affected by a CVE is a good example where this could be used.

You may want to read http://rpm5.org/community/rpm-devel/5356.html
and
associated thread discussion to understand how it all works, but
overall it is much better suited to cover per-symbol export and import
between ELF objects. The ABI requirements become explicit and ability to
automatically detect which applications are using which symbols from a
library affected by a CVE is a good example where this could be used.

There are other tools to query this information. It's not necessary to
expose it at the RPM level for this purpose.
Thanks,
Florian

But you get upgraded even now. Firefox gets major-version upgrades
even within the life of the Fedora version, as do other packages.

Firefox gets upgraded for security reasons. Sticking to an old version is a
very bad idea.
Other packages get upgraded because it is considered safe to upgrade them
for all users, otherwise they do not get upgraded within a release. That is
the point of having Fedora releases. Even with Modularity, I do not expect
maintainers to keep supporting an old upstream branch when the new one is a
safe drop-in replacement. They will just upgrade the package within the same
module branch, just as they do in the distro branch now.
The problem starts when a module branch is arbitrarily discontinued for a
newer one that is NOT a safe drop-in replacement. That is what coordinated
distro release EOL dates are intended to protect users against. (Note that
it is in general the maintainer's duty to backport fixes as needed!)
"Lifecycle separation" irremediably breaks that, and is thus broken by
design.
Kevin Kofler

This is exactly why the separate Core and Extras were such a PITA and
why
the Core-Extras Merge was done. Doing the opposite now is a BAD idea.

Core-Extras was a bad ideas because Core was developed inside Red Hat
in a non-open way and did not allow community involvement beyond that
of beta testing. Merging it all together was probably the only
realistic way to fix that (and fixing that was absolutely the best
thing to do), but if we'd kept separate repos, we'd have addressed the
technical problems sooner and had a lot more of the flexibility that we
_really_ need to keep growing.
--
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

Core-Extras was a bad ideas because Core was developed inside Red
Hat
in a non-open way and did not allow community involvement beyond that
of beta testing. Merging it all together was probably the only
realistic way to fix that (and fixing that was absolutely the best
thing to do), but if we'd kept separate repos, we'd have addressed the
technical problems sooner and had a lot more of the flexibility that we
_really_ need to keep growing.

That was only one aspect of the problem. The bigger issue is that packages
routinely had to be built without some features because the package was in
Core and the feature would have required something from Extras (an issue we
are still seeing with RHEL and EPEL nowadays), packages had to be brought
from Extras to Core as dependencies, and in some cases, packages actually
had to be moved from Core to Extras to be able to use dependencies from
Extras. This issue is now coming back with the modules. Some packages also
actually had to be split into 2 SRPMs from the same sources, one in Core and
one in Extras, instead of using subpackages, due to build-time requirements
on Extras packages. (One example was the package I got sponsored with:
redhat-artwork-kde, split out from redhat-artwork as part of
https://fedoraproject.org/wiki/Archive:UnleashKDE – I maintained that for 2
weeks, then the Core-Extras Merge happened and it was already obsolete. ;-)
That was an interesting start of a packager career. ;-) )
Having a unified Everything repository from which individual binary packages
can then be picked from is a huge advantage. It is just too common that you
have things such as:
* optional plugin subpackages requiring extra dependencies to build. You
want to build the package with those dependencies, then split the plugins
into subpackages so that the main runtime package can be installed without
those dependencies. But with the module isolation, you cannot do that
anymore.
* documentation requiring huge toolchains such as LaTeX to build, but not to
view.
* dependencies required only for the test suite, as in the autotools case.
In addition, having a unified end of life makes system maintenance A LOT
easier for the users.
I consider the whole concept of modularity fatally flawed by design. You are
about to eat the cake and to not realize that you will not have it anymore
then. The feature's promise that you can eat your cake and have it too is
impossible to fulfill.
Kevin Kofler