Problem/Motivation

We want Drupal 8 to be fast by default. One aspect of being fast by default is the front-end performance, and one aspect of that is the amount of data we send to the browser. JavaScript minification can be a huge help in that regard, but has long been impossible, for multiple reasons:

The most fundamental blocker. Drupal <8 used to be developer-friendly by default rather than fast by default, which meant that if we'd ship with JS aggregation/minification enabled by default, we'd have to detect file system changes on every request, which would be detrimental for performance. #2226761: Change all default settings and config to fast/safe production values changes that: aggregation can be enabled by default, and when developing, it's easy to disable aggregation.

The tertiary blocker. The license problem: proper minification deletes all non-essential data, including license information. That's the part this issue addresses.

The last mile. We couldn't find consensus on which JS minification tool to use. This issue doesn't aim to solve that, it only aims to make it possible to add JS minification at a later point in time to Drupal 8, since adding that is not an API change, it's improving an existing feature. (Our current "minification" strategy is: "just use the entire file, and append a defensive semi-colon".) That could even happen after beta 1 (see #3).

This allows us to generate a "JavaScript License Information" page, as mandated by JSLWL. We link to that page on every Drupal page that contains any JS. And on that page, we can list every JS asset along with its license, thanks to 1.

However, we must also list the license and source for aggregated JavaScript assets (which we want to be minified in the future). JSLWL allows us roughly 2 ways of doing that:

The source code file can be a single, unminified JavaScript file, a .tar.gz archive, or a .zip archive. If a source archive includes multiple JavaScript files, the archive must include a file named 00-INDEX that lists the order in which individiual source files should be concatenated to produce a single file that's equivalent to what's hosted on the site.

So either: an unminified aggregate, or an archive containing all aggregated JS files and a describing file. The latter is much more painful to do, so we choose the first, for which we already have the necessary infrastructure. The JS asset collection optimizer (asset.js.collection_optimizer) tracks its aggregated (and minified) JS files (we can get them via AssetCollectionOptimizerInterface::getAll()); by adding a secondary JS asset collection optimizer asset.js.collection_optimizer_license_web_labels_annotator which annotates each contained asset (with a link to the license information) without minifying it, we effectively get two key-value maps with the same keys (because the same collections of JS assets are passed to them and they use the same JS collection grouper) but with different files (one minified, another annotated and unminified). On the JSLWL "JavaScript License Information" page, we can now list the unminified annotated aggregate as the source for the minified aggregate.

Remaining tasks

Review.

User interface changes

None that are visible, but a new "JavaScript License Information" page at /system/jslicense.

API changes

(Not really an API change, but an internal behavior change.) An unminified aggregate is generated for every minified aggregate.

Route/response addition: /system/jslicense.

(Not really an API change, but a markup addition.) Each page has a <link rel="jslicense" href="/system/jslicense" /> tag in the HTML head.

This is excessive IMHO. Does this means that every js library that does not have the 'gpl-compatible' tag will affect negatively the performance of the site? What about non-external libraries, like drupal-ajax ? Are they bundled together?

Agreed, that's why I made the change. You're just +1ing this renamed YAML alias, right?

Thanks, changed.

I figured that made most sense because we currently have modified versions of those two libraries. But yes, let's add the license information now. Fixed.

This is what the FSF, EFF etc. sites do, and what the standard prescribes. There was talk of using a <link> element instead. Though that's not yet in the standard, IMHO we could do that too, because it enables the same thing: automated discovery of JS asset licenses.

drupal.ajax is GPL-compatible, so of course it's aggregated. Everyt JS asset used in Drupal core is GPL-compatible, and almost every JS asset in modules or themes on drupal.org is. This is necessary to be able to describe the license of aggregated JS assets.
I could go one step further, and allow non-GPL-compatible assets also to be aggregated together, but in a separate group, so that we'd still be able to determine the license of JS aggregates.

2. Indeed +1 the renamed the YAML alias to "jquery_ui_version". (I understood the opposite by mistake)
And, yes, I really like this patch and the motivation behind, in the meanwhile doesn't affect negatively the performance. Thank you for working on this.

Agreed, and that's what I did first. But that causes slightly different markup to be output: it causes a "region" wrapper <div> to be created, which is not the case with this approach. Since it's a hidden link anyway, I think it's better to not create the "region" <div>.

Because it's a DUTB test.

#9:

The license names listed at http://www.gnu.org/licenses/javascript-labels.html are not mandatory, they're considered "good" ( Good license identifiers and their associated links are: […]). Obviously, the list they have there is not even close to comprehensive.

Aha! Yeah, I wanted to keep consistency, I never realized "remote" stood for "remote git repo". Thanks for that, I agree that "url" is much better :) I like your thinking regarding "source", but I agree it'd be confusing. So I went with "url".

Because there are many GPL-compatible licenses, and aside from that, there are numerous licenses. We can't possibly include a comprehensive list to do this automatically.

Good point. I figured we wanted to explicitly define the licenses of vendor libraries, because we (Drupal) don't own them. That removes any ambiguity. I don't have a strong opinion though, so if you want me to change that, I'll do it.

Again, JSLWL doesn't have enforced "license identifiers"; it's free-form. Easily demonstrated by looking at https://weblabels.fsf.org/www.fsf.org/CURRENT/, where they don't use "GNU-GPL-2.0-or-later", but "GNU General Public License version 2 or later".

P.S.: wow, I didn't realize "schemata" was an alternative plural form of "schema"! :D

RE: GPL3: that won't be a problem, since Drupal is already marked as "GPL2+". The only difference it will make here is that our aggregates would be labeled GPL3, and we could also include Apache-licensed JS assets in our JS aggregates. (The biggest difference license-compatibility wise is the Apache license compatibility.)

The state entry can't necessarily be relied upon - at least the proposal to aggregate based on libraries in #1014086: Race conditions, stampedes and cold cache performance issues with css/js aggregation would remove the need to write aggregates to state since we could determine the aggregate to serve based entirely on URL. Even if we don't fix that in core before 8.0.0, the system is pluggable (i.e. agrcache and advagg both swap it out for 7.x already and neither use the core aggregate variable).

@Bojhan yes this will add a page, and possibly a visible link to that page on every other.

However, you're absolutely right that asset aggregation plugin overrides would break this. I don't see any other way of fixing that other than adding a new API to list all known aggregates, unfortunately.
Which makes me think we should we descope this issue a little bit: introduce license metadata for asset libraries here, expose /system/jslicense for each JS asset, but don't provide license information for aggregates just yet, because that requires a new API.

It sounds really bad to me that incompatible licenses should not be aggregated together in one file and causing several http requests in a browser. That may makes me setting a non-gpl libs as true just to make it compressed in one file. The intention of #156124: JS and CSS aggregation deletes license information was to keep the license above the code (remote value). Performance is more important.

We also do not care about CSS framework licenses for many years and just remove all comments (including the license comments). This cannot be a blocker for JS minification. Otherwise we need to remove CSS aggregation also as long as this feature is in. The ONLY reason why we have removed JS minification from core in past was that *valid* JS code has been destroyed by the minification.

Even minified CSS can quite easily be "unminified", the only thing missing then are comments. By its very nature, it cannot be obfuscated (e.g. variable names being renamed to be nonsensical), because CSS selectors and properties need to be interpreted by the browser.

CSS does not contain logic/executable code (except in rare cases "logic" in the form of the calc() property, but that's always trivial). I think that the FSF therefore considers the inability to obfuscate CSS (i.e. you can always access the source code) in combination with the Linking Exception, this continues to meet the GPL's requirements (provided that you also meet, for each linked independent module, the terms and conditions of the license of that module).

There is one thing I cannot find in the patch. How can I define that the file is already a minified version? In case a file is already minified it is a waste of cpu to double minify them. I think we need to combine them in such a case only.

The reason I hadn't considered it yet, is precisely because of what you say: we don't have a flag to indicate that a file is already minified.
Personally, I think that all JS files we ship with should be unminified. Minification should be done by Drupal's JS minifier (which could use UglifyJS or whatever external library is best or carries your preference). Because only then you can easily switch to the unminified version, and hence only then you can easily debug.

I'm sorry to say, but you are wrong. Check out the assets folder. Ckeditor, html5shiv and others are minified. I also think it's wrong to use UglifyJS in all cases as the maintainers of libs have made extreme tests with several minifiers and decided for the best from their test results, latency, ux and other reasons. If you get this in here without such a "disable" directive we will double minify a lot of code.

I have not seen these minifier code, but minifying 5mb js code on the fly sounds like a crazy task to me. Isn't UglifyJS not a nix only tool, too? This is a showstopper to me.

Already since the reroll in #23, the "only allow gpl-compatible JS files in the aggregate" part of the original patch has been removed. I've now also updated the issue summary to reflect that.

As of #25/#26, we have a new soft blocker, which is: Drupal core doesn't yet allow us to know which files already are minified.

(Yes, hass, you were absolutely right that I'm wrong. CKEditor and others are minified. Of those, I think only CKEditor is rightfully minified, because minifying that is a massive amount of work (because it's a massive amount of JS) and you rarely need to debug it (only when developing CKEditor plugins). html5shiv could also be acceptable because it's extremely, extremely rare that one would need to debug that, plus one doesn't develop against that. But one needs to step through jQuery's code quite frequently, so hence it shouldn't be minified.
I also gave a bad example — Drupal of course wouldn't ship with UglifyJS, because that'd imply a dependency on node.js.
All that being said: without actually having a JS minifier in core, my "should not be minified" reasoning is pointless.)

I think it's a soft blocker, because the double minification really is an existing bug in Drupal core; it already happens in HEAD!

Having a (hidden) link at the bottom of the page is not great, especially because I'm guessing people will end up blacklisting /system/jslicense in their robots.txt (because it gives away most of the modules used on the site (security issue too maybe?)).

Having a <link rel="jslicense" href=""> would serve the same purpose. I'm reaching out to the relevant mailing list to know what they'll say.

Other than that the PHP looks good to me, not that I'm a great judge for that :D

I'm not sure this row is correct - if I understand https://www.gnu.org/licenses/javascript-labels.html correctly the third row should be a link to a non minified - source code version of the javascript. Maybe the libraries info needs to have a link we can use here.

We should be focused on shipping D8 right now, and only working on non-criticals that have long-term consequences for D8. This issue can easily be done in 8.1; the necessary API changes have already been made.

webchickCreditAttribution: webchick at Acquia commented 22 May 2015 at 15:25

While that is true, I think this is still worth working on in 8.0.x, since:

a) It's an enabler for a performance improvement, and we need more of those. ;)
b) (raised today by nod/jhodgdon) If we try and do JSDoc, without this we're going to bloat the size of JS files unnecessarily.

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

The changes to template_preprocess_html() didn't apply as it has changed substantially since 8.0.0-beta11. I've attached one file of the rejected changes to the theme.inc file and another with beta11's template_preprocess_html with the patch applied, so it can be seen in context.

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)