I now have a small set of preliminary results, these are from a trial with one of our teams, scope is:
Pages: UK home (https://www.bbc.co.uk/) and Travel (https://www.bbc.co.uk/travel)
Assets tested: https://mybbc.files.bbci.co.uk/*
Data source: CDN edge responses
Firefox v49+ represents ~2% of our user base
Measurement metric: % ratio of 304's on mybbc.files.bbci.co.uk for HTTP (immutable doesn't work) vs. HTTPS
Duration: 3 weeks of data without immutable, 1 week with immutable (i'll update this in ~2 weeks)
Result: ~10.8% reduction in 304's served once immutable is enabled
Caveat: This is a small dataset on a small number of pages. Hopefully it's interesting in some way though.
We're looking to expand the trial into more of our teams and we have seen no issues thus far, except that the CDN strips Cache-Control extensions currently in some configurations - I have a ticket open with them via which I am hoping to rectify that.
Please let me know if you need more info, more than happy to provide whatever i can.
Cheers

Also, i did some semi-scientific page reload tests in Fx 49 and saw:
* Home page
* 12.6% reduction in domLoaded time
* 84.9% reduction in num requests to network
* Travel home page
* 47.6% improvement in domLoaded time
* 89.1% reduction in num requests to network
This was mean values over 5 test runs per page, outliers *not* removed, on my laptop over a min. 1Gb/s link to the internet.
Hopefully that provides something of a guide on visible effects.

I'd also be very interested to know if there is any movement on this.
I just ran a few more numbers, now that we have some more data and, as a set of ballpark figures, we're now seeing ~31% fewer 304's served from our CDN edge on a subset of our static assets (which have immutable enabled) - this is obviously the effect of *just* Firefox v49+ which is ~7% of our audience right now.
Extending these numbers, this *could* translate into saving ~8Tn requests/month (totalling ~2.4TB/month, assuming a mean size of 323 Bytes/304) against our primary CDN - if all our pages were HTTPS, all UA's supported immutable and all our assets were immutable. The truth is somewhere underneath that of course but it's a steer. For ref, Chrome is ~36% of all UA's for us, Safari (mainly mobile/tablet) is ~28%, IE/Edge combined ~15% so Chrome would be a big win for us and I assume that Chrome implementing would have a positive impact on the Safari an MSFT teams decision on whether/not to implement.

To add to #12, we might consider implementing Immutable if the following conditions are met:
1. there are substantial remaining opportunities
2. and they can only be addressed by adding this new API.
In other words, what we are looking for:
1. data showing that there is a significant volume of resources that are inadequately validated (e.g. validation requests for still fresh resources).
2. lack of option to fix this on the Chrome side.
Note: we have some following follow-up work that will further improve the situation: location.reload, reload behavior on offline error pages, etc. See the meta bug for more details: issue 612701.

WRT above - @kenjibaheux@chromium.org, please could you clarify which Chrome version(s)/dates i should look for in our stats to determine whether we have significant numbers of 304's which are unnecessary - i have log data i can get hold of from our CDN traffic. We should have enough traffic to validate with reasonable accuracy.
Cheers

The main change has been live since Chrome 54.
There should be a noticeable drop of 304 when comparing Chrome 53- to 54+.
+toyoshim to provide more details about the additional changes: summary, status, launch plan.

Page reloads that are initiated by window.location.reload() still result in 304s. Is this intentional?
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36

see issue 670237 for details, but in short, I'm a running separate field study to change the location.reload behavior. M57 should have a chrome://flags to enable it explicitly, and 50% users in Canary/Dev should have the feature enabled.

Great to hear immutable will hit Webkit - does this mean it'll make it to Desktop *and* mobile?
Safari (mobile and desktop) constitutes a little under 30% of our audience so i'd love to get immutable for both :-)

Re #21,
Well it's not just for reloading a page, it also affects opening a new tab. Since the assets are immutable chrome would never have to (try-)fetch the assets again. This would result in not only a refresh being faster but also a load without refresh.
If the new reload behaviour works in a similar way then immutable would indeed not be needed but I guess the new behaviour only works for a refresh of a already open page?

just in case the value of immutable is not understood
* its deterministic
* it applies to things other than subresources (i.e. xhr revalidations can become cache hits)
* There are resources that are fresh but not current - immutable lets you disambiguate between them.
I think its fine to apply heuristics as you've done, but it would be better to also accept immutable as input in cases where you were about to do the revalidation anyhow (xhr?)

Does anyone have data to show how immutable can improve performance for e.g, top 100 sites? If other browsers can have such data, that would help us to consider to implement this.
I put Holist-Interop just in case, but this won't cause user-visible compatibility issues.

toyoshim@chromium.org -Ah sorry, i think i see what you are asking for - stats from our CDN on the reduction in 304's pre/post Chrome reload behaviour change. I'll get that as soon as i can and feed back here. Let me know if i misunderstood :-)

FF/immutable vs Chrome/old vs Chrome/new would be nice if possible.
Furtunately, Chrome 56 still have a flag to manage reload behavior via flag.
chrome://flags/#enable-non-validating-reload-on-normal-reload
This may be useful for such comparison, but this flag was already removed in later versions.
Our expectation is new reload achieves similar performance gain as FF can do with immutable.

toyoshim@chromium.org - OK, leave that with me and i will get the data as soon as i have a chance.
I expect (from a little testing i did with Chrome) that you're right though as a user, i agree with Patrick that it would be very helpful to have a level baseline we can rely on i.e. all browsers implementing immutable as a final step.

It's untriaged in the sense that we haven't made a firm decision yet. We are asking for data to show that it's still beneficial despite the changes we did to Reload (adding needs feedback).
Let me know if this is not the right way to do this.

OK, apologies again for the delay - it's a worthy cause though, i've been TLS'ing and H2'ing all the things :-)
I just ran a few tests against our UK home page beta (https://www.bbc.co.uk/home/five/beta) with firefox 52 and chrome 56 (OSX), using OSX network link conditioner in 3G mode (downstream: 780Kbps, 0% loss, 100ms latency, upstream: 330Kbps, 0% loss, 100ms latency). Mean value of 5 page loads after initial cache priming, times are @ domContentLoaded:
Chrome:
Reload optimisation heuristics enabled: 2.5 sec
Reload optimisation heuristics disabled: 4.2 sec
Thus heuristics make it ~40% faster
Firefox:
Regular reload (i.e. using immutable): 1.9 sec
Hard reload (i.e. bypassing immutable): 4.8 sec
Thus immutable make it ~60% faster
I realise there's a ton of nuance/detail here but it seems like immutable might well provide a reasonable chunk of benefit over and above the Chrome heuristics. This confirms my agreement with Patrick.
It's tricky to get CDN stats for the number of 304's pre/post change as i'd have to have logs from pre the change which i don't unfortunately.
Is that what you need or is there anything else?

Is it possible to investigate what kinds of additional network requests Chrome makes on reloads? I expect all resources except for the main html file can be served by disk cache even without immutable support if everything are setup correctly. I'm afraid that Chrome unexpectedly make network requests against files under specific conditions.

+1 to Toyoshim's request.
We can't really conclude from the A/B above that adding support for immutable to Chrome would bring the improvement from 40% to 60%.
Ignoring non-warranted sub-resource validation/requests, the case where immutable would help is when the asset has become stale. I believe that this should be rare because these assets should use max-age: 1 year (maximum value). But I could be missing something.

From what we have so far, there is no evidence that this feature brings significant value over what we have (i.e. the reworked reload which brought the number of our unwarranted validation requests to low levels).

Hi all.
In the past, the browsers whenever possible used the cache.
Meaning, if the browser had the item in cache and it's valid (freshness, authorization, ...) then the browser will use it.
The only exception was a Hard Reload (this implicit says to the browser to ignore the cache).
It’s a binary condition. And the owners of an application know the browser cache behavior.
Now, 25/July/2017, with Google Chrome/Chromium 59.0.3071.115, I get a CSS item with cache-control: public, max-age:<7 days>.
When I do a Normal Reload the browser, sometimes, sends a request to the web Server, and transfers all the CSS again; and sometimes it retrieves the CSS from cache.
But for the same page I have a JS item with exactly the same cache-control directive, and it always retrieves the item from cache. See picture in attached.
It’s a random condition:
• Links/Normal Reload: some items use cache, some don’t, sometimes;
• Hard Reload: Doesn’t use cache.
I (and imagine most people) don’t understand this cache behavior.
What item types respect the cache and what types doesn’t?
Why doesn’t Chrome respect the max-age directive?
Is this because I have only max-age= 7 days? Is this because it’s HTTPS?
Where is the algorithm description for the browser choosing from cache or not, so the application developers can tune accordingly?
I imagine that one of the reasons Mozilla created the “immutable” tag has to say: If the item as the immutable tag and it’s a Normal Reload, then get from local cache.
This will allow application developers to once again have control over the browser cache behavior.
(Although the main advantage of immutable, is that the browser does less calculations because it doesn’t have to calculate if the item in cache, is expired or not).
The application developers should have an option to say “get this item from cache”, and the browser should respect that. The immutable tag could (also) be a solution for this.
Regards, Tiago Abreu.

#47 - it just sounds like a bug. Please, file an issue for it (if you could not find one already), along with a URL to reproduce it in order to alleviate any server issues. You can comment here with the issue number.