I'm afraid I've mucked this up. I'm not new to experimenting with JJB but I don't have much Docker experience. I think the only things I changed were to try running the container as (what turned out to be) a nonexistent user, node, and to run an interactive session with -it bash in which I performed read only commands like cd and ls. I've since wiped the workspace, reset the Jenkins config as usual with jenkins_jobs, and tried it on different Jenkins nodes but the build now succeeds every time so I must conclude that I've altered the container state some how. How can I "reset" the container back to its initial state? Obviously, fixing the problem is good but I want to do so in a well-defined and permanent way.

Downloading the NPM version (v65.0.3312.0) directly into the container src/ mount (and unzipping and referencing the local executable-- e.g., see patchset) works. Docker command used (after temporarily disabling the --rm and clear JJB commands and a fresh build):

Update the Docker NPM image (npm-browser-test) to use the latest version of Chromium from apt. I believe this requires releng privileges and impacts other jobs.

Accept that downloading Chromium is wasteful but working. sshing into integration-slave-docker-1005, wget https://storage.googleapis.com/chromium-browser-snapshots/Linux_x64/526987/chrome-linux.zip consistently completed in less than a second. In actual CI builds, it's still less than 10 seconds. There are only 33 patchsets so far and I don't think we'll be making so many more. There are better things we can work on.

Override PUPPETEER_DOWNLOAD_HOST instead of PUPPETEER_SKIP_CHROMIUM_DOWNLOAD and point to a WMF maintained repo of Chromium builds. This ticket will need discussion and possibly re-estimation if we go this route.

@hashar since you built the original image, does that seem ok to you? Seems like we'll need to test the updated image with the 12 existing tests that utilize this image to ensure there aren't regressions which is time-consuming, but that's the only problem I can foresee...

Update the Docker NPM image (npm-browser-test) to use the latest version of Chromium from apt. I believe this requires releng privileges and impacts other jobs.

That is what we should do. With ideally one day having the CI container rebuild automatically whenever a new Chromium is made available in Debian Stretch. It is correct that only releng/roots can get the containers deployed on the CI infrastructure.

Accept that downloading Chromium is wasteful but working. sshing into integration-slave-docker-1005, wget https://storage.googleapis.com/chromium-browser-snapshots/Linux_x64/526987/chrome-linux.zip consistently completed in less than a second. In actual CI builds, it's still less than 10 seconds. There are only 33 patchsets so far and I don't think we'll be making so many more. There are better things we can work on.

The chromium-render in production would run with the version from Debian Stretch. So supposedly it is nicer to test with that version instead of a randomly downloaded one. At some point the service will be run in a container which would have Chromium installed from the Debian package as well. So that is why we have instructed puppeter to skip downloading chromium entirely. From a past comment:

Override PUPPETEER_DOWNLOAD_HOST instead of PUPPETEER_SKIP_CHROMIUM_DOWNLOAD and point to a WMF maintained repo of Chromium builds. This ticket will need discussion and possibly re-estimation if we go this route.

Building our own Chromium is challenging. It is not that easy to build and maintain, it is way better to share the burden / delegate that task to Debian.org.

Which is what we would get if we rebuilt the `npm-browser-test` container again today, seemingly: https://github.com/wikimedia/integration-config/blob/master/dockerfiles/npm-browser-test/Dockerfile.template#L4
If that version (64) behaves as expected (am I reading that correctly?), then this may be solvable via a simple changelog bump https://github.com/wikimedia/integration-config/blob/master/dockerfiles/npm-browser-test/changelog#L1 to trigger a rebuild of the package.
@hashar since you built the original image, does that seem ok to you? Seems like we'll need to test the updated image with the 12 existing tests that utilize this image to ensure there aren't regressions which is time-consuming, but that's the only problem I can foresee...

Yes that is exactly it. Touching the changelog file would cause docker-pkg on contint1001 to rebuild the container from scratch and thus get whatever latest versions are available in the apt repository.

Seems like we can move the Jenkins job to test pipeline so it get runs on every new patchset?

I've made the promotion here. For production and development reasons described in the patchset, I've moved @hashar's Puppeteer download disablement to the Jenkins Job Builder config. Thanks for all the help!

@Jdrewniak, would you mind QAing this guy and signing it off? Basically, you submit a patch to the chromium-render repo and see that it passes or fails as expected. Additionally, you should see in the Jenkins logs explicit mention that Chromium download has been skipped! If you have _any_ trouble, please ping me! Thank you!