Since Raptor tests require to add marks in code, if we do that without any dynamic patching way, Gaia codebase would be full with `Performance.mark` without any visible & meaningful relations among them. The better way is to collect all these marks into some patching files, and apply them only when we want to run the specific Raptor test. For example, if we want to test LockScreen performance from locking to unlocking automatically, we may need to patch two functions with such marks:
LockScreen#lock = function() {
Performance.mark('LockScreen#lock')
// ....
}
LockScreen#unlock = function() {
Performance.mark('LockScreen#unlock');
// ....
}
We can collect them in a file `lockscreen-lock-unlock.esp.js`:
Espect.select('lockscreen.js LockScreen#lock')
.before(function() {
Performance.mark('LockScreen#lock')
})
Espect.select('lockscreen.js LockScreen#unlock')
.before(function() {
Performance.mark('LockScreen#unlock')
})
(The Espect is a tool written by me for the purpose to patch Gaia during build time)
https://github.com/snowmantw/espect.js
And apply such `.esp.js` file only when we want to run such Raptor test. So the codebase keeps clean from such marks, and since we collect all related marks in the same file, we know they're for the same test. This is useful especially when we want to test multiple functions across different apps (ex: to test LockScreen unlocking to Homescreen shows up). The whole idea is here:
https://docs.google.com/presentation/d/1DemVx_963IX19LVWE0THKQKm9S40X24XGOAtwOhupA0/edit?usp=sharing
I now have a WIP for that. How to use it is:
1. Apply the patch to your Gaia or just checkout my branch
2. npm install
3. as Commit message: APP=<your app, like system> DEBUGUY=<where the debuguy is> TESTDIR=<where the esp files are> make
After that, you can check the file your `.esp.js` is for would be patched. There is one demo esp file under `/apps/system/lockscreen/raptor`, which should patch the `lockscreen.js`. The sample make command is:
APP=system DEBUGUY=gaia/node_modules/debuguy TESTDIR=gaia/apps/system/lockscreen/raptor make
I would update the patch to make it ready.
(BTW, Debuguy is another tool developed by our device team. It's used here to perform multiple dynamic patching at once).
https://github.com/seiyugi/debuguy

I've update the patch. Now the command is:
APP=<app> TESTDIR=<where the test dir is> RAPTOR_TRANSFORM=1 make
Can still specify the debuguy path with an additional option:
RAPTOR_TRANSFORMER_PATH=<path to debuguy>
But it's optional.

And I now added one build test: if someone change the code to make espect couldn't match new code, like the file got moved away or removed, the test would fail and we can catch such regression. It's now in 'system/test/build/integration/raptor_transform.js'.

I could put the files in Gaia repo, but what confuse me is it may still depend on other 3rd packages that may also have their own deep dependencies. So if I really need to check it in Gaia, this means I also need to check-in the whole node_modules after I install it locally. However, I don't see this happens in the current Gaia repo, so I don't know if it's legit.
And in my recollection there is a taskcluster-npm-mirroring or cache server to serve such requests during npm install. I've re-read the thread on mailing list again, but I still don't know if it works on b2g-inbound or mozilla-central. And Ricky and I just don't get if it works like:
1. register our packages in package.json as usual
2. when package.json in Gaia updated and when it's trying on Taskcluster, the cache server would mirror all necessary packages when it's first time installing those modules
3. after the cache built, the dependencies on CI server are done, so there is no need to fetch packages from external network anymore
Since Bug 1141792 it describes to remove node module tarball, but what we should do to make this work is a bit confused. And I don't know if check-in the whole module repo into Gaia is not conflicting with such solution.

NI Aus because I still don't know which approach is correct: let Taskcluster NPM cache to handle the package Gaia depends on? Or manually parsing all dependencies of such module and land them all in Gaia repo?

I can't see where you're adding node module dependencies in your patch. Are you sure it's up to date?
NPM cache works on *ALL* trees. That's why it's not specified in the email where it works. You should let it take care of your dependencies. Just make sure the dependencies are present in the appropriate package.json.

Comment on attachment 8634587[details][review]
[gaia] snowmantw:bug1181069 > mozilla-b2g:master
Sorry I forgot: since Tim and me may be the first few user of this feature, I add feedback to him. I would also fix what Tim says before landing.
For now it could be tested with:
APP=<app> TESTDIR=<where the test dir is> RAPTOR_TRANSFORM=1 make
For example:
APP=system TESTDIR=gaia/apps/system/lockscreen/raptor RAPTOR_TRANSFORM=1 make
And the profile would be transformed with the code of 'performance.mark'. (I forgot it's lowercase and would update in the patch. Anyway the architecture and the flow works).