We need to create a new testrunner to run the Endurance Tests. This will take an argument for the number of iterations the test snippets will be repeated, and include the metrics gathered during the test in the report.

This patch includes an additional testrunner for the endurance tests project, which takes two command line arguments: a delay in milliseconds to wait before starting each iteration, and the number of iterations to perform. The endurance results are added to the report. Currently the endurance results will be stored in memory, however it is intended that we will cache these on disk in the future.

Comment on attachment 507192[details][diff][review]
Testrunner script for endurance tests v1
Looks good to me from here. I'll admit that I'm not the most expert on the python wrappers, though, so I'll be interested to hear whimboo's feedback.

Comment on attachment 508374[details][diff][review]
Testrunner script for endurance tests v1.1
>+ self._mozmill.persisted['endurance'] = {'delay': self.delay,
>+ 'iterations': self.iterations,
>+ 'results': self.results}
Do we really have to set results here? I wonder how the endurance tests work if you wanna debug on test and the results property is not available. Can this last line be removed? That would also make the class member 'results' obsolete, which is also not necessary.
>+# Portions created by the Initial Developer are Copyright (C) 2010
2011 please.
>+base_path = os.path.dirname(os.path.abspath(__file__))
>+sys.path.append(os.path.join(base_path, 'libs'))
>+
>+from testrun import EnduranceTestRun
Importing of dependent modules has been already changed in the other scripts. Please update accordingly.
>+ parser.add_option("--delay",
>+ default=0,
>+ dest="delay",
>+ metavar="INT",
Metavar should be "DELAY"
>+ parser.add_option("--iterations",
>+ default=1,
>+ dest="iterations",
>+ metavar="INT",
Similar here.
Otherwise looks good. For the next cycle I will also execute a test-run. Looks like we can get this landed today.

(In reply to comment #6)
> Comment on attachment 508374[details][diff][review]
> Testrunner script for endurance tests v1.1
>
> >+ self._mozmill.persisted['endurance'] = {'delay': self.delay,
> >+ 'iterations': self.iterations,
> >+ 'results': self.results}
>
> Do we really have to set results here? I wonder how the endurance tests work if
> you wanna debug on test and the results property is not available. Can this
> last line be removed? That would also make the class member 'results' obsolete,
> which is also not necessary.
If I remove this the tests will fail. I'd need to first patch the endurance shared module so that it doesn't attempt to set the _results variable to the value of this. I have removed the class member 'results'.
> >+# Portions created by the Initial Developer are Copyright (C) 2010
>
> 2011 please.
Done.
> >+base_path = os.path.dirname(os.path.abspath(__file__))
> >+sys.path.append(os.path.join(base_path, 'libs'))
> >+
> >+from testrun import EnduranceTestRun
>
> Importing of dependent modules has been already changed in the other scripts.
> Please update accordingly.
Done.
> >+ parser.add_option("--delay",
> >+ default=0,
> >+ dest="delay",
> >+ metavar="INT",
>
> Metavar should be "DELAY"
Done.
> >+ parser.add_option("--iterations",
> >+ default=1,
> >+ dest="iterations",
> >+ metavar="INT",
>
> Similar here.
Set to ITERATIONS, however I'm not sure I like how this forced a new line when using --help

Comment on attachment 513166[details][diff][review]
Updated shared-module for removal of empty results array in persisted object. v1.2.1
>+ this._results = [];
> if (persisted.endurance) {
> this._delay = persisted.endurance.delay;
> this._iterations = persisted.endurance.iterations;
>- this._results = persisted.endurance.results;
>+ persisted.endurance.results = this._results;
The last line is a bit suspicious here, at least for the moment and as long as we don't write the results to the disk in the automation script. This change would reset the results array of the persisted object each time a new endurance test gets executed. So you will lose all the formerly collected information.
I would suggest that we have a short call later and can figure that out on the phone. Seems like we have to change a bit.

(In reply to comment #14)
> Comment on attachment 513166[details][diff][review]
> Updated shared-module for removal of empty results array in persisted object.
> v1.2.1
>
> >+ this._results = [];
> > if (persisted.endurance) {
> > this._delay = persisted.endurance.delay;
> > this._iterations = persisted.endurance.iterations;
> >- this._results = persisted.endurance.results;
> >+ persisted.endurance.results = this._results;
>
> The last line is a bit suspicious here, at least for the moment and as long as
> we don't write the results to the disk in the automation script. This change
> would reset the results array of the persisted object each time a new endurance
> test gets executed. So you will lose all the formerly collected information.
>
> I would suggest that we have a short call later and can figure that out on the
> phone. Seems like we have to change a bit.
So this is the reason I needed the empty results array in the persisted object. There were several issues with removing this as suggested in comment 7. I have reverted this and tested locally.

Comment on attachment 513317[details][diff][review]
Testrunner script for endurance tests v1.3
>+import os
>+import sys
Please remove both of those imports because they aren't necessary anymore. Then we are good to land.
r=me with that change.