At this years DConf I gave a very short overview of the
benchmarking library I'm working on [1]. The source can be found
here [2]
The general idea goes as following.
Unit tests in D help us to find bugs and avoid regressions, but
they do not cover performance regressions. That is bad, we are
basically, currently just hoping that performance diminishes.
So my idea is to eventually get this library into phobos, write
benchmarks for all functions in phobos, execute the benchmarks
for every merge into master, use gnuplot to display the results
on the dlang webpage, profit.
Right now I need people to give feedback on the library. You can
find it on code.dlang.org [3]
[1]
https://www.youtube.com/watch?v=QELK73JSpFk&list=PL3jwVPmk_PRxo23yyoc0Ip_cP3-rCm7eB&index=30
[2] https://github.com/burner/benchmark
[3] http://code.dlang.org/packages/std_benchmark

On Thursday, 1 June 2017 at 21:08:43 UTC, Robert burner Schadek
wrote:

So my idea is to eventually get this library into phobos, write
benchmarks for all functions in phobos, execute the benchmarks
for every merge into master, use gnuplot to display the results
on the dlang webpage

And reject out of hand any PR which reduces performance in order
to "ratchet" performance over time.
I would also do this to a lesser degree to DMD as well.

On Thursday, 1 June 2017 at 21:08:43 UTC, Robert burner Schadek wrote:

So my idea is to eventually get this library into phobos, write
benchmarks for all functions in phobos, execute the benchmarks for
every merge into master, use gnuplot to display the results on the
dlang webpage

And reject out of hand any PR which reduces performance in order to
"ratchet" performance over time.
I would also do this to a lesser degree to DMD as well.

That's indeed worth it. Robert, we discussed first deploying the library
and a driver for dmd/druntime/phobos on dub, is this your plan? Thanks!
-- Andrei

That's indeed worth it. Robert, we discussed first deploying
the library and a driver for dmd/druntime/phobos on dub, is
this your plan? Thanks! -- Andrei

I think a practical approach is to create another dub project
that solely consists of benchmarks for functions of phobos, and
prints pretty pictures of their performance.
This will properly help to find shortcomings of the benchmark
library, and help to develop idioms for benchmarking and random
testing.
If that works out, the next step is properly to integrate that in
the phobos/druntime/dmd CI. Which properly means putting the
benchmark library through the experimental integration process,
and finding a place for the benchmarks to life.

If that works out, the next step is properly to integrate that
in the phobos/druntime/dmd CI. Which properly means putting the
benchmark library through the experimental integration process,
and finding a place for the benchmarks to life.

As I've pointed out before, having the benchmark support code be
part of the public standard library is not required for the
Phobos project to make use of it internally.
— David

And reject out of hand any PR which reduces performance in
order to "ratchet" performance over time.
I would also do this to a lesser degree to DMD as well.

I think being binary about it is the wrong approach.
Lets say we fix a bug, and the function is slower afterwards. Do
we merge the bugfix? I say yes.
Lets say we add a feature to a function that makes more useful.
Do we merge it? I say yes.
This is just another hammer, it does not mean everything is a
nail now.

So my idea is to eventually get this library into phobos, write
benchmarks for all functions in phobos, execute the benchmarks for every
merge into master, use gnuplot to display the results on the dlang
webpage, profit.

In Xcode you can actually have benchmarks that behave as tests. That is,
you write your benchmark. Define a boundary for a set of platforms, if
the benchmark takes longer to run then the boundary it will fail the
test. Then you don't need anyone to constantly look at the result. Of
course, this only work if you have a limited set of platforms with a
consistent performance.
--
/Jacob Carlborg