Patrick Steinhardt <ps@xxxxxx> writes:
> Wire up benchmarks in Meson. The setup is mostly the same as how we wire
> up our tests. The only difference is that benchmarks get wired up via
> the `benchmark()` option instead of via `test()`, which gives them a bit
> of special treatment:
>
> - Benchmarks never run in parallel.
>
> - Benchmarks aren't run by default when tests are executed.
>
> - Meson does not inject the `MALLOC_PERTURB` environment variable.
>
> Using benchmarks is quite simple:
>
> ```
> $ meson setup build
> # Run all benchmarks.
> $ meson test -C build --benchmark
> # Run a specific benchmark.
> $ meson test -C build --benchmark p0000-*
> ```
I really appreciate the efforts, and I'd love to run the perf tests with
Meson, but at the moment I don't know how these changes are useful
because this doesn't print anything relevant:
$ meson test -C build --benchmark 'p0005-*'
ninja: Entering directory `/home/toon/devel/git/build'
[1/28] Generating GIT-VERSION-FILE with a custom command (wrapped by meson to set env)
1/1 p0005-status OK 7.39s
Ok: 1
Expected Fail: 0
Fail: 0
Unexpected Pass: 0
Skipped: 0
Timeout: 0
Full log written to /home/toon/devel/git/build/meson-logs/testlog.txt
> Other than that the usual command line arguments accepted when running
> tests are also accepted when running benchmarks.
>
> Note that the benchmarking target is somewhat limited because it will
> only run benchmarks for the current build. Other usecases, like running
> benchmarks against multiple different versions of Git, are not currently
> supported. Users should continue to use "t/perf/run" for those usecases.
> The script should get extended at one point in time to support Meson,
> but this is outside of the scope of this series.
Yeah, this is unfortunate, but totally understandable. I've been digging
in `t/perf/run` and `f/perf/aggregate.perl` and it doesn't look easy to
adapt to Meson.
But I was wondering, instead of trying to fully integrate Meson into
those scripts, could we modify the scripts so they work with binaries
built by Meson? I mean, if we could run
`cd t/perf && ./run ../../build1 ../../build2 p0005*` and it would
simply run the benchmarks in those directories (without trying to check
out code and build the sources). I think this would help a lot already.
--
Toon
