On Thu, Aug 07, 2025 at 02:14:13PM +0200, Pablo Neira Ayuso wrote: > On Thu, Aug 07, 2025 at 01:51:09PM +0200, Phil Sutter wrote: > > On Wed, Aug 06, 2025 at 07:05:02PM +0200, Pablo Neira Ayuso wrote: > > > On Fri, Aug 01, 2025 at 06:11:05PM +0200, Phil Sutter wrote: > > > > Add the various testsuite runners to TESTS variable and have make call > > > > them with RUN_FULL_TESTSUITE=1 env var. > > > > > > > > Signed-off-by: Phil Sutter <phil@xxxxxx> > > > > --- > > > > Makefile.am | 6 ++++++ > > > > 1 file changed, 6 insertions(+) > > > > > > > > diff --git a/Makefile.am b/Makefile.am > > > > index ba09e7f0953d5..4fb75b85a5d59 100644 > > > > --- a/Makefile.am > > > > +++ b/Makefile.am > > > > @@ -409,5 +409,11 @@ EXTRA_DIST += \ > > > > tests \ > > > > $(NULL) > > > > > > > > +AM_TESTS_ENVIRONMENT = RUN_FULL_TESTSUITE=1; export RUN_FULL_TESTSUITE; > > > > > > I use make distcheck to build the tarballs. > > > > > > I would prefer not to run the tests at the time of the release > > > process, I always do this before release, but I prefer not to inline > > > this to the release process. > > > > Oh, good to know. Running just 'make dist' is no option for you? > > I can just modify the script to do so, no idea on the implications. There is more to distcheck than just the 'make check' call, so it's definitely worth doing it. The best option might be to run 'make distcheck' before the release for a complete test run and only 'make dist' during the release process. Though this requires to run 'make distcheck' as root, not sure if that is a good idea. > Or wait for one more hour for the test run to finish during the > release process. Does not seem feasible, especially for a redundant test run you're not interested in. It also implies that you're creating the distribution on a system which is able to pass (or skip) all tests, which may not be the case. > > BTW: There is the same situation with iptables, though if called as > > unprivileged user there is only the xlate test suite which runs (and > > quickly finishes). > > > > > Maybe we can make this work this way? > > > > > > export RUN_FULL_TESTSUITE=1; make check > > > > > > so make check is no-op without this variable? > > > > > > Does this make sense to you? > > > > It seems odd to enable 'make check' only to disable it again, but > > there's still added value in it. > > > > I'm currently looking into distcheck-hook and DISTCHECK_CONFIGURE_FLAGS > > in order to identify the caller to 'make check' call. > > > > An alternative would be to drop fake root functionality from shell > > test suite, then it would skip just like all the other test suites if run > > as non-root (assuming you don't run 'make distcheck' as root). > > Looking at the current release script I have, it all runs as non-root. > > Maybe simply as a run-all.sh under nftables/tests/? Also an option, yes. Or a custom 'make testrun' or so. > It is not so elegant as make check, but still allows for running _all_ > tests with minimal changes. A cool feature of automake is to run tests in parallel. I have a local branch which implements that for iptables. Makefile.am has to list every test case though, which adds overhead (and room for errors) when writing new tests. > tests/build is slow but very useful. Yes, I don't see a way to instruct 'make distcheck' to perform multiple builds with different configure options. tests/build is funny: It runs 'make distcheck' itself, so adding tests/build/run-tests.sh to TESTS variable in Makefile.am should lead to an infinite distcheck loop. :D > > Another side-quest extracted from this mail: There's an odd failure from > > shell test suite when run by 'make distcheck': > > > > | E: cannot execute nft command: ../../tests/shell/../../src/nft > > | ERROR tests/shell/run-tests.sh (exit status: 99) This happens due to the separate source and build directories, the compiled 'nft' tool (actually: wrapper) is not in the same tree as the test runner. Cheers, Phil
