On Wed, 11 Jun 2025 at 15:38, Thomas Weißschuh <thomas.weissschuh@xxxxxxxxxxxxx> wrote: > > Currently testing of userspace and in-kernel API use two different > frameworks. kselftests for the userspace ones and Kunit for the > in-kernel ones. Besides their different scopes, both have different > strengths and limitations: > > Kunit: > * Tests are normal kernel code. > * They use the regular kernel toolchain. > * They can be packaged and distributed as modules conveniently. > > Kselftests: > * Tests are normal userspace code > * They need a userspace toolchain. > A kernel cross toolchain is likely not enough. > * A fair amout of userland is required to run the tests, > which means a full distro or handcrafted rootfs. > * There is no way to conveniently package and run kselftests with a > given kernel image. > * The kselftests makefiles are not as powerful as regular kbuild. > For example they are missing proper header dependency tracking or more > complex compiler option modifications. > > Therefore kunit is much easier to run against different kernel > configurations and architectures. > This series aims to combine kselftests and kunit, avoiding both their > limitations. It works by compiling the userspace kselftests as part of > the regular kernel build, embedding them into the kunit kernel or module > and executing them from there. If the kernel toolchain is not fit to > produce userspace because of a missing libc, the kernel's own nolibc can > be used instead. > The structured TAP output from the kselftest is integrated into the > kunit KTAP output transparently, the kunit parser can parse the combined > logs together. > > Further room for improvements: > * Call each test in its completely dedicated namespace > * Handle additional test files besides the test executable through > archives. CPIO, cramfs, etc. > * Compatibility with kselftest_harness.h (in progress) > * Expose the blobs in debugfs > * Provide some convience wrappers around compat userprogs > * Figure out a migration path/coexistence solution for > kunit UAPI and tools/testing/selftests/ > > Output from the kunit example testcase, note the output of > "example_uapi_tests". > > $ ./tools/testing/kunit/kunit.py run --kunitconfig lib/kunit example > ... > Running tests with: > $ .kunit/linux kunit.filter_glob=example kunit.enable=1 mem=1G console=tty kunit_shutdown=halt > [11:53:53] ================== example (10 subtests) =================== > [11:53:53] [PASSED] example_simple_test > [11:53:53] [SKIPPED] example_skip_test > [11:53:53] [SKIPPED] example_mark_skipped_test > [11:53:53] [PASSED] example_all_expect_macros_test > [11:53:53] [PASSED] example_static_stub_test > [11:53:53] [PASSED] example_static_stub_using_fn_ptr_test > [11:53:53] [PASSED] example_priv_test > [11:53:53] =================== example_params_test =================== > [11:53:53] [SKIPPED] example value 3 > [11:53:53] [PASSED] example value 2 > [11:53:53] [PASSED] example value 1 > [11:53:53] [SKIPPED] example value 0 > [11:53:53] =============== [PASSED] example_params_test =============== > [11:53:53] [PASSED] example_slow_test > [11:53:53] ======================= (4 subtests) ======================= > [11:53:53] [PASSED] procfs > [11:53:53] [PASSED] userspace test 2 > [11:53:53] [SKIPPED] userspace test 3: some reason > [11:53:53] [PASSED] userspace test 4 > [11:53:53] ================ [PASSED] example_uapi_test ================ > [11:53:53] ===================== [PASSED] example ===================== > [11:53:53] ============================================================ > [11:53:53] Testing complete. Ran 16 tests: passed: 11, skipped: 5 > [11:53:53] Elapsed time: 67.543s total, 1.823s configuring, 65.655s building, 0.058s running > > Based on v6.15-rc1. > > Signed-off-by: Thomas Weißschuh <thomas.weissschuh@xxxxxxxxxxxxx> > --- > Changes in v3: > - Reintroduce CONFIG_CC_CAN_LINK_STATIC > - Enable CONFIG_ARCH_HAS_NOLIBC for m68k and SPARC > - Properly handle 'clean' target for userprogs > - Use ramfs over tmpfs to reduce dependencies > - Inherit userprogs byte order and ABI from kernel > - Drop now unnecessary "#ifndef NOLIBC" > - Pick up review tags > - Drop usage of __private in blob.h, > sparse complains and it is not really necessary > - Fix execution on loongarch when using clang > - Drop userprogs libgcc handling, it was ugly and is not yet necessary > - Link to v2: https://lore.kernel.org/r/20250407-kunit-kselftests-v2-0-454114e287fd@xxxxxxxxxxxxx > > Changes in v2: > - Rebase onto v6.15-rc1 > - Add documentation and kernel docs > - Resolve invalid kconfig breakages > - Drop already applied patch "kbuild: implement CONFIG_HEADERS_INSTALL for Usermode Linux" > - Drop userprogs CONFIG_WERROR integration, it doesn't need to be part of this series > - Replace patch prefix "kconfig" with "kbuild" > - Rename kunit_uapi_run_executable() to kunit_uapi_run_kselftest() > - Generate private, conflict-free symbols in the blob framework > - Handle kselftest exit codes > - Handle SIGABRT > - Forward output also to kunit debugfs log > - Install a fd=0 stdin filedescriptor > - Link to v1: https://lore.kernel.org/r/20250217-kunit-kselftests-v1-0-42b4524c3b0a@xxxxxxxxxxxxx > > --- Thanks very much for persisting with this, and sorry for the delayed response. I've taken quite a liking to it: it'd definitely have made my life easier more than once. As a more general wrapping of selftests in KUnit tests, I do think that there's still some risk of confusion as to when a KUnit UAPI test makes sense versus a simple selftest. The UAPI tests are definitely (IMO) easier to build and run, but won't be easier to debug, or to run on an existing, non-test system as a part of troubleshooting (which has been a complaint when selftests have been ported to KUnit in the past). Nevertheless, I'm pretty happy to have this be a part of KUnit, though I have three slight reservations: 1. There's no real _user_ of this yet -- save for the small test of /proc/self/comm in the example -- I'd like to see a real-world test using this. 2. There's a fair bit of complexity here, and we're already a bit behind with KUnit reviews. I'd love it if you could commit to helping maintain the KUnit parts of this in MAINTAINERS. 3. We need to make sure that there's a clear understanding of when to use this, versus in-kernel KUnit tests, versus kselftest. This'll probably involve (a) making sure Shuah is on board -- or at least not strongly opposed, and (b) updating Documentation/dev-tools/testing-overview.rst. But thanks very much -- it's working well in my testing here, and running the tests is very pleasant. Cheers, -- David > Thomas Weißschuh (16): > kbuild: userprogs: avoid duplicating of flags inherited from kernel > kbuild: userprogs: also inherit byte order and ABI from kernel > init: re-add CONFIG_CC_CAN_LINK_STATIC > kbuild: userprogs: add nolibc support > kbuild: introduce CONFIG_ARCH_HAS_NOLIBC > kbuild: doc: add label for userprogs section > kbuild: introduce blob framework > kunit: tool: Add test for nested test result reporting > kunit: tool: Don't overwrite test status based on subtest counts > kunit: tool: Parse skipped tests from kselftest.h > kunit: Always descend into kunit directory during build > kunit: qemu_configs: loongarch: Enable LSX/LSAX > kunit: Introduce UAPI testing framework > kunit: uapi: Add example for UAPI tests > kunit: uapi: Introduce preinit executable > kunit: uapi: Validate usability of /proc > > Documentation/dev-tools/kunit/api/index.rst | 5 + > Documentation/dev-tools/kunit/api/uapi.rst | 12 + > Documentation/kbuild/makefiles.rst | 38 ++- > MAINTAINERS | 2 + > Makefile | 7 +- > include/kunit/uapi.h | 24 ++ > include/linux/blob.h | 31 +++ > init/Kconfig | 7 + > lib/Makefile | 4 - > lib/kunit/Kconfig | 10 + > lib/kunit/Makefile | 20 +- > lib/kunit/kunit-example-test.c | 15 ++ > lib/kunit/kunit-example-uapi.c | 54 ++++ > lib/kunit/uapi-preinit.c | 63 +++++ > lib/kunit/uapi.c | 294 +++++++++++++++++++++ > scripts/Makefile.blobs | 19 ++ > scripts/Makefile.build | 6 + > scripts/Makefile.clean | 2 +- > scripts/Makefile.userprogs | 13 +- > scripts/blob-wrap.c | 27 ++ > tools/include/nolibc/Kconfig.nolibc | 15 ++ > tools/testing/kunit/kunit_parser.py | 13 +- > tools/testing/kunit/kunit_tool_test.py | 9 + > tools/testing/kunit/qemu_configs/loongarch.py | 2 + > .../test_is_test_passed-failure-nested.log | 10 + > .../test_data/test_is_test_passed-kselftest.log | 3 +- > 26 files changed, 686 insertions(+), 19 deletions(-) > --- > base-commit: f07a3558c4a5d76f3fea004075e5151c4516d055 > change-id: 20241015-kunit-kselftests-56273bc40442 > > Best regards, > -- > Thomas Weißschuh <thomas.weissschuh@xxxxxxxxxxxxx> >
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
