On Thu, 26 Jun 2025 at 18:23, Sasha Levin <sashal@xxxxxxxxxx> wrote: > > On Thu, Jun 26, 2025 at 10:37:33AM +0200, Dmitry Vyukov wrote: > >On Thu, 26 Jun 2025 at 10:32, Dmitry Vyukov <dvyukov@xxxxxxxxxx> wrote: > >> > >> On Wed, 25 Jun 2025 at 17:55, Sasha Levin <sashal@xxxxxxxxxx> wrote: > >> > > >> > On Wed, Jun 25, 2025 at 10:52:46AM +0200, Dmitry Vyukov wrote: > >> > >On Tue, 24 Jun 2025 at 22:04, Sasha Levin <sashal@xxxxxxxxxx> wrote: > >> > > > >> > >> >6. What's the goal of validation of the input arguments? > >> > >> >Kernel code must do this validation anyway, right. > >> > >> >Any non-trivial validation is hard, e.g. even for open the validation function > >> > >> >for file name would need to have access to flags and check file precense for > >> > >> >some flags combinations. That may add significant amount of non-trivial code > >> > >> >that duplicates main syscall logic, and that logic may also have bugs and > >> > >> >memory leaks. > >> > >> > >> > >> Mostly to catch divergence from the spec: think of a scenario where > >> > >> someone added a new param/flag/etc but forgot to update the spec - this > >> > >> will help catch it. > >> > > > >> > >How exactly is this supposed to work? > >> > >Even if we run with a unit test suite, a test suite may include some > >> > >incorrect inputs to check for error conditions. The framework will > >> > >report violations on these incorrect inputs. These are not bugs in the > >> > >API specifications, nor in the test suite (read false positives). > >> > > >> > Right now it would be something along the lines of the test checking for > >> > an expected failure message in dmesg, something along the lines of: > >> > > >> > https://github.com/linux-test-project/ltp/blob/0c99c7915f029d32de893b15b0a213ff3de210af/testcases/commands/sysctl/sysctl02.sh#L67 > >> > > >> > I'm not opposed to coming up with a better story... > > > >If the goal of validation is just indirectly validating correctness of > >the specification itself, then I would look for other ways of > >validating correctness of the spec. > >Either removing duplication between specification and actual code > >(i.e. generating it from SYSCALL_DEFINE, or the other way around) , > >then spec is correct by construction. Or, cross-validating it with > >info automatically extracted from the source (using > >clang/dwarf/pahole). > >This would be more scalable (O(1) work, rather than thousands more > >manually written tests). > > > >> Oh, you mean special tests for this framework (rather than existing tests). > >> I don't think this is going to work in practice. Besides writing all > >> these specifications, we will also need to write dozens of tests per > >> each specification (e.g. for each fd arg one needs at least 3 tests: > >> -1, valid fd, inclid fd; an enum may need 5 various inputs of > >> something; let alone netlink specifications). > > I didn't mean just for the framework: being able to specify the APIs in > machine readable format will enable us to automatically generate > exhaustive tests for each such API. > > I've been playing with the kapi tool (see last patch) which already > supports different formatters. Right now it outputs human readable > output, but I have proof-of-concept code that outputs testcases for > specced APIs. > > The dream here is to be able to automatically generate > hundreds/thousands of tests for each API in an automated fashion, and > verify the results with: > > 1. Simply checking expected return value. > > 2. Checking that the actual action happened (i.e. we called close(fd), > verify that `fd` is really closed). > > 3. Check for side effects (i.e. close(fd) isn't supposed to allocate > memory - verify that it didn't allocate memory). > > 4. Code coverage: our tests are supposed to cover 100% of the code in > that APIs call chain, do we have code that didn't run (missing/incorrect > specs). This is all good. I was asking the argument verification part of the framework. Is it required for any of this? How?
