On 5/26/25 8:44 AM, Patrick Steinhardt wrote: > On Fri, May 23, 2025 at 12:33:23PM -0700, Junio C Hamano wrote: >> Ramsay Jones <ramsay@xxxxxxxxxxxxxxxxxxxx> writes: >> >>> Question: should meson (or indeed prove) fail the test because of an >>> unexpected _pass_? >> >> Yes, it is a very good question. I do not mind if the answer is "it >> should, and the make and prove shouldn't let unexpected pass go >> unnoticed". The difference between the build systems bothers me > > Indeed, a good question. The TAP specification [1] has this to say: > > Should a todo test point begin succeeding, the harness may report it > in some way that indicates that whatever was supposed to be done has > been, and it should be promoted to a normal Test Point. > > Harnesses must not treat failing TODO test points as a test failure. > > Harneses should report TODO test points found as a list of items > needing work, if that is appropriate for their use case. > > So if my reading of this is correct then Meson isn't wrong in reporting > this as an error -- "in some way" basically gives it full permission to > do so. So this is plain old undefined behaviour we rely on :/ I don't see how you can possibly compare this to UB. It is a documented variant behavior and thus "implementation-defined", not "undefined". Also, https://www.gnu.org/software/automake/manual/automake.html#Generalities-about-Testing """ It’s not uncommon, especially during early development stages, that some tests fail for known reasons, and that the developer doesn’t want to tackle these failures immediately (this is especially true when the failing tests deal with corner cases). In this situation, the better policy is to declare that each of those failures is an expected failure (or xfail). In case a test that is expected to fail ends up passing instead, many testing environments will flag the result as a special kind of failure called unexpected pass (or xpass). """ > I don't think it's inherently a bad thing to fail on unexpected passes. > After all, it shows that our assumption that the test fails is broken, > and that we should have a look why that is. But I can see arguments both > ways. As Phillip noted, treating them as ordinary passes undermines the reason for having them. -- Eli Schwartz
Attachment:
OpenPGP_signature.asc
Description: OpenPGP digital signature
