On Mon, May 26, 2025 at 09:54:40AM -0400, Eli Schwartz wrote: > On 5/26/25 8:44 AM, Patrick Steinhardt wrote: > > On Fri, May 23, 2025 at 12:33:23PM -0700, Junio C Hamano wrote: > >> Ramsay Jones <ramsay@xxxxxxxxxxxxxxxxxxxx> writes: > >> > >>> Question: should meson (or indeed prove) fail the test because of an > >>> unexpected _pass_? > >> > >> Yes, it is a very good question. I do not mind if the answer is "it > >> should, and the make and prove shouldn't let unexpected pass go > >> unnoticed". The difference between the build systems bothers me > > > > Indeed, a good question. The TAP specification [1] has this to say: > > > > Should a todo test point begin succeeding, the harness may report it > > in some way that indicates that whatever was supposed to be done has > > been, and it should be promoted to a normal Test Point. > > > > Harnesses must not treat failing TODO test points as a test failure. > > > > Harneses should report TODO test points found as a list of items > > needing work, if that is appropriate for their use case. > > > > So if my reading of this is correct then Meson isn't wrong in reporting > > this as an error -- "in some way" basically gives it full permission to > > do so. So this is plain old undefined behaviour we rely on :/ > > > I don't see how you can possibly compare this to UB. It is a documented > variant behavior and thus "implementation-defined", not "undefined". Fair enough, "implementation-defined" is the better way to put it. > Also, > > https://www.gnu.org/software/automake/manual/automake.html#Generalities-about-Testing > > """ > It’s not uncommon, especially during early development stages, that some > tests fail for known reasons, and that the developer doesn’t want to > tackle these failures immediately (this is especially true when the > failing tests deal with corner cases). In this situation, the better > policy is to declare that each of those failures is an expected failure > (or xfail). In case a test that is expected to fail ends up passing > instead, many testing environments will flag the result as a special > kind of failure called unexpected pass (or xpass). > """ > > > I don't think it's inherently a bad thing to fail on unexpected passes. > > After all, it shows that our assumption that the test fails is broken, > > and that we should have a look why that is. But I can see arguments both > > ways. > > As Phillip noted, treating them as ordinary passes undermines the reason > for having them. Yup, and I tend to agree. Patrick
