Fail the gate when a test unexpectedly skips #50

Closed
opened 2026-10-24 00:10:41 -05:00 by ethan-hann · 0 comments
Owner

From the test-suite review (WARN-002, .tools/reviews/test-suite/code-review-test-suite.md).

If a skip probe breaks, a group of real tests turns into skips and the gate still reports a pass. That happens when a probe such as TestLinks.TryCreateFolderLink (tests/Savepoint.Core.Tests/Git/Engine/TestLinks.cs) starts returning false. The same holds for HonorsFileModes. For IgnoresCase, only an always-false break hides tests, because FileFolderCollisionTests catches an always-true one on case-sensitive systems.

Write the test report the gate already allows (--report-xunit), compare its skipped test names with a checked-in list for each OS, and fail on any skip the list does not name. Report an expected skip that now runs as a note. Key the list by test name, not by count, so adding a platform-specific test is a one-line edit next to it.

Today's counts: 26 skips on the Windows gate and 75 in Linux CI (CI runs since 2026-10-23, as an ordinary user). The local WSL gate (build/gate-linux.sh) skips more because its 9p mount cannot hold sockets or permissions, so it needs its own list or a switch that says it is not CI. Both build/gate.ps1 and build/gate.sh need the step.

Done when making TryCreateFolderLink return false everywhere fails the gate and names the tests that started skipping.

From the test-suite review (WARN-002, `.tools/reviews/test-suite/code-review-test-suite.md`). If a skip probe breaks, a group of real tests turns into skips and the gate still reports a pass. That happens when a probe such as `TestLinks.TryCreateFolderLink` (`tests/Savepoint.Core.Tests/Git/Engine/TestLinks.cs`) starts returning false. The same holds for `HonorsFileModes`. For `IgnoresCase`, only an always-false break hides tests, because `FileFolderCollisionTests` catches an always-true one on case-sensitive systems. Write the test report the gate already allows (`--report-xunit`), compare its skipped test names with a checked-in list for each OS, and fail on any skip the list does not name. Report an expected skip that now runs as a note. Key the list by test name, not by count, so adding a platform-specific test is a one-line edit next to it. Today's counts: 26 skips on the Windows gate and 75 in Linux CI (CI runs since 2026-10-23, as an ordinary user). The local WSL gate (`build/gate-linux.sh`) skips more because its 9p mount cannot hold sockets or permissions, so it needs its own list or a switch that says it is not CI. Both `build/gate.ps1` and `build/gate.sh` need the step. Done when making `TryCreateFolderLink` return false everywhere fails the gate and names the tests that started skipping.
ethan-hann added this to the 1.0 milestone 2026-10-24 00:10:41 -05:00
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
ethan-hann/SavePoint#50
No description provided.