Skip to content

[Windows] Skip unsupported/flaky tests and run the Windows unit tests in CI - #3680

Merged
glbrntt merged 9 commits into
apple:mainfrom
jakepetroules:windows-tests-pass-with-skip
Jul 29, 2026
Merged

glbrntt merged 9 commits into
apple:mainfrom
jakepetroules:windows-tests-pass-with-skip

Conversation

@jakepetroules

Copy link
Copy Markdown
Member

Motivation:

All of the test targets now compile on Windows (#3663, #3665), but swift test doesn't yet pass: a number of tests exercise functionality that isn't supported or isn't yet functional on Windows, some crash or hang, and a few are flaky on the WSAPoll-based selector. Before the Windows unit tests can run in CI, those tests need to be skipped (with reasons) so a full swift test exits cleanly.

Modifications:

Skip the tests that don't pass on Windows, using XCTSkip / Swift Testing's .disabled / a gated setUp where a fixture itself traps, each with a reason. Broadly:

  • Unsupported transports/APIs: Unix domain sockets, datagram/UDP channels, pipe channels, raw sockets, vsock, pre-connected sockets, and FileRegion sends.
  • Whole suites for subsystems that aren't functional on Windows yet, including the NIOFileSystem family (NIOFSTests, NIOFSIntegrationTests, NIOFSFoundationCompatTests) whose Windows syscall shims are still stubbed, plus EchoServerClientTest, ChannelTests, DatagramChannelTests, StreamChannelTest, FileRegionTest, PipeChannelTest, RawSocketBootstrapTests, ThreadTest, and SocketOptionProviderTest.
  • Real crashes surfaced by a full run: System.enumerateDevices() (mis-sized GetAdaptersAddresses buffer), the WSAPoll selector trapping on an invalid fd and on schedule-after-shutdown, and a connection-reset promise leak.
  • Flaky, timing-sensitive tests: the event-loop tick-count metrics tests (the WSAPoll selector wakes far more often than epoll/kqueue).

Then enable the Windows 6.3 + nightly jobs in the reusable unit-tests workflow (which runs swift test) in both the PR and push workflows, and drop the separate build-only Windows job (which only ran swift build --build-tests).

Windows uses the default swift test arguments rather than the Linux -Xswiftc -warnings-as-errors override, because a #warning in the (skipped) stream-channel tests would otherwise fail the build.

All skips are Windows-only; there is no behaviour change on other platforms.

Result:

swift test passes on Windows, and the Windows unit tests run in CI.

Disable the tests that crash, hang, or fail on Windows so `swift test`
runs green for the L0-L3 targets, using `XCTSkip` (and gated `setUp`
where a fixture itself traps) rather than compiling the tests out.

- NIOCoreTests: skip `testEnumeratingDevices` (`System.enumerateDevices()`
  currently traps on Windows due to a mis-sized `GetAdaptersAddresses`
  buffer).
- NIOHTTP1Tests: skip the four `HTTPServerClientTest` FileRegion variants
  (sending a `FileRegion` over a socket is not yet supported).
- NIOPosixTests: skip the transports that are not yet functional on
  Winsock -- Unix domain sockets, datagram/UDP channels, pipe channels,
  raw sockets, vsock, pre-connected sockets, dual-stack IPv4/IPv6 binds,
  and FileRegion sends -- either per-method or per-class. `setUp`/
  `tearDown` bodies that would trap before the skip takes effect
  (PipeChannelTest, SocketOptionProviderTest) are guarded too.

All changes are Windows-only; no behaviour changes on other platforms.
The NIOFileSystem family (`NIOFS`, `_NIOFileSystem`, and the FoundationCompat
layers) is not yet ported to Windows: every syscall in its Windows shim is a
`fatalError` stub pending follow-up PRs. Its test targets therefore trap at
runtime on essentially every test (real syscalls, and the mocking TLS shim).

Skip all XCTest classes in NIOFSTests, NIOFSFoundationCompatTests, and
NIOFSIntegrationTests on Windows via a `setUpWithError` that throws `XCTSkip`,
so `swift test` runs green until the shims are implemented. Windows-only; no
behaviour change on other platforms.
A full, unfiltered `swift test` also runs the Swift Testing suites (which a
per-target XCTest `--filter` run skips), surfacing a few more Windows gaps:

- NIOTransportAccessibleChannelCoreTests.testUnderlyingTransportConformanceForExpectedChannels:
  creates a PipeChannel, which is unsupported on Windows.
- EventLoopTest.testStructuredConcurrencyMTELGStartStop: asserts the selector's
  debugDescription, but the WSAPoll selector has no epoll-style descriptor.
- StrictCrashTests.eventLoopScheduleAfterShutdown: an exit test expecting a trap
  that does not fire on Windows.
- SocketChannelTest.testServerClosesTheConnectionImmediately (XCTest): passes in
  isolation but leaks a promise under full-suite load due to unreliable
  connection-reset handling on Windows.

Disabled via `.disabled`/`XCTSkip` on Windows only. `swift test` now exits 0.
EventLoopMetricsDelegateTests asserts on how many times the event loop ticks,
but the WSAPoll-based selector wakes far more often than epoll/kqueue (observed
~192 ticks where 2...5 are expected) and can tick spuriously even with no work.
That makes the tick-count assertions unreliable and flaky on Windows, so the
whole suite is skipped there via a `setUpWithError` that throws `XCTSkip`.
Now that `swift test` passes on Windows, enable the Windows 6.3 and nightly
jobs in the reusable unit-tests workflow (which runs `swift test`) in both the
PR and push workflows, and drop the separate build-only Windows job, which only
ran `swift build --build-tests` and is now subsumed by the unit-tests run.

Windows uses the default `swift test` arguments rather than the Linux
`-Xswiftc -warnings-as-errors` override: a `#warning` in the (skipped) stream
channel tests would otherwise fail the build.

@glbrntt glbrntt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small thing to change but LGTM otherwise

Comment thread Tests/NIOPosixTests/EventLoopTest.swift Resolved
jakepetroules and others added 2 commits July 28, 2026 00:42
Address review feedback: replace the repeated
`.disabled("...") { #if os(Windows) return true #else return false #endif }`
closures with a single `System.isWindows` test-module helper and the
`.disabled(if:_:)` trait, i.e. `.disabled(if: System.isWindows, "...")`.
Comment on lines +25 to +27
// Several of these socket echo integration tests hang or crash on
// Windows (Unix domain sockets, dual-stack IPv4/IPv6 binds, and others),
// so the whole suite is skipped there pending functional networking.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unacceptable language check isn't happy with this:

Tests/NIOPosixTests/EchoServerClientTest.swift:25:59:        // Several of these socket echo integration tests hang or crash on

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops, fixed!

@glbrntt glbrntt added the πŸ”¨ semver/patch No public API change. label Jul 28, 2026
The soundness check flags "hang"; reword the EchoServerClientTest and ThreadTest
skip reasons to use "stall" instead.
TypedHTTPServerUpgradeTestCase.testSendRequestCloseImmediately half-closes the
connection right after writing the upgrade request; the half-close timing races
the upgrade on Windows under load, so it fails intermittently ("I/O on closed
channel"). It passes reliably in isolation. Skip it on Windows.
@glbrntt
glbrntt merged commit b07752c into apple:main Jul 29, 2026
52 of 58 checks passed
@jakepetroules
jakepetroules deleted the windows-tests-pass-with-skip branch July 29, 2026 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

πŸ”¨ semver/patch No public API change.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants