Repository navigation
Winsock Fixes - #3433
Winsock Fixes#3433
Conversation
This results in a partially working winsock
BUG: A connection currently only becomes communicative after a new connection comes in. But Windows TCP is working after that.
| var realHandle: HANDLE? = nil | ||
| let success = DuplicateHandle( | ||
| GetCurrentProcess(), // Source process | ||
| GetCurrentThread(), // Source handle (pseudo-handle) | ||
| GetCurrentProcess(), // Target process | ||
| &realHandle, // Target handle (real handle) | ||
| 0, // Desired access (0 = same as source) | ||
| false, // Inherit handle | ||
| DWORD(DUPLICATE_SAME_ACCESS) // Options | ||
| ) |
There was a problem hiding this comment.
Windows gives pseudo handles by default, so they're always correct for the current thread. However, when spawning work on another thread, this handle results in getting the incorrect ThreadID, causing QueueUserAPC to notify the wrong (or non-existing) thread. This in turn prevents SleepEx from waking up the eventloop
There was a problem hiding this comment.
We don't use QueueUserAPC anymore. However the general statement remains true. Maybe we can rework the comments, to be less about QueueUserAPC. We maybe also want to document somewhere why QueueUserAPC is not an option.
There was a problem hiding this comment.
Resolved β these comments were reworked away from QueueUserAPC; the rationale for why APCs are not usable now lives in the selector wakeup section in SelectorWSAPoll.swift.
|
|
||
| static var currentThread: ThreadOpsSystem.ThreadHandle { | ||
| GetCurrentThread() | ||
| var realHandle: HANDLE? = nil |
| registrationID: SelectorRegistrationID | ||
| ) throws { | ||
| fatalError("TODO: Unimplemented") | ||
| if let index = self.pollFDs.firstIndex(where: { $0.fd == UInt64(fileDescriptor) }) { |
There was a problem hiding this comment.
Deregistering is just removing the FD. However, we can't just remove it here as we're iterating over the same pollFDs at the same time. deregister0 is called down the stack of try body((SelectorEvent(io: selectorEvent, registration: registration))). So the available indices of pollFDs changes causing a crash
There was a problem hiding this comment.
can we add this comment to the local array that we use for this? so that we document what this array is used for?
There was a problem hiding this comment.
Resolved β pollFDs is now documented (together with its pollFDIndexes and deregisteredFDs companions) in SelectorGeneric.swift.
| @@ -131,6 +133,19 @@ extension Selector: _SelectorBackendProtocol { | |||
|
|
|||
| try body((SelectorEvent(io: selectorEvent, registration: registration))) | |||
There was a problem hiding this comment.
This line often calls deregister0 indirectly, so we effectively can't mutate pollFDs in deregister0
| // now clean up any deregistered fds | ||
| // In reverse order so we don't have to copy elements out of the array | ||
| // If we do in in normal order, we'll have to shift all elements after the removed one | ||
| for i in self.deregisteredFDs.indices.reversed() { | ||
| if self.deregisteredFDs[i] { | ||
| // remove this one | ||
| let fd = self.pollFDs[i].fd | ||
| self.pollFDs.remove(at: i) | ||
| self.deregisteredFDs.remove(at: i) | ||
| self.registrations.removeValue(forKey: Int(fd)) | ||
| } | ||
| } |
There was a problem hiding this comment.
Do the deregister0 work of cleaning up after the polling is done
| func initialiseState0() throws { | ||
| self.pollFDs.reserveCapacity(16) | ||
| self.deregisteredFDs.reserveCapacity(16) | ||
| self.lifecycleState = .open |
There was a problem hiding this comment.
Lifecycle never became open yet
| #if os(Windows) | ||
| case .winsock(WSAEWOULDBLOCK): | ||
| return false | ||
| #endif |
There was a problem hiding this comment.
accept returns WSAEWOULDBLOCK
There was a problem hiding this comment.
Why doesn't the current logic for syscall(blocking: true) cover this?
|
CC @fabianfett and @zamderax |
This way new TCP connections will be functional
| #if os(Windows) | ||
| if | ||
| let err = err as? IOError, | ||
| case .winsock(WSAEWOULDBLOCK) = err.error |
There was a problem hiding this comment.
What I've found is that when WSAPoll gives the server socket a .read event when a client is inbound:
- Server accepts the client through
readable()->readable0()->ServerSocketChannel.readFromSocket() - The
socket.accept(..)finds a socket SelectableEventLooprepeats this again again to get another read (see maxMessagesPerRead)- This time,
socket.accept(..)runs intoWinSDK.INVALID_SOCKET - Gets the error using
WSAGetLastError() - The error ends up reaching
.winsock(WSAEWOULDBLOCK)
In hindsight, I just noticed that accept() returns an optional so I'll leverage that instead.
| #if os(Windows) | ||
| case .winsock(WSAEWOULDBLOCK): | ||
| return false | ||
| #endif |
There was a problem hiding this comment.
Why doesn't the current logic for syscall(blocking: true) cover this?
It seems there's no socketpair API on Windows
It's used pretty heavily, and writev is implemented on Windows.
|
Hey guys, is it something that can be accepted? What is the status of this PR? Can someone please share plans for the Windows support roadmap? |
|
@3a4oT this needs a proper review first. And there's one gaping hole in the implementation's polling still that needs fixing. Hopefully I'll figure that out soon. |
| private static func createWakeupSocketPair() throws -> (NIOBSDSocket.Handle, NIOBSDSocket.Handle) { | ||
| // Create a UDP socket to receive wakeup signals | ||
| let readSocket = try NIOBSDSocket.socket(domain: .inet, type: .datagram, protocolSubtype: .default) | ||
|
|
||
| do { | ||
| // Bind to loopback on an ephemeral port | ||
| var addr = sockaddr_in() | ||
| addr.sin_family = ADDRESS_FAMILY(AF_INET) | ||
| addr.sin_addr.S_un.S_addr = WinSDK.htonl(UInt32(bitPattern: INADDR_LOOPBACK)) | ||
| addr.sin_port = 0 // Let the system assign a port | ||
|
|
||
| try withUnsafeBytes(of: &addr) { addrPtr in | ||
| let socketPointer = addrPtr.baseAddress!.assumingMemoryBound(to: sockaddr.self) | ||
| try NIOBSDSocket.bind(socket: readSocket, address: socketPointer, address_len: socklen_t(MemoryLayout<sockaddr_in>.size)) | ||
| } | ||
|
|
||
| // Get the assigned port | ||
| var boundAddr = sockaddr() | ||
| var size = Int32(MemoryLayout<sockaddr>.size) | ||
| try NIOBSDSocket.getsockname(socket: readSocket, address: &boundAddr, address_len: &size) | ||
|
|
||
| // Create the write socket | ||
| let writeSocket = try NIOBSDSocket.socket(domain: .inet, type: .datagram, protocolSubtype: .default) | ||
|
|
||
| do { | ||
| // Connect the write socket to the read socket's address | ||
| guard try NIOBSDSocket.connect(socket: writeSocket, address: &boundAddr, address_len: size) else { | ||
| throw IOError(winsock: WSAGetLastError(), reason: "connect") | ||
| } | ||
| return (readSocket, writeSocket) | ||
| } catch { | ||
| _ = try? NIOBSDSocket.close(socket: writeSocket) | ||
| throw error | ||
| } | ||
| } catch { | ||
| _ = try? NIOBSDSocket.close(socket: readSocket) | ||
| throw error | ||
| } | ||
| } |
There was a problem hiding this comment.
This is the code that needs to be fixed.
|
|
||
| import CNIOWindows | ||
|
|
||
| let IPPROTO_UDP = Int32(17) |
There was a problem hiding this comment.
IPPROTO_UDP is of IPPROTO type on Windows. Windows has some weird mappings of POSIX constants
| static var UIO_MAXIOV: Int { | ||
| fatalError("unsupported OS") | ||
| // TODO: This is a placeholder value. Find the correct one for Windows. | ||
| return 64 |
There was a problem hiding this comment.
This variable doesn't formally exist on Windows, but vector writing does work on Windows. We should really make it unavailable for Windows, and update all dependencies that indirectly use this constant. I had a few places that were affected
| @usableFromInline | ||
| var deregisteredFDs = [Bool]() |
There was a problem hiding this comment.
Would it make sense to just store indexes here? This would prevent us from linear searching the pollFDs. Please also add a comment what we use this array for.
There was a problem hiding this comment.
Resolved β there is now a pollFDIndexes map (fd β index) so reregister0/deregister0 are O(1) instead of scanning pollFDs, and it is documented.
|
|
||
| try withUnsafeBytes(of: &addr) { addrPtr in | ||
| let socketPointer = addrPtr.baseAddress!.assumingMemoryBound(to: sockaddr.self) | ||
| try NIOBSDSocket.bind(socket: readSocket, address: socketPointer, address_len: socklen_t(MemoryLayout<sockaddr_in>.size)) |
There was a problem hiding this comment.
Using sockaddr_in is weird, as we have previously cast the pointer to sockaddr.
There was a problem hiding this comment.
Resolved by the later redesign β the wakeup pair is AF_UNIX/sockaddr_un throughout now, so the odd sockaddr_in cast is gone.
8c67972 to
9332b56
Compare
|
we will also need to make the CI green |
|
CC @fabianfett can I get a review? CI is failing on benchmark changes, which seem in favour of this PR actually π But the alloc count changed - I don't modify that though. I think it's better for you to make sure it's correct. |
@Joannis I've asked @glbrntt to look into this soon. Fabian is away at the moment. |
| private static func generateUniqueSocketPath() throws -> String { | ||
| // Get the temp directory path | ||
| var tempPathBuffer = [CChar](repeating: 0, count: Int(MAX_PATH) + 1) | ||
| let tempPathLen = GetTempPathA(DWORD(tempPathBuffer.count), &tempPathBuffer) |
There was a problem hiding this comment.
This should use GetTempPath2W -- the 'A' functions are not Unicode safe and there are numerous reasons this could be a path with Unicode characters in it, or a path longer than the 260-chars soft limit.
Take a look at https://github.com/swiftlang/swift-subprocess/blob/049ddf99c37c50411ce2997dc92a36eb74460b98/Sources/Subprocess/Platforms/Subprocess%2BWindows.swift#L1696, which helps fill an appropriately-sized buffer for you, up to the actual maximum supported by Windows (Int16.max). (please just copypaste this implementation, we'll want it in a bunch more places too, I'm sure)
Example usage here: https://github.com/swiftlang/swift-subprocess/blob/049ddf99c37c50411ce2997dc92a36eb74460b98/Sources/Subprocess/Platforms/Subprocess%2BWindows.swift#L775
| let listenerSocket = try NIOBSDSocket.socket(domain: .unix, type: .stream, protocolSubtype: .default) | ||
| defer { | ||
| _ = try? NIOBSDSocket.close(socket: listenerSocket) | ||
| _ = socketPath.withCString { DeleteFileA($0) } |
There was a problem hiding this comment.
Should use DeleteFileW; see other comment for why you can't guarantee this is ASCII, or less than 260 chars.
| #if os(Windows) | ||
| import WinSDK | ||
| #endif |
| } | ||
|
|
||
| private let error: Error | ||
| package let error: Error |
There was a problem hiding this comment.
I see you've opened this up to use in SocketChannel.swift and avoid the fatalError path in errnoCode. We'll need to think about how to handle that API a bit better in the future. For now could you document that calling errnoCode will fatalError on Windows?
|
|
||
| static func joinThread(_ thread: ThreadOpsSystem.ThreadHandle) { | ||
| let dwResult: DWORD = WaitForSingleObject(thread, INFINITE) | ||
| let dwResult: DWORD = WaitForSingleObject(thread.handle, INFINITE) |
There was a problem hiding this comment.
I believe you need to call CloseHandle here as well.
See also:
|
|
||
| static var currentThread: ThreadOpsSystem.ThreadHandle { | ||
| GetCurrentThread() | ||
| var realHandle: HANDLE? = nil |
There was a problem hiding this comment.
Isn't this leaking handles? IIUC each "real" handle from DuplicateHandle is its own entity which needs a call to CloseHandle?
There was a problem hiding this comment.
I think it also means Thread.currentThreadID won't return a stable value on Windows since each handle will have a different value despite pointing to the same underlying kernel resource. A quick search found this which seems more appropriate: https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-getcurrentthreadid
| } | ||
|
|
||
| func close0() throws { | ||
| // Close the wakeup sockets |
There was a problem hiding this comment.
Should this func be acquiring the externalSelectorFDLock? Both the epoll and kqueue selectors do.
|
Hey @odrobnik - I did reply on your messages actually! I was at an event in Munich which took a week out of my schedule (no Windows laptop with me). |
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015AbqR9AdcXCNDegGY3fxpc
Address review feedback (jakepetroules): the ANSI GetTempPathA cannot represent temp directories outside the active code page and is capped at the 260-char MAX_PATH soft limit. Switch to the wide GetTempPath2W via a buffer-filling helper (modeled on swift-subprocess) that grows up to Int16.max, decoding the result as UTF-16. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015AbqR9AdcXCNDegGY3fxpc
Follow-up to the GetTempPath2W change, covering the rest of the outstanding review comments on the Winsock Fixes PR (apple#3433): - SelectorWSAPoll: delete the on-disk wakeup socket path with DeleteFileW instead of DeleteFileA, matching the Unicode/long-path safety of the GetTempPath2W lookup (jakepetroules). - SelectorWSAPoll: take externalSelectorFDLock in close0, matching the epoll and kqueue selectors, so wakeup-socket teardown is serialized against wakeup0 (glbrntt). - ThreadWindows.joinThread: CloseHandle the duplicated, owning thread handle after the join to stop leaking kernel handles (glbrntt). - Thread.currentThreadID: on Windows use GetCurrentThreadId(), which is stable and allocation-free, instead of the value of a freshly DuplicateHandle'd HANDLE (unstable and leaked) (glbrntt). - Thread.withCurrentThread: CloseHandle the current-thread handle that currentThread duplicated, on Windows (glbrntt). - IOError.errnoCode: document that it traps on Windows when the error is in the windows/winsock domain rather than errno (glbrntt). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015AbqR9AdcXCNDegGY3fxpc
|
I also got a new Windows CI job spun up in #3644, and @glbrntt has marked that as required so we can guarantee no portion of the in-progress Windows build regresses as we continue to get more targets building successfully. @Joannis since this PR gets NIOPosix and 16 other downstream targets successfully building, can you please add those to MATRIX_WINDOWS_COMMAND in the GitHub Actions as part of this PR so we can "lock them in"? Here's a diff which does that, feel free to pull it into your branch: ea9314b |
β¦hreadex handle leak) Two fixes not covered by the direct review-feedback commits already on this branch: - ThreadWindows.run: close the owning handle returned by `_beginthreadex`. The handle used to `join` the spawned thread is the one the thread duplicates for itself in `routine`; the `_beginthreadex` handle is never used, so it leaked one handle per spawned thread (and left an unused variable). Closing it does not affect the still-running thread. - BaseSocketChannel: remove the unused `import WinSDK` (glbrntt review comment). Also corrects a stale doc comment on `pollFDIndexes` to match the in-place compaction that `whenReady0` performs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With this branch's Winsock fixes, NIOPosix and everything that just
transitively depended on it now compiles cleanly on Windows. Extend
the Windows CI target list from 11 to 28: add NIO, NIOPosix,
NIOTestUtils, and all 14 example executables (NIOTCPEcho{Server,
Client}, NIOEcho{Server,Client}, NIOHTTP1{Server,Client},
NIOChat{Server,Client}, NIOWebSocket{Server,Client}, NIOMulticastChat,
NIOUDPEcho{Server,Client}, NIOAsyncAwaitDemo).
Still failing and therefore still excluded:
- NIOPerformanceTester (missing stdout in scope)
- NIOCrashTester (missing setenv in scope)
- NIOFS / NIOFileSystem family (POSIX file primitives via swift-system)
|
@jakepetroules, @odrobnik, I just pulled in your changes. There are no more open points, correct? Can someone trigger the CI? Thanks |
glbrntt
left a comment
There was a problem hiding this comment.
Looks like the format check has failed. Could you run the formatter?
glbrntt
left a comment
There was a problem hiding this comment.
Awesome, thanks to all involved in getting this over the line!
Thanks to Apple too for allowing it! It's on main now. Now we are waiting for it to become a release right? |
β¦no longer traps Two winsock gaps that apple#3433 left: 1. A socket's pending error (`SO_ERROR`) is a winsock code on Windows, but the three places that read it (`BaseSocketChannel.reset()`, `Socket.finishConnect()`, `SocketChannel.error()`) filed it under `errno`. A refused connect therefore read "connection reset (error set): Unknown error) (errno: 10061)", and `shouldCloseOnErrnoCode` compared a winsock code with POSIX constants. They now go through `IOError(socketErrorCode:reason:)`, which files the code under its POSIX errno where one exists (ECONNREFUSED, "connection refused") and under `winsock` otherwise. 2. `IOError.errnoCode` called `fatalError` for a winsock or Windows code. Callers cannot know to check the domain first (NIO's own `BaseSocket` reads it after a failed `IPV6_V6ONLY`), and an app that shows a failed bind's `strerror(errnoCode)` died instead of reporting a taken port. It now returns the POSIX errno of the same meaning for a winsock code where there is one, else the code itself, which matches no POSIX constant. The winsock-to-POSIX table spells the winsock codes as numbers so NIOCore keeps its selective WinSDK imports. Everything is inside `#if os(Windows)`. `WindowsPortTests/WinsockErrorTests` covers a refused connect, a bind on a taken port and `errnoCode` itself; the refused-connect test fails on the previous commit (10061 is not 107, "Unknown error"). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBLwkmBCzQTzq1itHKFmtH
This PR gets TCP (Servers) on Windows mostly working. I'll annotate my PR for clarity.
The one bug:
If you open a TCP client connection to a Windows TCP server, it doesn't read/write yet until a second connection comes in. This happens because
reregister0is called afterWSAPollis called - but WSAPoll isn't woken up to know this so it doesn't know the new socket requests read/writes/...Also the EventLoop cannot be woken up during
el.execute { .. }becauseSleepExisn't running. WSAPoll doesn't respond to this signal it seems.This makes the TCP server functionality not yet usable, but I thought with this draft we could figure out what I'm missing.
Alternative considered: We can make
WSAPolllimited to wake up every 1 ms for example, that way we can still get to these events at some point.