Repository navigation
Fix DockerExecutor connection reset error with server readiness check - #1684
Conversation
There was a problem hiding this comment.
Thanks a lot for the fix: this was really necessary!!
Note that the root cause was that while the Docker container reported as "running", the Jupyter Kernel Gateway service inside the container needed additional time to fully initialize and be ready to accept connections. This PR adds proper server readiness checks and retry logic to ensure the Jupyter Kernel Gateway is fully operational before attempting to create kernels and websocket connections.
I just fixed code quality, changed the check endpoint from /api to /api/kernelspecs (to be completely sure kernel-management is operational) and extracted the functionality as a method, so it is aligned with ModalExecutor, and linked the corresponding GH issue.
…huggingface#1684) Co-authored-by: Ubuntu <ubuntu@ip-172-31-4-23.us-west-2.compute.internal> Co-authored-by: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com>
Issue: Trying to send request to jupyter API before the process is ready - gives error occasionally.
Solution: Wait until the jupyter API shows ok health status before continuing.