Repository navigation
[AMD] Add ROCm 10 (gfx942 / gfx950) release images - #36434
Conversation
Build gfx942 and gfx950 images from AMD's prerelease wheel stack and wire targeted MI300/MI35x performance validation through the scratch registry.
|
Validation scope has been expanded to the complete existing AMD PR and nightly suites in run 32932367013. After both gfx942 and gfx950 images are published, the release workflow now invokes:
The PR, extra, nightly, normal-container, and disaggregated-container paths all route Experiment recordBoth baseline images using AITER
The same workflow has now started both complete validation graphs with
|
ROCm 10 RC4 + AITER
|
bingxche
changed the title
ROCm 10.0.0 GA + AITER
|
ROCm 10.0.0 GA + AITER
|
The upstream GA switch (#36434) reserves gfx1250 in ROCM_DEVICE_ARCH_LIST but stops short of the CI-side detection, which still matched only gfx942/gfx950. A gfx1250-rocm1000 image therefore fell through to the inference branch, where its PIP_CONSTRAINT stamp made it look like a rocm724 gfx942 image and skewed both the AITER pin lookup and MORI_GPU_ARCHS.
…42-950 # Conflicts: # .github/workflows/nightly-test-amd-rocm720.yml
bingxche
changed the title
Four shared build steps on this branch differ from the ROCm 10 release images, so a gfx942 or gfx950 build here does not produce what #36434 validated. None of them exist for the CDNA archs' sake, so put those archs back on the released path and leave gfx1250 on exactly what it was brought up against. The srt_hip_torch2_11 extras were renamed from srt_hip_rocm724 when the ROCm 7.14 flavor arrived, so that one extra could name a torch generation rather than a ROCm flavor and serve 7.2.4, 7.14 and 10.0.0 alike. With 7.14 gone the consumers are 7.2.4 and 10.0.0 again, which is what the upstream name already covers. The lists are identical line for line, so this is a rename back. It also repairs the AMD installer, which asks for dev_hip_rocm724. The MORI RDMA/MPI/gRPC packages and the wider libabsl removal both arrived with the gfx1250 bring-up. gfx1250 keeps them; the CDNA images go back to the package set and the -dev-only removal they were released with. Dropping libabsl20220623 takes out the runtime library that components built earlier in the file link against, which is what the comment directly above it warns not to do. The rocm720 torch METADATA patch was generalized to also serve 7.14. That other consumer is gone, and the rewrite turned a missing triton pin from a hard failure into a silent skip, so restore the released version. Left alone deliberately: PYTORCH_ROCM_ARCH and the pip constraints file. The first is load-bearing for gfx1250 (tilelang reads the global value and has no per-build override), and the second is what keeps pip from moving torch out from under the gfx1250 image during the extras install.
yctseng0211
left a comment
yctseng0211
left a comment
There was a problem hiding this comment.
build validation : https://github.com/sgl-project/sglang/actions/runs/33163766664
The wheel-SDK base ships the devel tree's runtime libs as byte-identical copies of the core/libraries ones (ROCm/TheRock#7818), and devel/lib puts unversioned *.so aliases on LD_LIBRARY_PATH, so an unversioned dlopen (torch: libamd_smi.so; triton/proton: libamdhip64.so, libhsa-runtime64.so, libhipblaslt.so) pulls a second copy of an already-loaded runtime into the process. Two comgr copies abort at startup with LLVM's "Option registered more than once"; the base image already collapses the libamd_smi pair, where the second copy silently breaks device enumeration (sgl-project/sglang#36434). Collapse every byte-identical top-level devel duplicate back onto its original.
Summary
Add ROCm 10 release images for gfx942 (MI300/MI325) and gfx950 (MI350/MI355X) using AMD's stable wheel-based ROCm SDK.
Pinned stack:
3.8.0+git4cff872cc16d44b93a528b2a4bfd6d8d3409116d465872a9Changes
gfx942-rocm1000andgfx950-rocm1000Docker targets. Each image installs only its target architecture's ROCm SDK and PyTorch device wheels.rocm10-mi30xandrocm10-mi35xtags torocm/sgl-dev, then mirrors the same tags tolmsysorg/sglang-rocm.rocm10support to the AMD PR, extra, and nightly workflows. The scheduledallmatrix remains unchanged and continues to cover ROCm 7.2.4 and 7.2.0.rocm1000flavor, including its torch, MORI, and AITER compatibility handling.Temporary branch triggers, scratch-registry publishing, runtime probes, and chained validation jobs have been removed. Validation results remain recorded in the PR comments.
Validation
a6d2b564fd671724a3720b8edf70e8d674e4d694: run 33055565205.References
CI States
Latest PR Test (Base): ✅ Run #33161266798
Latest PR Test (Extra): ❌ Run #33161266722
Latest PR Test (AMD ROCm 7.2): ❌ Run #33161266823