Repository navigation
Description
Built from the fork cheemu21/kid-with-sword-remake (its V6 state), not from main; the fork has issues turned off, and most of its Ocarina of Time code comes from here. Possibly related to #1326 (a GPU hang about 10 minutes in), but here it comes within seconds to minutes, on an M1, together with memory pressure.
Setup
- MacBook Air M1 (8-core GPU,
applegpu_g13g), 16 GB, fanless - macOS 27.0.1, Xcode 27.2, Python 3.15.2 (Homebrew, arm64)
- The fork at
d1d672ac(branchkid-with-sword-remake) - Game: a
.nupwith update v131072 and no base game (the script took the update as the whole game). One RomFS file is reported damaged and replaced at every start:Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs("not a zstd stream, begins 00 00 00 00") - Built with
--platform mac-app --env-file mac_env.txt(DM_UPSCALER=metalfx,DM_FPS_HOLD=off,DM_ALIAS_SIZE=0x200000000) and the resolution overlays from the game note (handheld 0.666667, docked 0.9375,flat). Build: 42,853 of 42,854 shaders, pipeline archive forapplegpu_g13gmade (30,515 render + 4,125 compute bases) - Launcher: Handheld, 30 fps, Upscaling off, full screen
What happens
The picture freezes for good while the game goes on (sound and input still work). The log shows a segment command buffer failing with Caused GPU Hang Error (00000003:kIOGPUCommandBufferCallbackErrorHang), then kIOGPUCommandBufferCallbackErrorSubmissionsIgnored on every later submission.
It happened in every one of 7 runs, at the start of the game, between a few seconds after loading the save and about 3 min of gameplay (frames 565 to 3836). In the last run I stood still for 1.5 minutes without a hang; it froze a few seconds after I started walking (frame 3512). An earlier guess (picking up a blue rupee) did not hold: it also froze without one.
No more memory pools than the 4095 entries line (the known Ocarina of Time hang in docs/reference/shaders.md), no new pipelines or shader libraries in the frames before the hang.
What I tried (no change to the hang)
| Change | Result |
|---|---|
| 720p overlay instead of the game's dynamic resolution | GPU time about the same (about 50 ms โ 53 ms per frame in gameplay), still hangs |
DM_GUEST_FILE=1 |
footprint about 11 GB โ 8.4 GB, frame p50 about 45-53 ms โ 41 ms, GPU allocated unchanged (about 9 GB), still hangs |
DM_NVN_DISPATCH_OVERLAP=0 |
still hangs |
Performance before the hang
- Frame time p50 about 42 ms (about 24 fps) even when standing still; the host steps the present interval between 2 and 3 all the time
DM_NVN_GPUPROF=every:300: GPU busy 17-22 ms a frame (render 12-19 ms, compute / copy 4-6 ms), but first start to last end 27-44 ms, so the GPU is idle for about 40% of the frame span. Largest passes: the first compute / copy encoder (about 5 ms, about 100 dispatches) and the 1280x720 main passes (2-4 ms)- GPU allocated 8.4-9.9 GB, resident 7.5-8.4 GB (with guest files)
The hang comes with memory pressure
In a 7th run I sampled kern.memorystatus_vm_pressure_level and vm.swapusage every 2 s next to the game. Loading the save took free memory from 74% to about 46%; then macOS reported pressure level 2 (warn) for 11 s, swap grew from 1.5 GB to 3.0 GB, free memory fell to 21%, and the GPU hang (frame 565) fell inside that window, a few seconds after the save had loaded. Pressure went back to normal right after the hang. Activity Monitor showed the game at about 13 GB at that moment. This also fits the other runs: no hang standing still, a hang soon after loading or walking into new parts of the area.
For comparison, the fork's game note (docs/games/kid-with-sword-remake.md) measured a peak footprint of 5.2-5.5 GB on a 16 GB MacBook Air M5 with tex_release 2; here the footprint is 8-11 GB (8.4 GB with guest files, whose pages still take RAM) with about 9 GB GPU allocated. I do not know why the M1 needs so much more, or whether the scenes compare.
DM_NVN_HANGLOG, mentioned in docs/reference/shaders.md, does not seem to exist in the fork's tree, so I could not name the hung shaders. I can provide full logs if useful.
Activity
lunks commentedon Nov 7, 2026
Thanks for the careful report and the memory sampling. Three things would help tell the causes apart:
pools without a buffer address left out(they give a count and megabytes).main(or a fresh clone of this repository) instead of the fork, with the same save, so we know whether the fork's code matters.If the last pipelines are task/mesh ones, #1585 is a candidate: it fixes a hang where a culling task shader reads a pointer into a memory pool missing from the raw pool table. That is unproven for your case. The memory-pressure window you measured also fits the system-memory stop in #1589, which is open.
LeonardoBai12 commentedon Nov 7, 2026
Three GPU hangs from another M1 with 16 GB, two of them on a build from
main, with the shader ranking the host now logs. This answers your three questions, @lunks, as far as these logs can.Setup
DM_UPSCALER=game.The three hangs
ErrorHang, then 28 รSubmissionsIgnoredErrorHang, 1 failed command bufferErrorHang, 1 failed command bufferThe memory columns come from the last
memory up to frameandon-screenlines before each hang.1.
pools without a buffer address left out: none in any of the three logs.2. The last pipelines before the hang
a5288d0a6c4971fd_b, mesh0a44d06526b90d19_b, fragmentb5e76fbd0fdfe143_ba5288d0a6c4971fd_b, meshcdccb0b8e37b83b1_b, fragment5978c86a51251517_ba5288d0a6c4971fd_b, mesh9b11e51ca22b3608_b, fragmentaa3f1dca9e8828f0_b). No new pipelines or shader libraries in the frames just before.a5288d0a6c4971fd_bis in the failed command buffer's shader list in all three runs. It is probably drawn in most frames, so this may mean nothing.3. On main: B and C are on main 24fe1e5 plus the PRs above, not on a fork. None of those PRs is #1585.
The host's ranking (
game shaders by failed/ran), first entries. Each log has one failure, so the ranking is thin.0a44d06526b90d19_b1/104,b5e76fbd0fdfe143_b1/104,5be2891fce11da4c_b1/134,5fcff4b4a6aa8d71_b1/143,92890e7c46754e66_b1/1451baec668d4dba75c_b,23863906e989c1ca_b,3a4fe787c1901068_b, โฆ), then3b4dc0b6a51cbd4b_b1/100. The A pair is further down at 1/185.3b4dc0b6a51cbd4b_b1/157, thenfa20be590aae1d99_b1/839,6bb6772fb6988a3f_b1/853,fb46ac21db7ab722_b1/8533b4dc0b6a51cbd4b_branks high in both runs on main.Not checked: whether macOS reported memory pressure at those moments (I didn't sample it), and #1585 or #1589 on this Mac. I can post the full logs, or try a build with #1585 if that helps.
LeonardoBai12 commentedon Nov 7, 2026
Three more sessions on the same M1 Pro (16 GB) and build (main 24fe1e5 plus #1543, #1497, #1331, #1453, #1377, #1379). All three ended in
GPU access revokedand exit 70. #1332's alert worked each time.ErrorPageFault), then hangs at 1,333, 1,850, 2,352 and 2,666 sIn D and E the game recovered from the earlier episodes. The last one repeated on every frame until macOS revoked access.
On memory. D alone would point to memory pressure, but E and F hang at the same ~13 GB as the sessions that recovered, with no other apps open. So I don't think memory pressure explains the hangs here. A report on #1326 shows the same hang on an M2 Max with 32 GB.
Shader ranking. I compared the top 40 shaders of the
game shaders by failed/ranline across all six sessions (AโF). No shader appears in three or more of them. The hangs also came in different places: walking, rolling, cutscenes, a shop, water. I can't see a single draw that triggers them.Next: I'm building main bd979ce with #1585 (rebased; chain 9eb5e5a9a0ec here) and will report whether the hangs change on this Mac.
Not checked: memory pressure sampled from outside the game, and
DM_NVN_GPU_WATCHfrom #1550 (not in this build).LeonardoBai12 commentedon Nov 8, 2026
Two more sessions on the same M1 Pro (16 GB, macOS 26.6.1), now on a build from main e940e1f (its own frame step,
fps_target 60) plus #1707, #1497, #1331, #1453, #1377 and #1379. Chain 094f939bfef9, all 42,848 shaders built, render scale 0.7, handheld,DM_UPSCALER=game.Both sessions ended in
GPU access revokedand exit 70, at the same place: leaving Hyrule Castle with Impa, in the cutscene that shows Hyrule Field.ErrorHangat about 430 s (recovered), then fatal at 444 s (frame ~3000)What H shows
47f9b38490dcf9a2and99280b7b8aae78dfare in all three failed command buffers that list shaders.f75708d176bb6591is in none.gpuEvent-dmhost-*.ipsreports from these two sessions are allMMU interrupt(restart_reason 10) withfw_ta_statebusy andfw_cl_stateidle, as in the earlier ones.MTLCompilerServicereports (EXC_BAD_ACCESS, SIGBUS or SIGSEGV, inlibLLVM.dylibunderAGXCompilePlan::execute), and the host loggedXPC_ERROR_CONNECTION_INTERRUPTEDfor taska7f22b4a795ff4d1_b/ meshac2f3feb11427426_b/ fragmenteddc12907cefc9af_b. I don't know whether this is related to the hang.So on this Mac #1707 didn't stop the hang, and it came sooner than on the 24fe1e5 build (16โ44 min there, about 7 min here). The base moved too, so that is not a clean comparison. This matches @ralph-ferza's result on #1326 with
DM_NVN_BARRIER=full.Next: a run with
DM_NVN_SKIP_MESH=1on the same route, to see whether the hang goes away here as in @reefbluesky's runs.LeonardoBai12 commentedon Nov 8, 2026
DM_NVN_SKIP_MESH=1result on the same M1 Pro and build (main e940e1f plus #1707 and the PRs listed above):ErrorHang,PageFaultorGPU access revoked, and macOS wrote nogpuEventreport.So on this Mac too the hang goes away with the task / mesh draws dropped, as in @reefbluesky's runs on #1326.
LeonardoBai12 commentedon Nov 8, 2026
A long session on main 04419bc, on the same M1 Pro (16 GB, macOS 26.6.1). On top: #1331, #1453, #1377, #1379 and the SDK guard; no #1707, #1497 or #1585. Chain 89f44f9b6bb0, all 42,848 shaders built. Handheld,
DM_UPSCALER=game,DM_FPS_TARGET=60,DM_DYNAMIC_RESOLUTION=0(full resolution),DM_PSO_MTL4=0.Result: 4,402 s of play (Kakariko, the graveyard, the mask quest, Castle Town and back, Hyrule Field, Death Mountain), ended by quitting (exit 0). There was no
GPU access revoked.The hang is still there, but every episode recovered:
ErrorHang, 22 failed submissions, from 948 s to 3,625 s (948, 1122, 1125, 1330, 1339, 1415, 1466, 1775, 1776, 1790, 1814, 1998, 2009โ2013, 2151, 2161, 2164, 2228, 2693, 3625).gpuEvent-dmhost-*.ipsreports, allMMU interruptwithfw_ta_statebusy, as before.47f9b38490dcf9a2is in 10 of the 16 failed buffers that list shaders,99280b7b8aae78dfin 6, andf75708d176bb6591in none.GPU allocated was 12.4โ14.0 GB.
Other lines from this log:
a7f22b4a795ff4d1_b/ meshac2f3feb11427426_b/ fragmenteddc12907cefc9af_bat 843 s (XPC_ERROR_CONNECTION_INTERRUPTED).[RED] texture type mismatch: slot 7 declared as Metal type 3 while texture ID 878 is type 7 (NVN target 2, format 0x3f), for vertex shaders9b310b3349825e9a_b,c0aff0eee66cda0f_band09ab31c69af710c9_b. On Death Mountain a broken-looking piece of geometry was floating in the air. I can't tell whether it is related.I can't say which of 04419bc's changes made the episodes recoverable here. The faster 128-bit loads in the mesh shaders are one guess, not tested.
LeonardoBai12 commentedon Nov 8, 2026
A second session on the same build (main 04419bc and the same settings as above): 2,261 s on Death Mountain, in Goron City and the Lost Woods, ended by quitting (exit 0).
ErrorHang, 1 failed submission), which recovered. Culling task47f9b38490dcf9a2was in its failed buffer. Nothing else for the rest of the session.a7f22b4a795ff4d1_bonce more, at 75 s.DM_PSO_MTL4=0(146 s on the first start after the chain change).To me this session played almost perfectly.
lunks commentedon Nov 8, 2026
Thanks, this is useful. We had a lead from one of our own hangs (M5 Pro, a build with
04419bc2and withoutea0c98df): the failed pass held 13 shaders that had run in no earlier command buffer, and two task/mesh pipelines were built at first use at frames 4264 and 4265 (comment on #1326). Your three runs don't fit it: the last task/mesh pipeline was built about 200 frames before the hang in A and B, and about 7,500 frames before it in C. So we're dropping "a freshly built pipeline" as the cause.What your runs and ours share: task
a5288d0a6c4971fd_bis in every failed list, and the hang is in a pass, not the last buffer.3b4dc0b6a51cbd4b_b, which ranks high in your B and C, isn't in our failed list.We read the failed pass offline: its task, mesh and vertex shaders do no raw global loads, and its only loop is in the task
a5288d0a6c4971fd, bounded by a constant-buffer value (c3[193].xy). So a garbage count there is the remaining way for it to spin. We now log that value at a failure and will report what it shows.ycarogabriel-lgtm commentedon Nov 8, 2026
More base M1 data on memory and swap (MacBook Pro M1, 16 GB, macOS 26.6.2; OoT v131072; scripted Kokiri Forest walks of 300-420 s on a local branch of open PRs, older main).
Measured: the game alone sits at about 10-13 GB footprint (about 10.8 GB in the app, 12.4 GB GPU allocated on the Metal HUD; 13.0 GB at frame 25,800 in one long session). Each walk adds about 1 GB of swap; with other loads on the Mac it reached 5-9 GB.
Measured, so memory pressure alone is not the trigger here: a hang at swap below 5 GB with all pool chunks off; and no hang with swap up to 8.9 GB when the
a5288d0a6c4971fdtask draws were dropped. With the task draws cut into their own command buffer, 4 of 4 420 s walks were clean (details on #1326).Not tested:
DM_GUEST_FILE=1A/B (our one attempt did not enable it for lack of free disk, so it is invalid), and 04419bc or later main for GPU allocated (our runs predate it).To reproduce: sample
vm.swapusageandkern.memorystatus_vm_pressure_levelevery 2 s during the load and the area change.Task isolation off by default; DM_NVN_ISOLATE_TASK=1 turns it on for โฆ
Task isolation off by default; DM_NVN_ISOLATE_TASK=1 turns it on for โฆ
Drop DM_NVN_A5288_C3 from the OoT diagnostics PR