Skip to content

Ocarina of Time on an M1 (16 GB): GPU hang (kIOGPUCommandBufferCallbackErrorHang) under memory pressure, about 24 fps at 720p / 30 fpsย #1605

Open

Description

@CrampingCondor

Built from the fork cheemu21/kid-with-sword-remake (its V6 state), not from main; the fork has issues turned off, and most of its Ocarina of Time code comes from here. Possibly related to #1326 (a GPU hang about 10 minutes in), but here it comes within seconds to minutes, on an M1, together with memory pressure.

Setup

  • MacBook Air M1 (8-core GPU, applegpu_g13g), 16 GB, fanless
  • macOS 27.0.1, Xcode 27.2, Python 3.15.2 (Homebrew, arm64)
  • The fork at d1d672ac (branch kid-with-sword-remake)
  • Game: a .nup with update v131072 and no base game (the script took the update as the whole game). One RomFS file is reported damaged and replaced at every start: Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs ("not a zstd stream, begins 00 00 00 00")
  • Built with --platform mac-app --env-file mac_env.txt (DM_UPSCALER=metalfx, DM_FPS_HOLD=off, DM_ALIAS_SIZE=0x200000000) and the resolution overlays from the game note (handheld 0.666667, docked 0.9375, flat). Build: 42,853 of 42,854 shaders, pipeline archive for applegpu_g13g made (30,515 render + 4,125 compute bases)
  • Launcher: Handheld, 30 fps, Upscaling off, full screen

What happens

The picture freezes for good while the game goes on (sound and input still work). The log shows a segment command buffer failing with Caused GPU Hang Error (00000003:kIOGPUCommandBufferCallbackErrorHang), then kIOGPUCommandBufferCallbackErrorSubmissionsIgnored on every later submission.

It happened in every one of 7 runs, at the start of the game, between a few seconds after loading the save and about 3 min of gameplay (frames 565 to 3836). In the last run I stood still for 1.5 minutes without a hang; it froze a few seconds after I started walking (frame 3512). An earlier guess (picking up a blue rupee) did not hold: it also froze without one.

No more memory pools than the 4095 entries line (the known Ocarina of Time hang in docs/reference/shaders.md), no new pipelines or shader libraries in the frames before the hang.

What I tried (no change to the hang)

Change Result
720p overlay instead of the game's dynamic resolution GPU time about the same (about 50 ms โ†’ 53 ms per frame in gameplay), still hangs
DM_GUEST_FILE=1 footprint about 11 GB โ†’ 8.4 GB, frame p50 about 45-53 ms โ†’ 41 ms, GPU allocated unchanged (about 9 GB), still hangs
DM_NVN_DISPATCH_OVERLAP=0 still hangs

Performance before the hang

  • Frame time p50 about 42 ms (about 24 fps) even when standing still; the host steps the present interval between 2 and 3 all the time
  • DM_NVN_GPUPROF=every:300: GPU busy 17-22 ms a frame (render 12-19 ms, compute / copy 4-6 ms), but first start to last end 27-44 ms, so the GPU is idle for about 40% of the frame span. Largest passes: the first compute / copy encoder (about 5 ms, about 100 dispatches) and the 1280x720 main passes (2-4 ms)
  • GPU allocated 8.4-9.9 GB, resident 7.5-8.4 GB (with guest files)

The hang comes with memory pressure

In a 7th run I sampled kern.memorystatus_vm_pressure_level and vm.swapusage every 2 s next to the game. Loading the save took free memory from 74% to about 46%; then macOS reported pressure level 2 (warn) for 11 s, swap grew from 1.5 GB to 3.0 GB, free memory fell to 21%, and the GPU hang (frame 565) fell inside that window, a few seconds after the save had loaded. Pressure went back to normal right after the hang. Activity Monitor showed the game at about 13 GB at that moment. This also fits the other runs: no hang standing still, a hang soon after loading or walking into new parts of the area.

For comparison, the fork's game note (docs/games/kid-with-sword-remake.md) measured a peak footprint of 5.2-5.5 GB on a 16 GB MacBook Air M5 with tex_release 2; here the footprint is 8-11 GB (8.4 GB with guest files, whose pages still take RAM) with about 9 GB GPU allocated. I do not know why the M1 needs so much more, or whether the scenes compare.

DM_NVN_HANGLOG, mentioned in docs/reference/shaders.md, does not seem to exist in the fork's tree, so I could not name the hung shaders. I can provide full logs if useful.

Activity

  1. lunks commented on Nov 7, 2026

    @lunks
    Contributor

    Thanks for the careful report and the memory sampling. Three things would help tell the causes apart:

    1. From the log, the lines containing pools without a buffer address left out (they give a count and megabytes).
    2. The last pipelines built or loaded before the hang, and whether any of them is a task or mesh pipeline.
    3. A run on a build from main (or a fresh clone of this repository) instead of the fork, with the same save, so we know whether the fork's code matters.

    If the last pipelines are task/mesh ones, #1585 is a candidate: it fixes a hang where a culling task shader reads a pointer into a memory pool missing from the raw pool table. That is unproven for your case. The memory-pressure window you measured also fits the system-memory stop in #1589, which is open.

  2. LeonardoBai12 commented on Nov 7, 2026

    @LeonardoBai12
    Contributor

    Three GPU hangs from another M1 with 16 GB, two of them on a build from main, with the shader ranking the host now logs. This answers your three questions, @lunks, as far as these logs can.

    Setup

    The three hangs

    Run When Hang lines Outcome Footprint / GPU allocated Memory pools
    A 182 s, frame ~4480, rolling in Kokiri Forest 8 ร— ErrorHang, then 28 ร— SubmissionsIgnored picture froze, game closed 10.5 GB / 12.8 GB 4,684
    B 256 s, frame 4866, Kokiri Forest 4 ร— ErrorHang, 1 failed command buffer recovered, play went on 10.7 GB / 13.0 GB 4,553
    C 2,333 s, frame 85995, inside the Deku Tree 4 ร— ErrorHang, 1 failed command buffer recovered, play went on 11.2 GB / 12.9 GB 4,559

    The memory columns come from the last memory up to frame and on-screen lines before each hang.

    1. pools without a buffer address left out: none in any of the three logs.

    2. The last pipelines before the hang

    • In A and B, the last pipelines built before the hang were task/mesh pipelines with the same task shader, about 200 frames earlier:
      • task a5288d0a6c4971fd_b, mesh 0a44d06526b90d19_b, fragment b5e76fbd0fdfe143_b
      • task a5288d0a6c4971fd_b, mesh cdccb0b8e37b83b1_b, fragment 5978c86a51251517_b
    • In C the last task/mesh pipeline was built much earlier, at frame 78429 (task a5288d0a6c4971fd_b, mesh 9b11e51ca22b3608_b, fragment aa3f1dca9e8828f0_b). No new pipelines or shader libraries in the frames just before.
    • a5288d0a6c4971fd_b is in the failed command buffer's shader list in all three runs. It is probably drawn in most frames, so this may mean nothing.

    3. On main: B and C are on main 24fe1e5 plus the PRs above, not on a fork. None of those PRs is #1585.

    The host's ranking (game shaders by failed/ran), first entries. Each log has one failure, so the ranking is thin.

    • A: 0a44d06526b90d19_b 1/104, b5e76fbd0fdfe143_b 1/104, 5be2891fce11da4c_b 1/134, 5fcff4b4a6aa8d71_b 1/143, 92890e7c46754e66_b 1/145
    • B: ten shaders at 1/44 (1baec668d4dba75c_b, 23863906e989c1ca_b, 3a4fe787c1901068_b, โ€ฆ), then 3b4dc0b6a51cbd4b_b 1/100. The A pair is further down at 1/185.
    • C: 3b4dc0b6a51cbd4b_b 1/157, then fa20be590aae1d99_b 1/839, 6bb6772fb6988a3f_b 1/853, fb46ac21db7ab722_b 1/853
    • No shader is in all three rankings. 3b4dc0b6a51cbd4b_b ranks high in both runs on main.

    Not checked: whether macOS reported memory pressure at those moments (I didn't sample it), and #1585 or #1589 on this Mac. I can post the full logs, or try a build with #1585 if that helps.

  3. LeonardoBai12 commented on Nov 7, 2026

    @LeonardoBai12
    Contributor

    Three more sessions on the same M1 Pro (16 GB) and build (main 24fe1e5 plus #1543, #1497, #1331, #1453, #1377, #1379). All three ended in GPU access revoked and exit 70. #1332's alert worked each time.

    Session Hang episodes (time since launch) Last one GPU allocated before it
    D page faults at 1,006 s (4 ร— ErrorPageFault), then hangs at 1,333, 1,850, 2,352 and 2,666 s 2,666 s, in a potion shop, catching a bug in a bottle 16.5โ€“17.8 GB through the session (more than the Mac's RAM)
    E 36, 657, 930 and 1,243 s 1,243 s, falling into the water outside Hyrule Castle 13.2โ€“13.8 GB, steady
    F 990 s 990 s, just after leaving the castle courtyard 13.3 GB

    In D and E the game recovered from the earlier episodes. The last one repeated on every frame until macOS revoked access.

    On memory. D alone would point to memory pressure, but E and F hang at the same ~13 GB as the sessions that recovered, with no other apps open. So I don't think memory pressure explains the hangs here. A report on #1326 shows the same hang on an M2 Max with 32 GB.

    Shader ranking. I compared the top 40 shaders of the game shaders by failed/ran line across all six sessions (Aโ€“F). No shader appears in three or more of them. The hangs also came in different places: walking, rolling, cutscenes, a shop, water. I can't see a single draw that triggers them.

    Next: I'm building main bd979ce with #1585 (rebased; chain 9eb5e5a9a0ec here) and will report whether the hangs change on this Mac.

    Not checked: memory pressure sampled from outside the game, and DM_NVN_GPU_WATCH from #1550 (not in this build).

  4. LeonardoBai12 commented on Nov 8, 2026

    @LeonardoBai12
    Contributor

    Two more sessions on the same M1 Pro (16 GB, macOS 26.6.1), now on a build from main e940e1f (its own frame step, fps_target 60) plus #1707, #1497, #1331, #1453, #1377 and #1379. Chain 094f939bfef9, all 42,848 shaders built, render scale 0.7, handheld, DM_UPSCALER=game.

    Both sessions ended in GPU access revoked and exit 70, at the same place: leaving Hyrule Castle with Impa, in the cutscene that shows Hyrule Field.

    Session Frame rate Hangs Kept log
    G 30 fps cap fatal, about 03:22 local no (overwritten)
    H 60 fps cap, hold on 4 ร— ErrorHang at about 430 s (recovered), then fatal at 444 s (frame ~3000) yes

    What H shows

    • The culling tasks 47f9b38490dcf9a2 and 99280b7b8aae78df are in all three failed command buffers that list shaders. f75708d176bb6591 is in none.
    • GPU allocated: 3.7 GB up to frame 900, then 15.1 GB from frame 1200 on, 15.6 GB at frame 3000 (resident 11.5 GB). The earlier builds sat at about 13 GB.
    • The 9 gpuEvent-dmhost-*.ips reports from these two sessions are all MMU interrupt (restart_reason 10) with fw_ta_state busy and fw_cl_state idle, as in the earlier ones.
    • Apple's shader compiler crashed while the game ran: 11 MTLCompilerService reports (EXC_BAD_ACCESS, SIGBUS or SIGSEGV, in libLLVM.dylib under AGXCompilePlan::execute), and the host logged XPC_ERROR_CONNECTION_INTERRUPTED for task a7f22b4a795ff4d1_b / mesh ac2f3feb11427426_b / fragment eddc12907cefc9af_b. I don't know whether this is related to the hang.

    So on this Mac #1707 didn't stop the hang, and it came sooner than on the 24fe1e5 build (16โ€“44 min there, about 7 min here). The base moved too, so that is not a clean comparison. This matches @ralph-ferza's result on #1326 with DM_NVN_BARRIER=full.

    Next: a run with DM_NVN_SKIP_MESH=1 on the same route, to see whether the hang goes away here as in @reefbluesky's runs.

  5. LeonardoBai12 commented on Nov 8, 2026

    @LeonardoBai12
    Contributor

    DM_NVN_SKIP_MESH=1 result on the same M1 Pro and build (main e940e1f plus #1707 and the PRs listed above):

    • One session of 2,901 s (frame 59,565), ended by quitting (exit 0). No ErrorHang, PageFault or GPU access revoked, and macOS wrote no gpuEvent report.
    • It passed the spot where the last two sessions died (leaving Hyrule Castle with Impa, the Hyrule Field cutscene), then Kakariko Village, the graveyard in the rain and a long walk on Hyrule Field.
    • GPU allocated was 15.1โ€“15.7 GB for the whole session, the same as the sessions that hung, so the memory level doesn't decide it here.
    • The picture is wrong, as expected: fences, crates and some rocks are missing.

    So on this Mac too the hang goes away with the task / mesh draws dropped, as in @reefbluesky's runs on #1326.

  6. LeonardoBai12 commented on Nov 8, 2026

    @LeonardoBai12
    Contributor

    A long session on main 04419bc, on the same M1 Pro (16 GB, macOS 26.6.1). On top: #1331, #1453, #1377, #1379 and the SDK guard; no #1707, #1497 or #1585. Chain 89f44f9b6bb0, all 42,848 shaders built. Handheld, DM_UPSCALER=game, DM_FPS_TARGET=60, DM_DYNAMIC_RESOLUTION=0 (full resolution), DM_PSO_MTL4=0.

    Result: 4,402 s of play (Kakariko, the graveyard, the mask quest, Castle Town and back, Hyrule Field, Death Mountain), ended by quitting (exit 0). There was no GPU access revoked.

    The hang is still there, but every episode recovered:

    • 44 ร— ErrorHang, 22 failed submissions, from 948 s to 3,625 s (948, 1122, 1125, 1330, 1339, 1415, 1466, 1775, 1776, 1790, 1814, 1998, 2009โ€“2013, 2151, 2161, 2164, 2228, 2693, 3625).
    • macOS wrote 7 gpuEvent-dmhost-*.ips reports, all MMU interrupt with fw_ta_state busy, as before.
    • The culling task 47f9b38490dcf9a2 is in 10 of the 16 failed buffers that list shaders, 99280b7b8aae78df in 6, and f75708d176bb6591 in none.
    • From about 30 min on, the ground in Hyrule Field and Kakariko went missing for moments. I think these are the failed buffers, but I haven't matched the times.

    GPU allocated was 12.4โ€“14.0 GB.

    Other lines from this log:

    • Apple's compiler failed again on task a7f22b4a795ff4d1_b / mesh ac2f3feb11427426_b / fragment eddc12907cefc9af_b at 843 s (XPC_ERROR_CONNECTION_INTERRUPTED).
    • [RED] texture type mismatch: slot 7 declared as Metal type 3 while texture ID 878 is type 7 (NVN target 2, format 0x3f), for vertex shaders 9b310b3349825e9a_b, c0aff0eee66cda0f_b and 09ab31c69af710c9_b. On Death Mountain a broken-looking piece of geometry was floating in the air. I can't tell whether it is related.

    I can't say which of 04419bc's changes made the episodes recoverable here. The faster 128-bit loads in the mesh shaders are one guess, not tested.

  7. LeonardoBai12 commented on Nov 8, 2026

    @LeonardoBai12
    Contributor

    A second session on the same build (main 04419bc and the same settings as above): 2,261 s on Death Mountain, in Goron City and the Lost Woods, ended by quitting (exit 0).

    • One hang episode at 463 s (2 ร— ErrorHang, 1 failed submission), which recovered. Culling task 47f9b38490dcf9a2 was in its failed buffer. Nothing else for the rest of the session.
    • Apple's compiler failed on task a7f22b4a795ff4d1_b once more, at 75 s.
    • GPU allocated 12.6 GB at the end.
    • Pipeline prepare took 5.6 s with DM_PSO_MTL4=0 (146 s on the first start after the chain change).

    To me this session played almost perfectly.

  8. lunks commented on Nov 8, 2026

    @lunks
    Contributor

    Thanks, this is useful. We had a lead from one of our own hangs (M5 Pro, a build with 04419bc2 and without ea0c98df): the failed pass held 13 shaders that had run in no earlier command buffer, and two task/mesh pipelines were built at first use at frames 4264 and 4265 (comment on #1326). Your three runs don't fit it: the last task/mesh pipeline was built about 200 frames before the hang in A and B, and about 7,500 frames before it in C. So we're dropping "a freshly built pipeline" as the cause.

    What your runs and ours share: task a5288d0a6c4971fd_b is in every failed list, and the hang is in a pass, not the last buffer. 3b4dc0b6a51cbd4b_b, which ranks high in your B and C, isn't in our failed list.

    We read the failed pass offline: its task, mesh and vertex shaders do no raw global loads, and its only loop is in the task a5288d0a6c4971fd, bounded by a constant-buffer value (c3[193].xy). So a garbage count there is the remaining way for it to spin. We now log that value at a failure and will report what it shows.

  9. ycarogabriel-lgtm commented on Nov 8, 2026

    @ycarogabriel-lgtm

    More base M1 data on memory and swap (MacBook Pro M1, 16 GB, macOS 26.6.2; OoT v131072; scripted Kokiri Forest walks of 300-420 s on a local branch of open PRs, older main).

    Measured: the game alone sits at about 10-13 GB footprint (about 10.8 GB in the app, 12.4 GB GPU allocated on the Metal HUD; 13.0 GB at frame 25,800 in one long session). Each walk adds about 1 GB of swap; with other loads on the Mac it reached 5-9 GB.
    Measured, so memory pressure alone is not the trigger here: a hang at swap below 5 GB with all pool chunks off; and no hang with swap up to 8.9 GB when the a5288d0a6c4971fd task draws were dropped. With the task draws cut into their own command buffer, 4 of 4 420 s walks were clean (details on #1326).
    Not tested: DM_GUEST_FILE=1 A/B (our one attempt did not enable it for lack of free disk, so it is invalid), and 04419bc or later main for GPU allocated (our runs predate it).
    To reproduce: sample vm.swapusage and kern.memorystatus_vm_pressure_level every 2 s during the load and the area change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions