ipustack.sys (NPU MCDM driver): long-running NPU inference triggers repeated TDR live kernel dumps (LKD_0x141_Tdr:6) — ~0.7s system stalls and multi-GB transient commit every ~38s

[tomhda]

Summary

Running a legitimate long NPU inference through VitisAIExecutionProvider (a single Run taking ~6.6 s on the NPU) makes Windows' dxgkrnl declare an engine timeout on the NPU device and capture throttled live kernel dumps about every 38 seconds for as long as inference continues:

• WER bucket: LKD_0x141_Tdr:6_IMAGE_ipustack.sys (0x141 = VIDEO_ENGINE_TIMEOUT_DETECTED, blamed image = the NPU MCDM kernel driver ipustack.sys, v32.0.203.329)
• Each capture freezes the whole system for ~0.7 s and transiently commits up to ~7 GB of memory (free/zero page lists drain by the same amount; available RAM momentarily drops to ~1.2 GB).
• The inference itself is unaffected: no context reset, no errors, no submission loss, no measurable slowdown. The dumps are pure collateral from the timeout heuristic.

End users cannot reasonably be asked to modify TdrDelay / CrashControl registry values, so we believe the NPU KMD should accommodate long-running compute without tripping dxgkrnl's engine-timeout detection (MCDM long-running workload handling), or the timeout policy for NPU engines should be adjusted.

Environment

• Ryzen AI 7 PRO 350 (XDNA2 NPU), Windows 11 Home 10.0.26200
• NPU driver 32.0.203.329 (ipustack.sys, IpuMcdmDriver), driver date 2025-12-04
• Ryzen AI Software 1.8.0, onnxruntime 1.27.0 (VitisAI EP)
• Workload: SwinIR-M x4, 256x256 BF16 (the model from issue VitisAI EP (VAIML): native assertion crash ('Iteratees do not have equal length') on Slice pair with negative starts/ends + Concat (torch.roll export pattern) — 3-node repro #397 with the negative-Slice workaround applied), ~6.6 s per inference, 100% NPU subgraph

Evidence

1. Event log: 27 LiveKernelEvent entries since 2026-08, all with the same bucket LKD_0x141_Tdr:6_IMAGE_ipustack.sys, and only during time windows when the NPU was running this workload. C:\Windows\LiveKernelReports\WATCHDOG\WATCHDOG-*.dmp timestamps match one-to-one.
2. Pool tags synchronized with every spike (sampled via NtQuerySystemInformation(SystemPoolTagInformation) at 1 s): vTDR (owned by dxgkrnl.sys) release → ~2 s later vTDR re-allocation + Ldmp (ntoskrnl live-dump) allocation — this is the commit spike — → ~1 s later Ldmp free. The Ldmp allocation tick coincided with the memory spike in 16/16 captured events across three runs.
3. 0.1 s resolution sampling (1974 samples): commit +6.3–7.1 GB per event at 11–16 GB/s, system unschedulable for 0.66–0.71 s (both our sampler and xrt-smi probes stop being scheduled), released ~0.4 s after peak. Kernel pool deltas are only ~0.03% of the commit delta, and the sum of all user-process private commit stays flat (±66 MB), so the allocation is the live-dump capture itself.
4. Not a device allocation: the NPU adapter's DXGI shared usage stays constant to the byte (4.2620 GB across all 1974 samples).
5. No actual hang: xrt-smi shows the hardware context Active with Migr=0, Err=0 and monotonically increasing submissions through every event; tiles that overlap a dump complete in normal time (6.19 s vs 6.48 s median for non-overlapping tiles).
6. Periodicity: one dump per ~38 s ≒ every 6th inference at 6.6 s/inference (WER live-dump throttling). Models whose single inference stays well under the timeout (0.25–0.6 s — e.g. quantized Real-ESRGAN tiles, SESR) never trigger it.

Reproduction

1. Run any VitisAI EP model whose single Run keeps the NPU busy for >~6 s (we use SwinIR-M 256x256 BF16 from issue VitisAI EP (VAIML): native assertion crash ('Iteratees do not have equal length') on Slice pair with negative starts/ends + Concat (torch.roll export pattern) — 3-node repro #397; a large enough RRDB or a looped kernel should also work).
2. Watch Committed Bytes / Available MBytes at 1 s intervals and the Application event log for LiveKernelEvent code 141.
3. Observed: a multi-GB commit spike + ~0.7 s system stall every ~38 s, one WATCHDOG-*.dmp per event.

Expected behavior

Long-running NPU compute submitted through the official Ryzen AI stack should not be classified as a hung engine. Either the MCDM driver should split/annotate long dispatches to stay within dxgkrnl's expectations, or NPU engine timeout policy should tolerate legitimate long kernels. A user-visible ~0.7 s whole-system stall plus a ~7 GB transient commit every 38 s makes the NPU lane noticeably intrusive for exactly the "background inference" use case NPUs are marketed for.

Relationship to #397

Issue #397 covers the VAIML compiler crash that previously prevented SwinIR from compiling at all; with the workaround described there the model now runs 100% on the NPU (PSNR 38.5 dB vs fp32). This issue is the remaining OS-interaction problem when actually running it.