f_autoconvert: convert between hw formats via system memory - #18300
f_autoconvert: convert between hw formats via system memory#18300flowreen wants to merge 4 commits into
Conversation
|
You should be able to sequence it using the format filter to first implicitly convert by specifying format=nv12 or whatever the right one is, and then format=d3d11va (or whatever we call it). You could also ask Claude to implement the cuda d3d interop path. CUDA supports it but no one has been motivated to do it so far. |
|
I'm not a big fan of round-tripping transfer through system memory silently. Depending of image resolution and framerate this can be a lot of bandwidth that can silently kill the performance on lower end gpus. You should be about to export frame from CUDA, Vulkan, D3D12 as win32 handle and import that directly to d3d11, it will still do copy, but hopefully the GPU->GPU one. Also you can already use nvdec with d3d11vpp, like so |
|
Good idea, and you were both right that the round trip should not be the default. It turns out it can be done like this instead: the first version routed through system memory whenever the source was a hw format, which also meant a pair that could be mapped would never get the chance to. This version only falls back to system memory when the pair has no mapping available. It now decides per candidate target: bool download_first = !imgfmt_is_sw &&
!mp_hwupload_can_map(fmts[i], img->imgfmt);with Before, with After: For reference, on master with Based on Claude investigation, it found that It also confirmed the paths that were already zero copy are untouched and stay silent: d3d11 + d3d11va still goes Testing it ran: 2x and 3x @kasper93 on @philipl on the cuda/d3d interop suggestion: it did not get there. What it did get is the prerequisite, device derivation for D3D11VA, which lavu does not implement at all: That is device level only. It shares no frames and removes no copies, it just puts both devices on the same adapter. I plan to send it to ffmpeg-devel, but would rather hear any objections here first. It found the interop itself blocked in two places: lavu's (As I understand it: D3D11 hands out a frame as one slice of a stack of pictures, saying "here is the stack, take slice 3". The Vulkan side has nowhere to write down "slice 3", it can only point at a whole picture. So the only D3D11 texture it can accept is one holding exactly a single frame. A fix I have not attempted: lavu's dynamic d3d11va pools already allocate one texture per frame through It would also not remove the need for this branch regardless: D3D11 interop is one directional. Vulkan can import a D3D11 texture, D3D11 cannot import Vulkan memory. So vulkan to d3d11 has no mapping in that direction at all, and something has to carry it. |
|
You'll have to spend some time thinking about what design actually makes sense here. I don't know if Claude if capable of doing a good job on its own; my experience suggests it probably would not, and the comments reflect that. The cuda <-> vulkan interop is an example of one where there is a one-way import restriction. Cuda cannot export, it can only import - so the model is that vulkan always exports an image and then cuda imports it, and if the data needs to flow from cuda to vulkan, cuda does a GPU-side memcpy. And, in fact, even when data is flowing from vulkan to cuda, we do a copy anyway for consistency. GPU-side memcpys are cheap. So if d3d11 requires a one-way relationship, you'd probably want to handle it in the same way. You should look at/point Claude at the cuda interop API docs. Between that and the existing ffmpeg code, there should be enough general guidance. The majority (all?) of the work required here is on the ffmpeg side. As for whether any of the resulting code would be accepted, I don't know. ffmpeg doesn't have an official policy, but in practice this means "If the code meets the project standards and you can communicate about it showing an actual understanding of what you're doing, then who cares what tools you use". I would not consider what we have right now as meeting that bar, but I wouldn't be the one reviewing or approving any resulting ffmpeg PR. The fundamental d3d11 (or 12) interop with Vulkan and Cuda is interesting, and on some level long sought after, but it's a messy exercise that clearly hasn't motivated anyone sufficiently over the years. |
I agree, it may not be able to contextualize everything that's going on, let alone see the sane solution, except surface level glue/workarounds. Our frame copy/mapping/conversion code is already quite complex in what it does, and it's easy to get lost in it, with all the corner cases.
This is more complex interop, and basically needs not only "convert" code work, but also pre-existing knowledge in frame allocator. Though, it's not really nothing special. Most of interop need to have frames/memory allocated in certain way that is compatible with "exporting" or "sharing".
I always prefer to have same device pipeline, without sharing. I rectal trying to evaluate d3d12 hwdec use directly to our d3d11 renderer, but it was slow. I don't recall now if this was only GPU-GPU copy, or something more, but was disappointed in the result, where native d3d11 was faster in this case and better.
Yeah, most would be on ffmpeg side, maybe with an API to request sharing between certain apis up front. Though, I have not though about what is available now. I don't want to be this guy, but since we have multiple APIs, which all could be interoperable, this could be done in generic way, with some abstraction, maybe exiting ffmpeg API is enough. But the point being, that we should think about extensibility, not saying that every interop needs to be implemented, but most of the logic is in fact common between those APIs, they often use same underlying primitive to share memory. Anyway, I know nothing, I have not did into just now. |
Shareable textures cost nothing and allow libavutil to transfer frames directly to another device, for example d3d11 to vulkan.
libavutil can transfer frames between two hardware devices directly on the GPU only for some device pairs and surface formats, and there is no way to query support other than trying it. Attempt the direct transfer and fall back to downloading and reuploading the frame, remembering the result. Previously a failing direct transfer just failed the filter. Also add mp_hwupload_probe_hw_to_hw(), which tests with an actual frame whether an upload target can be reached without a copy through system memory, either by mapping or by a direct transfer.
The RA format description is only needed by vo_gpu. A libplacebo based VO ignores it and maps planes by their Vulkan format, which works for bit packed formats like X2BGR10 that ra_get_imgfmt_desc() cannot express. Keep failing for other RAs.
When a hw source has to be converted to a different hw format, build the chain around the uploader instead of rejecting the format. The frame goes through system memory only when a sw format conversion is needed, and when sw targets are also available the hw target is preferred only if a test transfer shows the frame can actually stay on the GPU. Previously this case either failed outright, because a direct transfer between two devices was attempted and libavutil could not do any, or the frame was downloaded even when the two devices could share it.
c0e65d2 to
e0bf0bc
Compare
3 participants
In short: nvdec produces CUDA frames,
d3d11vpponly accepts D3D11 frames, and mpv tries to upload one into the other directly. libavutil cannot transfer between two hardware frame contexts, so it fails on the first frame and the filter is silently disabled, leaving--vf=d3d11vppusable only with--hwdec=no. This routes the frame through system memory instead, using the download stagebuild_image_converter()already has but never built for this case.Background
This came out of trying to make
d3d11vppusable with--gpu-api=vulkan, which someone ran into here: https://www.reddit.com/r/mpv/comments/1v6msec/do_i_need_to_change_config_to_enable_rtxhdr_in_mpv/That is one user report rather than a groundswell, but the underlying bug here is not specific to that filter or to Windows: any hw-only target fed by a different hw decoder hits it. The
d3d11vpphalf is sent separately; this one is the generic part and stands on its own.Problem
When the target only accepts hardware formats and the decoder produces a different hardware format,
build_image_converter()hands the frame straight tomp_hwupload. libavutil cannot transfer between two hardware frame contexts, so it fails on the first frame:Reproducer on Windows with an NVIDIA GPU:
--hwdec=autoselects nvdec, and those CUDA frames cannot reach the D3D11 video processor. Playback continues with the filter silently disabled.Fix
The function already declares three stages, and
mp_chain_filters()wires all of them:but the
dst_all_hwbranch hardcodedhw_to_sw = false, so stage 0 was never built. Request the download when the source is a hardware format, and stop suppressing the scaler in that case, since the trip through system memory is what makes a sw format conversion possible.This branch is only reached when the source's hw format is not among the accepted targets, so the direct transfer it attempted always failed at runtime. The change cannot regress a path that previously worked.
It also removes the check that rejected a target whose upload format differed from the source sub format. That check assigned
filters[2]and shranknum_fmtsbeforecontinue, so the loop always ended withupload_created == falsewhile the rejected uploader stayed chained. It logged "Failed to create HW uploader" and then used it anyway.Testing
Windows 11, RTX 5090, driver 610.74,
--vo=gpu-next.Swept every decoder this machine can produce with
--gpu-api=vulkan --vf=d3d11vpp=nvidia-true-hdr:no,auto(nvdec),auto-copy,nvdec,nvdec-copy,dxva2,dxva2-copy,d3d11va,d3d11va-copy,vulkan. All produce correct output. Before this change the CUDA and Vulkan sources failed. That is two distinct cross-device pairs: CUDA to D3D11, and Vulkan to D3D11.No new copies on paths that were already zero-copy: d3d11 + d3d11va with and without the filter, vulkan + nvdec without the filter, and d3d11 + software decode all still report zero HW-downloading / HW-uploading.
Verified at pixel level rather than only from logs: the same source frame captured through each path gives SSIM 1.000000 and PSNR 95 dB against the d3d11 reference, and the two Vulkan captures are byte-identical to each other. A no-filter control sits at PSNR 28 dB, so the comparison is not vacuous.
Not tested: the vaapi and videotoolbox paths through this branch, as I have no hardware for either. Vulkan to D3D11 working suggests the mechanism is generic rather than CUDA-specific, but I cannot claim more than that.
I do not know this codebase's conventions well, so if you would prefer a different approach, a narrower fix, or this split differently across commits, say so and I will rework it.
AI disclosure
Disclosure: this patch was written with AI assistance (Claude Code), which diagnosed the bug, wrote the change, and ran the testing described above on my machine. I have reviewed it and understand what it changes and why, I take responsibility for it, and review responses will be written by me. filters/f_autoconvert.c carries no licence header and so falls under the repository default of LGPLv2.1+; I submit the change under that licence.