Week 3: first light on the Vulkan path

Part 1: the week I started building the real thing

Last week I got NVDEC playing on the path VLC already has, the OpenGL one, so I would have a working reference to compare against. This week I started building the path that does not exist yet: the same hardware-decoded frame, but going through Vulkan and libplacebo instead.

The goal for the week was narrow on purpose. Not the whole output, not every format, just one thing: get a single NVDEC frame decoded on the GPU, imported into Vulkan without a copy, and drawn by libplacebo with the right colors. If that works, the shape of everything after it is known. If it does not, better to find out in week three than in week eight.

The rule I set for myself was to copy my mentor's working code first as a single commit, and only then start changing it. That way every later commit is a real diff against something that once ran, instead of a wall of new code.

Part 2: the loader

The first piece is the loader, it lets libplacebo's Vulkan output talk to a hardware decoder at all. In libplacebo terms it is a little vtable: probe the format, load the backing images, import a frame, set up the sync. I brought my mentor's loader over from his Vulkan branch and walked it through compiling against libplacebo instead of the old Vulkan output: fixing the includes, pointing it at the interop contract we had already landed, and building it as a static archive the placebo plugin links.

The one real design call was keeping the connect step atomic. The old path had a three-call split, but that would have broken the Vulkan-to-OpenGL fallback, so I kept a single New-plus-Connect. If the Vulkan interop cannot be set up, the output has to quietly fall back to the software upload path rather than show a black screen. That fallback ladder is the whole reason this is safe to turn on by default, and it is worth more than any single feature I added this week.

Part 3: the NVDEC backend, copied and fixed step by step

With the loader in place I brought over the NVDEC backend the same way: copy the whole thing, get it building, then change it in small commits until it fits the new contract.

The interesting part of NVDEC is that the decoded frame never leaves the GPU. It arrives as an opaque chroma VLC calls NVD8, which is really an NV12 surface sitting in CUDA memory. My job is to take that CUDA memory into a Vulkan image without copying it back through the CPU, then let libplacebo draw it. My mentor's code already had the hard CUDA part working, so I kept that and focused on how the Vulkan side hands the result to libplacebo.

Part 4: understanding some stuff

Before I could decide how to hand the frame to libplacebo, I had to actually understand the format instead of copying someone else's choices. So I spent a real chunk of the week reading about chroma subsampling and the NV12 layout, from videos, articles and the Vulkan spec.

The short version I landed on: our eyes are much more sensitive to brightness than to color, so video formats keep the brightness (luma, the Y) at full resolution and throw away three quarters of the color (chroma, the U and V), storing it at half width and half height. That is 4:2:0. NV12 stores it as two planes: one full-size plane of Y, then one half-size plane where U and V are interleaved together.

The question that actually mattered was: who turns that YUV back into RGB for the screen? There are two answers, and picking one decides the whole design. Vulkan can do it in hardware with a special YCbCr sampler, which is what my mentor's branch used. Or you hand libplacebo the raw planes plus a description of the color, and libplacebo does the conversion in its own shader. Because the entire point of this project is to move onto libplacebo and let it own the color pipeline, the second answer is the right one. That is what let me drop the YCbCr sampler path.

The main things I read and watched for this:

• Videos are NOT stored in RGB: YUV vs RGB and digital color explained (YouTube)
• Chroma subsampling and bit-depth explained (4:2:0 vs 4:2:2 vs 4:4:4) (YouTube)
• NV12 format and UV plane (Stack Overflow)
• YCbCr sampler in Vulkan (Stack Overflow)

Part 5: two ways to hand libplacebo the planes

Deciding that libplacebo does the color conversion still left a smaller question: how do I physically give it the two planes? There were two ways to do it, and for a while I thought only one of them was open to me.

The first plan was per-plane. Make a separate Vulkan image for each plane, one for the full-size Y and one for the half-size UV, and wrap each of them on its own as its own texture. Two images, two wraps, and libplacebo gets a clean single-format texture for each plane with nothing left to untangle.

The second plan was multiplanar. Make one two-plane image that holds Y and UV together, wrap it a single time, and let libplacebo pull the individual planes out of it for me. One image, one wrap, and much less of my own code.

At first I assumed the second plan was not even possible. I thought wrapping an image handed you back one opaque texture, and that pulling the separate planes out of it was my problem to solve, so per-plane looked like the only real option. Then, reading through the libplacebo headers, I found that a wrapped multi-planar image already exposes its sub-planes: the texture it gives back has a planes array, one entry per plane. libplacebo does the splitting for you.

So I took the multiplanar route for the spike. It was less code, it leaned on the library instead of me stitching planes together by hand, and it felt like the more elegant of the two. What I did not know yet was that the single combined image is exactly the thing that needs the awkward memory layout validation would object to later, and that the per-plane plan I had just talked myself out of was the one I would end up coming back to.

Part 6: first light, and two crashes on the way to it

Getting the first frame on screen took fixing two crashes that both came from the same root, but they were simple to fix.

After that: actual video. NVDEC decodes on the GPU, the frame is imported into Vulkan with no CPU copy, and libplacebo draws it with correct colors. The same 4K file that plays through the OpenGL reference now plays through the Vulkan and libplacebo path I am building.

Part 7: turning on vulkan's validation

A picture on screen is not the same as correct code. Vulkan has a validation layer that checks every call against the spec, and NVIDIA's driver is famously lenient: it will happily accept things the spec forbids and still draw them. So the frames looking right told me nothing about whether the code was actually legal. I ran again with validation on, and it immediately found a real problem that had been hiding behind that leniency.

The problem was about how the frame's memory is imported. Each plane gets its own chunk of memory, which in Vulkan terms makes the image "disjoint". But the import also marks that memory as "dedicated" to the whole image, and the spec forbids doing both of those at once. On NVIDIA, with validation off, none of this matters and the video plays. With validation on it is flagged the moment it happens.

vk vkAllocateMemory(): pAllocateInfo->pNext.image (VkImage 0xc000000000c[VLC NVDEC Disjoint 0]) was created with VK_IMAGE_CREATE_DISJOINT_BIT. The Vulkan spec states: If image is not VK_NULL_HANDLE, image must not have been created with VK_IMAGE_CREATE_DISJOINT_BIT set in VkImageCreateInfo::flags (https://vulkan.lunarg.com/doc/view/1.4.350.1/linux/antora/spec/latest/chapters/memory.html#VUID-VkMemoryDedicatedAllocateInfo-image-01797)

The picture is on screen, which is the milestone I wanted, but validation says the code that draws it is breaking the spec. Chasing that error down, and finding out how much of the design it takes with it, is next week.

Here is the MR with all my work (if you are reading this blog later, the changes mentioned today will be force-pushed later, so you might not find it): here