Week 4: the green noise that proved the obvious fix was wrong

Part 1: the error I ended last week on

Last week ended on a good picture and a bad verdict. The Vulkan and libplacebo path was drawing NVDEC frames with correct colors, but the moment I turned on Vulkan's validation layer it flagged the code as illegal. The frame's image is "disjoint" (each plane has its own chunk of memory) and at the same time "dedicated" (that memory is tied to the whole image), and the spec forbids both at once. NVIDIA's lenient driver drew it anyway. This week was about actually fixing that, and the fix was not the one I reached for first.

Part 2: the experiment that looked like a fix and was not

The obvious move was to just remove the dedicated part and satisfy the spec. I did, the warning went away, and the picture fell apart: the luma stayed perfectly sharp, but the color went to green noise across every flat area, the sky and the water, while the detailed foreground hid it better.

That green was the useful result of the week. It taught me something: on NVIDIA the dedicated allocation is not optional. It is how CUDA reconstructs the image's memory layout when it maps the frame, and without it the chroma plane is read wrong.

Part 3: the corner, and the real fix

So I am left with a clean statement of the problem. The design I inherited needs a dedicated allocation for the CUDA import (the green noise proved that), but the spec forbids a dedicated allocation on a disjoint image. Both cannot be true at once on a single image.

I took this to my mentor, and his answer closed the loop on the whole week. The disjoint image only ever existed to present one combined image to that hardware YCbCr sampler, the exact path I had already decided to drop because libplacebo does its own color conversion. So the constraint that forced the illegal combination is one I no longer have.

That points straight at next week's work: stop using one combined image, and give libplacebo two separate single-plane images instead, one for luma and one for chroma. Each becomes an ordinary, non-disjoint image with its own perfectly legal dedicated allocation, which is exactly how the OpenGL interop already does it: one texture and one mapping per plane. It should remove the validation error and keep the picture correct, because nothing about the CUDA import changes, only how many images it feeds. That is the plan I am starting from on Monday.

The bigger thing I am taking from this week is how much the validation layer earned its keep. It caught a spec violation that a "it plays fine" screenshot would never have revealed, and chasing it down taught me more about how Vulkan memory, CUDA interop, and NV12 actually fit together than any amount of reading would have.

Here is the MR with all my work (if you are reading this blog later, the changes mentioned today will be force-pushed later, so you might not find it): here