Week 9: what VA-API breaks that NVDEC did not

Where this week started

Last week ended with me admitting I could not test VA-API at all. My machine had one NVIDIA card in it, the only VA-API driver that could bind to it was a shim VLC rejects on purpose, and the honest position was that I needed hardware. This week the hardware turned up and I could finally write the thing.

The other thing that happened this week is that the NVDEC and software interops finally went up on VLC's own repository, after living on my fork all summer. That is merge request 9856.

Before writing it I read. Four things, in roughly this order:

• the VA-API interop on my mentor's vulkan-vout branch, the experimental Vulkan output this project has been moving over to libplacebo, line by line
• the interop contract it has to plug into, and the NVDEC module sitting next to it, so I could see where the two diverge
• VLC's OpenGL VA-API interop, which has been solving the same export problem for years and was worth reading for how it handles the awkward parts
• the parts of the Vulkan specification covering imported memory and semaphores, which turned out to matter more than I expected

What VA-API is, and where it already differs from NVDEC

VA-API is an interface for asking a graphics card to decode video. Intel designed it, and it is the standard one on Linux: the driver underneath can be Intel's, AMD's or somebody else's, and the program asking for a decode does not care which.

You hand it a compressed stream and it hands back a surface. A surface is an opaque token meaning a decoded frame lives here. You cannot read it, you cannot ask what layout it is in, and you certainly cannot pass it to Vulkan. Getting anything out of a surface means asking VA-API to describe it in terms somebody else understands, which on Linux means a dma-buf.

NVDEC, which is the interop I already have working, does not behave like that at all. It gives back CUDA memory, and CUDA and Vulkan have their own private arrangement for handing memory to each other. No dma-buf, no negotiation about layout, no third party involved. That difference is small at the source and grows into almost everything else in this post.

The other pieces involved

Vulkan is an interface for asking a graphics card to draw. It knows nothing about video decoding. To draw a frame it needs a VkImage, which is its own description of a picture in memory.

A dma-buf is a Linux mechanism for sharing memory between two pieces of software. You get a file descriptor, which is a number, and whoever holds that number can ask the kernel for the same memory.

libplacebo is the rendering library VLC uses for the actual drawing. It does not take a VkImage directly. It takes its own texture object, built by wrapping one.

So a frame travels like this:

Three steps, and none of them moves a pixel:

1. The export asks VA-API to describe a surface as file descriptors. It reports how many buffers there are, how many planes, which buffer each plane sits in, at what offset and with what row stride.
2. The import creates a Vulkan image for the same pixels, then allocates memory that is not an allocation at all, only a handle to the buffer that already exists.
3. The wrap builds the libplacebo object around that image, and allocates nothing.

Why format modifiers exist

An image in memory is not a grid of pixels stored row by row. Graphics hardware stores them swizzled and tiled, sometimes compressed, because that suits how the hardware reads images. That is fine while one piece of software owns the memory, and it becomes a problem the moment you share it, because the receiving side has to know how the bytes are laid out or it reads nonsense.

A DRM format modifier is a 64-bit number naming one specific arrangement. The exporter says this buffer is in layout number such-and-such, and the importer either understands that number or refuses the buffer. Most of the VA-API interop code is about getting that number right.

None of this came up while I was working on NVDEC, and now I understand why. CUDA and Vulkan share memory through a mechanism of their own, and the copy into module-owned images means nobody has to agree on a layout at all. Going through dma-buf makes the layout the whole negotiation.

The same layer, two very different customers

NVDEC copies. The decoder hands over CUDA memory and the interop copies it, on the card, into Vulkan images the module allocated itself. It is a copy, but one that never leaves the card. VA-API does not copy at all. It takes the memory the decoder already wrote and describes it to Vulkan in place.

VA-API sounds better there, and in one way it is, but the copy buys NVDEC something valuable: its images belong to it and never change.

	NVDEC	VA-API
how the frame arrives	copied on the card into module-owned images	imported in place, no copy at all
the images	a fixed pool, the same image every time	one per decoder surface, dozens of them
memory layout	CUDA memory, modifiers never come up	dma-buf, where a modifier describes every buffer
synchronization	a timeline counter shared with CUDA	a sync file the timeline cannot accept

What I think will break

In week 5 I found the display destroying and recreating the wrapper around each plane image every frame, which libplacebo's documentation says is undefined behaviour. I replaced it with a cache: wrap each image once, keep it, reuse it. I wrote at the time that the images do not change for the life of the output and only their contents do.

That was true because NVDEC copies. Copying into its own fixed pool is what makes its images stable. VA-API imports instead, so its images belong to the decoder's surfaces, and the decoder cycles through several dozen. The slot the core picks and the surface the decoder produced have nothing to do with each other.

So the renderer would keep drawing the same pictures. They are genuine frames, so the picture looks correct. What is missing is everything in between, and to a viewer that reads as a very low frame rate. That's an actual problem that is going to need some redesign.

Two smaller things from the same place

A dma-buf can carry a sync file, a fence meaning the decoder has finished with this buffer. You can hand that to Vulkan and have the card wait instead of the processor. The vulkan-vout version does that, and it is the right way round.

It cannot work as the contract stands, and the reason is the counter I spent weeks 5 and 6 building. The specification says a sync file must be imported temporarily, and that a temporary import may not target a timeline semaphore. The shared counter both sides wait on is a timeline, because that is what two GPU clocks touching one image needed. So the only synchronization object available is the one that cannot take a sync file.

The other goes back to the two ways of handing libplacebo its planes, from weeks 3 and 7. The upload route gives the module a libplacebo device and lets it own its textures. The import route returns raw Vulkan images and leaves the core to own the wrapper. That asymmetry is what creates the slot problem above: the core ends up owning an object whose lifetime somebody else controls.

Shaping the contract around NVDEC was reasonable when NVDEC was the only thing using it, and I would not have known which parts were choices without a second module to compare against. That is the conversation I want to have with Thomas before writing much more.