Week 7: Making a software interop

Where this week started

Last week ended with the synchronization working. Two GPU clocks taking turns over one shared counter, NVDEC frames reaching the screen without a single CPU copy along the way. That was the part I was least sure I could pull off, and finishing it put the merge request in a state Thomas could review properly.

He reviewed it and handed back a priority list of what was still missing:

1. software pictures
2. VAAPI
3. MediaCodec

So this week was software.

Why software needs to be an interop at all

Software being first surprised me, because the libplacebo display already handles software pictures perfectly well. It has always had a plain upload path that takes the decoded planes and calls pl_upload_plane on them. Nothing is broken there.

The reason to move it is the other GSoC project on VLC's video output this summer. Abdelrahman Gamal has that one. Thomas Guillem mentors both of us. The two subjects were split back so they would not collide: I got decoder interop and the external rendering path, he got the filter chain. He is building the placebo filter capability that runs libplacebo filters, plus a GPU deinterlacer on top of it, and all of it plugs into the same display module I have been working on for two months.

A filter should not have to care where its input frame came from. If NVDEC frames arrive through an interop module and software frames arrive through a special case buried in display.c, then his filter code has to know the difference and handle both. If software is also an interop, there is one way in and his code only has to learn one. That is the whole argument, and it is a good one.

The OpenGL side of VLC already works this way. Its interop system has a software implementation sitting next to the hardware ones, so this is bringing the Vulkan path in line with a structure that has been there for years.

The vtable was the wrong shape

I started by reading the interface I wrote for NVDEC, and it was obvious that a software picture does not fit it. Every per-frame callback assumes the frame is already sitting in GPU memory owned by some other API:

int  (*import_images)(interop, VkImage *out, sync *s, picture_t *pic, uint32_t slot);
int  (*setup_acquire_sem)(interop, VkSemaphore sem, uint32_t slot);
VkSampler (*get_sampler)(const interop *);
void (*set_read_fence)(const interop *, picture_t *pic, int fd);

Read those with a software frame in mind and they all fall apart:

• import_images has nothing to import. The pixels are in ordinary host memory, so there is no foreign image to wrap.
• setup_acquire_sem has nothing to wait on. No second API is touching the frame, so there is no semaphore to import.
• set_read_fence has nobody to tell. There is no decoder buffer to hand back once the GPU has finished reading.
• get_sampler is pointless. Nothing here is an opaque external format, so the default sampler is already right.

Two ways out of that:

• Make the software module pretend to be a hardware one, allocating its own Vulkan images and hand rolling the staging copy into them. The header stays frozen, which is tempting when the header is not mine. But it means writing a page of Vulkan buffer and barrier code to reimplement something libplacebo already exports, and it leaves two callbacks as dead stubs that exist only to satisfy a shape.
• Widen the interface with an upload style callback next to the import style one, and let each module implement whichever pair suits it. Costs a header change and a branch in the display.

I went with widening.

The part I did not expect

Once the upload callback existed, I noticed while wiring it up that it never touches Vulkan. All it needs is a pl_gpu, which is libplacebo's own device handle. It never asks for a VkDevice, a queue or a semaphore.

That matters because libplacebo runs on OpenGL too. The whole interop system was built assuming it could borrow Vulkan handles from the backend, and it refuses to start if those handles are missing, which is exactly the case on the OpenGL backend. So I made the Vulkan device optional: hardware modules still require it, upload modules do not. The software interop now runs on both backends. I tested it with --pl-gpu=pl_vulkan and --pl-gpu=pl_opengl and got the same module loading either way.

The hardcoded NV12 finally went away

I ended the week 5 post saying the color description was still hardcoded to NV12 and needed to come from the negotiated format. It is gone now, and the software interop is what forced it.

A hardware picture's format is opaque. It describes no planes and no color, so the display cannot read the colorimetry off it and the code just asserted NV12 and moved on. That is wrong for the 4:4:4 NVDEC formats and completely wrong for a software picture, whose format is a real chroma that should be used as is. The same assumption existed one level down, in how planes map to color channels.

The fix is a field on the format probe where the module names the software chroma its planes actually hold. NVDEC fills in what it decoded and the software module fills in what it negotiated. Playing a hardware file now reports the opaque format and the real one next to each other. Small thing, but it means the renderer is being told rather than assuming.

The plane mapping got the same treatment. The old code said plane zero is one channel of luma and every other plane is two channels of chroma, which describes NV12 and nothing else. It is now derived from the per plane formats and checked once when the output opens, so an unsupported layout falls back instead of drawing garbage. The three plane 4:4:4 formats were already breaking that rule before software arrived.

Here is the MR with all my work (if you are reading this later, some of this week's commits may be force-pushed and reordered, so the exact history might look different): here