Week 8: reshaping the branch, then running out of hardware

Where this week started

The branch has been growing for two months, so it is worth saying where it stands. VLC can take a frame that NVDEC decoded on the GPU and hand it to libplacebo without a copy through system memory, which is what this project set out to do. Software frames go through the same interop interface, so a filter written against it does not have to care which kind of picture arrived. That part runs on the OpenGL backend too, which I did not plan and still think is the nicest accident of the summer.

Then Thomas reviewed the whole merge request and left twenty comments, and this week became about answering them. A couple were real bugs. Most were about the shape of the branch rather than what it does.

Splitting five commits into nine

The largest job was not a fix at all. My branch was five commits, and one of them was doing far too much: it added the Vulkan device helpers, the module loader, a build system change, and a new code path in the display, all in one go. Reviewing that means holding four unrelated things in your head and judging each of them at the same time.

It is nine commits now, each doing one thing, with the helpers arriving just before the code that uses them so nothing sits around unused waiting for a later commit to give it a purpose. The behaviour is identical either way.

Then VA-API, and a machine that cannot run it

With the comments answered I moved on to VA-API, which is next on Thomas's list. Before writing anything I checked what my machine could test. Less than I had assumed, it turned out.

I have an NVIDIA card and a Ryzen processor, and I had been quietly assuming the AMD half of that gave me a second GPU to test against. It does not. Turns out my processor is one of the desktop chips with no integrated graphics at all, so the only graphics device in the box is the NVIDIA card. The only VA-API driver that can bind to it is a shim that turns VA-API calls into NVDEC ones, and VLC refuses that driver on purpose:

const char *vendor = vaQueryVendorString(dpy);
if (vendor != NULL && strstr(vendor, "NVDEC") != NULL)
{
    msg_Err(o, "Rejecting VA-API NVDEC driver: %s", vendor);
    goto error;
}

VLC is right to do that. On an NVIDIA card you should be using the nvdec module directly, not a translation layer pretending to be something else. It is still an awkward thing to learn on the day you planned to start writing the module.

What I tried before accepting it

I assumed this was a configuration problem rather than a hardware one, so I spent a while looking for a setting that would give me a GPU.

I went into the BIOS looking for an integrated GPU to switch on, which is the normal fix when a machine claims not to have one. There is no such setting. There is no such GPU. That was when I stopped hunting for a workaround and checked the hardware itself, which reports exactly one display controller and no AMD graphics driver loaded at all.

What I told Thomas

I sent him the responses to his comments, along with the hardware problem. I can patch out that rejection locally and run the code, but what I would be testing is NVDEC frames travelling through a VA-API shaped interface. It exercises the plumbing and proves very little about VA-API itself. I would rather say that up front than have him find out later from a green tick that does not mean what it looks like.

He came back the same evening and answered more than I had asked:

• The file I could not decide on goes in video_output/vulkan, since it does not depend on libplacebo at all.
• Before any of this goes public, software and NVDEC 4:2:0 10-bit have to run clean, with no Vulkan validation warnings, using a 10-bit HEVC file that decodes to P010.
• Buy a cheap Radeon card and check VA-API decode on it.

So that part is settled. I will have to buy hardware.

What happens next

The 10-bit testing is the gate, so that is where I started. I encoded a 10-bit HEVC file and confirmed the card decodes it at all before trusting anything downstream of that, then ran it through both paths with the Vulkan validation layers turned on. Both clean.

The software decoder gives a real 10-bit chroma and it goes through as three planes. NVDEC gives an opaque picture that names P010 as the format its planes hold, and that goes through as two. No dropped frames, and nothing from the validation layers in either run.

Here is the MR with all my work (if you are reading this later, some of this week's commits may be force-pushed and reordered, so the exact history might look different): here