Week 2: NVDEC the easy way first

Part 1: making it work where it already works

Last week I decided NVDEC goes first. This week I made it actually play, but on the path VLC already has, not the one I am building.

VLC can already show NVDEC frames through OpenGL. It ships a small module, glinterop_nvdec, that hands a CUDA-decoded frame to OpenGL without copying it. So before I rebuild any of that on Vulkan and libplacebo, the sensible first move was to confirm the easy path runs on my machine and my file.

Getting there was mostly driver wrangling. I had been on the open nouveau driver, which gives you no CUDA, so NVDEC would not build at all. I switched to NVIDIA's proprietary driver, installed CUDA and nv-codec-headers, and then the nvdec plugins compiled. After that, the run itself:

vlc --vout=gl --codec=nvdec --play-and-exit test.mp4 -vv

The log shows the whole chain line up: the nvdec decoder device, the nvdec decoder, an opaque NVD8 chroma (the frame stays on the GPU), glinterop_nvdec doing the zero-copy handoff, and the gl output drawing it. A 4K H.264 file plays to the end with no CUDA errors.

It is not interesting to watch. It is the same video. The point is that it is now my reference. When the Vulkan version draws garbage, and it will, I can run the same file on the same GPU through this path and compare.

Part 2: reading, and a first call about the project

Most of the week was reading. My mentor has a full Vulkan output branch that already does hardware decode to screen, and my job is to move that logic onto libplacebo. So I went through it and there was concepts I wasn't fully understanding, so I used internet videos, guides and articles to understand more.

Then I had my first video call with him. It was less a code walkthrough and more about the project as a whole: where it is going, what counts as done, and the order things should land in. I went through the points I had built up from the reading and the design notes and checked that I was lined up with what he actually wants, and that I was not missing some larger piece. A few of my assumptions he confirmed, a couple he reframed, which is cheap to fix on a call and expensive to find out three weeks into writing code.

We ended with an agreement on where to start. The first piece is the on-screen video output, with NVDEC as the proof of concept. That keeps the scope small and lines up with the easy NVDEC path I already have running, so I can build the real thing next to a reference that works.

Part 3: a bug I tripped over on the way

While setting up to test all this, I hit a real bug, and it was not in my own code. Running the libplacebo output with its OpenGL backend on Wayland, the video showed up in the bottom part of the window with a black band across the top.

It is pre-existing, not something I introduced, and it only shows in a narrow spot: forcing the OpenGL backend of this output, under Wayland. I opened an issue ticket on the VLC main repo about it. I might come back later and try to fix it myself too.

The issue I opened: here