Week 1: first patch written

Before any of that: a ten-line build fix

Before I could test a single thing, I hit a wall. Building VLC with meson on Wayland, the libplacebo Vulkan output just would not come up. The meson build was missing the small module that turns a Wayland window into a surface Vulkan can draw on. The older autotools build had it, meson didn't, so on Wayland there was no Vulkan platform for libplacebo to use at all. So I wrote a fix for it, and it is already merged to the master branch.

That fix: MR !9328

Part 1: the most boring patch I'll write all summer

VLC's libplacebo output had one function, vlc_placebo_Create(), that sets up the whole GPU in a single shot. The interop work needs to get in the middle of that: Vulkan extensions must be requested before the device is created (you can't add them later), and the interop code needs the raw Vulkan handles afterwards. So my first MR splits it, and changes nothing else:

Before:  vlc_placebo_Create()              // everything at once

After:   vlc_placebo_New()                  // allocate
         vlc_placebo_Connect(params)        // create, with room to inject extensions
         vlc_placebo_GetVkHandles()         // raw handles for interop code

VLC tries Vulkan first and quietly falls back to OpenGL if it fails, and that fallback only survives if creation stays one atomic step. That's why it's a single Connect() taking extension lists, not separate create-instance and create-device calls.

Part 2: the assumption that didn't survive contact with the code

VLC has an experimental Vulkan output (my mentor wrote it) that already solves hardware import, and my job is to move that logic over to libplacebo. I assumed it imported each frame as separate Vulkan images, one per color plane, since that's the shape libplacebo likes. Reading the actual code: no. It imports one multi-planar VkImage per frame and uses a driver feature, VkSamplerYcbcrConversion, to turn the brightness and color planes into RGB at sample time.

The conflict: libplacebo owns its pipelines and samplers internally, and there's no way to hand it an externally created YCbCr sampler. So the import code can't move over as-is. Frames have to be exposed plane by plane, with libplacebo's shaders doing the color math. That's going to need some restructuring.

Part 3: asking the person who wrote it

I mailed my mentor, who is also the author of that code. He used the YCbCr sampler simply because he didn't want to write conversion shaders by hand, and he pointed out it's buggy on some implementations anyway. So, this week's decisions: libplacebo's shaders do the color conversion with frames wrapped per plane, and NVDEC goes first, because CUDA shares memory with Vulkan without the extra layout headaches that Intel and AMD's path brings. Prove the idea where it's simplest as first step.

The code is on https://code.videolan.org/ahmedsobhy/vlc/-/merge_requests/1