Hardware transcoding Jellyfin on an Intel NUC

Ryle · Sep 24, 2026

My home media server is Jellyfin running in a container on a small Intel NUC, serving my own library to a TV and a few other clients. For a while any playback that needed a transcode fell apart. The picture would stall, the client would restart the stream, and the bitrate would sink to about 300 kbps. It took some digging, but the cause ended up being the iGPU's encoder driver, which only accepts one rate-control mode, and it's not the one Jellyfin asks for. Once I fixed that, two more problems showed up: Dolby Vision Profile 5 files transcoding pink and green, and embedded subtitles taking minutes to appear.

Software encoding on four cores

At first it looked like a network problem: play a few seconds, rebuffer, restart at a lower bitrate, repeat. But the network was fine. The path to the server answered in about 50 ms the whole time.

The actual problem was in Jellyfin's encoding settings. Hardware acceleration was set to VAAPI, but hardware encoding was off. VAAPI is the Linux API for the Intel GPU's video engine, and with that combination Jellyfin decoded on the GPU and then encoded with software libx264 on the NUC's four CPU cores. Those cores were also busy with background jobs (a file sync scan was pinning one, so I disabled it). The encoder couldn't produce HLS segments as fast as the client used them up, so the player kept running dry, and the client's adaptive bitrate logic read the late segments as a slow link.

Exit code 234: the driver only does CQP

So I turned hardware encoding on, which I figured would fix it. Instead every transcode failed right away with FFmpeg exited with code 234, which the client showed as "media is not supported by this client". The ffmpeg log had the real reason: Driver does not support VBR RC mode.

Rate control is how an encoder decides how many bits to spend. VBR (variable bitrate) aims for a target average bitrate with a ceiling. CQP (constant quantization parameter) holds a fixed quality level and lets the bitrate end up wherever it ends up. On this iGPU, the Intel iHD VAAPI driver only supports CQP for H.264 encoding, so any encode that asks for -rc_mode VBR fails before it writes a frame.

And Jellyfin always asks for VBR. Whenever it targets a client bitrate, which is every transcode, it builds h264_vaapi arguments like -rc_mode VBR -b:v ... -maxrate ... -bufsize .... So on this hardware a stock Jellyfin hardware transcode can't succeed, and I haven't found a Jellyfin option in this image version that changes the rate-control mode. An encode the driver does accept looks like this:

ffmpeg -vaapi_device /dev/dri/renderD128 -i input.mkv \
  -vf 'format=nv12,hwupload' \
  -c:v h264_vaapi -rc_mode CQP -global_quality:v 23 \
  -c:a copy output.mkv

A wrapper in front of ffmpeg

What I ended up doing is putting a small shell script where Jellyfin expects its ffmpeg binary. It takes Jellyfin's argument list, drops -rc_mode, -b:v, -maxrate and -bufsize along with their values, inserts -rc_mode CQP -global_quality:v 23 after the input file, and then execs the real ffmpeg with the rewritten arguments. The real binary gets moved to ffmpeg.real next to it.

The downside is that the output bitrate isn't tied to what the client asked for anymore. A client on a constrained link gets whatever bitrate quality 23 produces, because Jellyfin's bitrate cap never makes it to the encoder.

After the change, a live transcode ran at about 17x realtime, ffmpeg exited 0, and the client's bitrate climbed back up once segments were arriving ahead of playback. That's from one session I checked; I didn't benchmark across files or resolutions.

Keeping the wrapper in place

The annoying part is that half of this fix doesn't stick. The wrapper script lives in Jellyfin's persistent config volume, so it survives restarts. The binary swap lives in the container's own filesystem, so any container recreate or image update puts the stock ffmpeg back. Nothing tells you this happened, the next transcode just fails with exit 234 again.

So I wrote a second script that puts the swap back. It checks three things inside the container: whether the ffmpeg path is a bash script, whether ffmpeg.real exists, and whether the wrapper is present in the config volume. If the first two are true, it exits without touching anything. Otherwise it moves the stock binary aside (only if ffmpeg.real doesn't already exist, so it handles a half-applied state from a crash), copies the wrapper into place, points it at the real binary, and restarts the container. There's also a --check mode that only reports and never restarts.

A host cron job runs it every five minutes, so after an image update it fixes itself. Since applying the fix restarts Jellyfin, the script first asks Jellyfin's sessions API whether anything is playing, and if so it waits for the next pass. There's a FORCE=1 override for when transcoding is already broken and waiting doesn't help.

This is a workaround and I'd still like to get rid of it. The cleaner fix would be setting CQP through a native Jellyfin option, if this image version has one, and putting the original binary back. Failing that, the swap should at least survive a recreate. I haven't done either yet.

Pink and green Dolby Vision

Once transcoding worked, some files played with a pink and green picture on clients that got a transcode, while clients playing the original stream directly looked fine. All of the affected files were Dolby Vision Profile 5 with a base-layer compatibility id of 0.

The thing that matters here is what each Dolby Vision profile's base layer contains. In Profile 7 and Profile 8.1, the base layer is ordinary HDR10, and the Dolby Vision part is extra metadata (or an enhancement layer) on top. A player that ignores Dolby Vision still gets a correct HDR10 picture. Profile 5 has no fallback like that. Its pixels are stored in Dolby's own color representation, and only a Dolby Vision-aware decoder or tone mapper knows how to turn them into normal color.

The ffmpeg logs I still had for the reported files showed VAAPI H.264 transcodes where the only color handling was setparams=...bt709 followed by a VAAPI scale. That tags the decoded frames as BT.709, the standard SDR color space, without actually converting anything. Relabeling Profile 5's pixels as BT.709 tells everything downstream to interpret the numbers in the wrong color space, and you get the pink/green cast. Changing the color tag won't fix it; the data needs an actual conversion.

Tone mapping is the conversion that would fix it, and I didn't have it. Both tone-mapping options were off, and initializing the OpenCL backend Jellyfin uses for it failed with Failed to get number of OpenCL platforms: -1001. Even if OpenCL worked, only a Dolby Vision-aware path handles Profile 5, so I left it off. For Profile 5 files, what's left is getting a version of the file without Dolby Vision, or a validated DV-aware transcode backend. Stripping the Dolby Vision metadata doesn't work here: there's no HDR10 base underneath, so the stripped file would show the same wrong colors everywhere.

Losslessly stripping Profile 7 and 8.1

For the profiles that do have an HDR10 base, removing Dolby Vision is cheap and lossless, and it helps my TV's client, which mishandles Dolby Vision anyway. ffmpeg's hevc_metadata bitstream filter can drop the Dolby Vision NAL units while copying every stream as-is:

ffmpeg -i input.mkv -map 0 -c copy \
  -bsf:v hevc_metadata=remove_dovi=1 -f matroska output.mkv

I have a cron script run this over the library every ten minutes, using the container's own ffmpeg and ffprobe, and it holds a flock so overlapping runs exit right away. For each file it reads dv_profile and dv_bl_signal_compatibility_id from the first video stream's side data. It only strips Profile 7, or Profile 8 with compatibility id 1. Anything else with Dolby Vision (Profile 5, 8.2, 8.4, 4) gets left alone and added to a list of files that need different handling.

A file only gets replaced after the remux passes three checks: the output has no Dolby Vision left, the video is still HEVC, and the durations match within two seconds. If any of those fail it deletes the temp file and keeps the original. It skips files modified in the last five minutes, and a state cache keyed on modification time and size means a normal pass over an unchanged library costs almost nothing. There's a seed mode that marks the existing library as known, and a dry-run mode that reports without changing anything.

One known bug: the filter applies to every video stream, so it fails on some hybrid files that have an embedded cover-art image as a second video stream. The verification catches that and keeps the original, but those files stay unstripped until I change the script to map only the main video stream.

Eight minutes to show a subtitle

The last one had nothing to do with the GPU. Picking an embedded subtitle track often did nothing for minutes, and the track would only show up after I'd switched between tracks a few times.

Jellyfin extracts embedded subtitles on demand, and one request extracts every subtitle stream in the file in a single ffmpeg pass before it serves any of them. That pass reads the whole container. On a 2160p file with 18 subtitle tracks sitting on the NUC's USB library disk, it took about eight and a half minutes. The client gives up long before that, and the track only works once a later request finds the finished extraction in the cache. Across the whole library, only 26 items had ever been extracted.

A sidecar file avoids all of that. A .srt or .ass file next to the video gets returned as an external stream and read straight off disk, no ffmpeg involved. So the fix is making sure a sidecar exists. For files that have an embedded English text track, a background job writes one sidecar per file with a targeted -map of just that stream. It's still a full read of the container, so I kept the job narrow on purpose: newest files first, a 35-day window by default, one extraction at a time, and it stops as soon as anyone starts playing something. Image-based subtitles (PGS, VOBSUB) can't be turned into a text sidecar, so those go on a separate list.

What I'd take from it

Every one of these looked like something else at first, whether that was bandwidth, codec support or a broken track. In each case the ffmpeg log or a single timing measurement showed what was actually going on, so now I check those first. The other thing I'd do again is pair any fix that lives in a container's filesystem with a scheduled check that puts it back, and have that check wait until nobody is watching before it restarts anything.