Video Ingestion System

A Kubernetes-native, distributed AV1 transcoding pipeline that runs on the homelab cluster. Upload a video, and it gets probed, optionally tuned, encoded to AV1, named, quality-scored and published to object storage with a download link. It is a personal project in a private repository, so this page describes it in general terms.

Origin

It started as a single-host bash script, first written in April 2025, that ran on vulcan, the dedicated AV1 encoder described on the homelab page. That script is no longer the running system, but it still earns its keep as the equivalence baseline: the new system's pinned encoder parameters have to match it. In May 2026 the script grew into a Kubernetes-native distributed transcoding pipeline.

Architecture

Eight Go binaries, split into long-running services and one-shot workers:

• Ingest API and web UI - two Knative services that scale with demand. Uploads stream into object storage and can be resumed, so a dropped connection does not mean starting over
• Orchestrator - a Deployment with leader election, so exactly one replica drives the pipeline at a time. It owns the job state machine, admission and queueing, retries and notifications
• Step workers - five of them (probe, CRF search, encode, naming, VMAF), each run as a Kubernetes Job. AV1 encodes take hours, so a step is a Job the orchestrator creates and watches, not a request it waits on
• Postgres (CloudNativePG) - the single source of truth. There is no message bus: the orchestrator finds and advances work by polling the database
• Object storage - S3-compatible, MinIO in the homelab, holding the sources and the finished outputs

The job state machine

Every job walks the same path. CRF search is the one optional state; when it is turned off, probing goes straight to encoding.

RECEIVED -> QUEUED -> PROBING -> [CRF_SEARCHING] -> ENCODING
         -> NAMING -> VMAF_SCORING -> FINALIZING -> COMPLETED

  any active state -> FAILED     (retries exhausted)
  any active state -> CANCELLED  (cancelled by the user)

VMAF scoring is informational. If it cannot produce a score, the job moves on to finalizing without one; a missing score never fails a job.

Toolchain

• HandBrakeCLI - the AV1 encode itself, using the SVT-AV1 encoder
• ab-av1 - VMAF-sampled CRF search with SVT-AV1, to pick a quality setting per source before the full encode
• VMAF - scores the finished AV1 output against the source

Engineering practices

• Requirements as Gherkin - the intended design is written down as Gherkin feature specs before the code that implements it
• Identity - OIDC in front of the UI and the ingest API, and SPIFFE for the workers, each trading its SPIFFE identity for short-lived object-storage credentials through STS
• Observability - every binary exports logs, traces and metrics over OpenTelemetry, and worker traces stitch under the orchestrator's span for each step
• Releases - a version-file bump goes through a pull request; merging it builds the images, and Flux rolls the new version out to the cluster