Video agent use cases

Can an AI agent check its own render before publishing?

Yes Reliable to automate today. Last checked 2026-10-14

What you are actually trying to do

You want a pipeline that refuses to publish a broken file, rather than one that discovers the problem after a thousand people have seen it.

The realistic standard is not 'is this good' — it is 'is this definitely broken', which is a question software can answer with certainty.

What works today

Deterministic checks catch almost everything: file size, real duration, stream presence, resolution, frame rate, audio loudness, black-frame and silence detection. No model, no cost, no ambiguity.

blackdetect and silencedetect catch the two most common silent failures in one pass each.

Loudness measurement against a target catches the 'why is this video so quiet' complaint before it happens.

A vision model on sampled frames handles the rest: is the caption cut off, is the subject framed, is this frame obviously corrupted. Sample frames — do not send the whole video.

Taste is not QC. Do not let a model block a publish because it thinks the pacing is off.

Where it breaks

Exit code 0 means nothing. ffmpeg routinely succeeds while writing a black or truncated file: the pipeline produced a black or 0-byte file.

Checking the wrong file. Pipelines that write to a temp path and publish from another will QC a file nobody ships. Check the exact artefact that gets uploaded.

Thresholds too tight cause constant false failures and the team starts overriding the gate, which is worse than having no gate.

No check on audio at all is the most common omission — a silent video passes every visual test ever written.

The gate warns instead of blocking. A warning in a log nobody reads is not a gate.

How to wire it up

1. Probe the artefact you are about to publish. Assert it exists, is above a sane byte floor, and has both a video and an audio stream.

   ffprobe -v error -show_entries format=duration,size \
     -show_entries stream=codec_type,width,height,r_frame_rate \
     -of json out.mp4

2. Assert the real duration is within tolerance of the planned duration from your EDL. This catches truncation and desync in one check.

3. Scan for black frames and silence. Any black run over about half a second, or silence where narration should be, is a hard fail.

   ffmpeg -i out.mp4 -vf blackdetect=d=0.5:pix_th=0.10 \
     -af silencedetect=n=-50dB:d=1.5 -f null - 2>&1

4. Measure integrated loudness and fail outside your target band, rather than discovering it on upload.

5. Sample three to five frames — opening, a captioned section, the end — and send them to a vision model with a narrow checklist: caption cut off, subject framed, visible corruption. Yes or no answers only.

6. Make the gate blocking, and make its failure message name the exact check and the measured value. 'QC failed' with no number teaches nobody anything.

What to use

• ffprobe — duration, streams, dimensions, byte size. Free and certain.
• ffmpeg blackdetect / silencedetect / loudnorm — the three checks that catch the majority of real failures.
• Any vision model — framing and caption checks on sampled frames, not whole video.
• Your own EDL — the source of truth the render is measured against.

Related questions

Should a vision model do all of QC?

No. It is slower, costs money, and is less certain than ffprobe about things ffprobe knows exactly. Use maths for measurable facts and vision only for judgement.

What should QC never block on?

Subjective quality. A gate that blocks on taste gets disabled within a week, and then nothing is checked at all.

What is the minimum worth implementing today?

Four checks: non-zero size, duration within tolerance, audio stream present and not silent, no long black run. That is an afternoon of work and it catches most of what actually goes wrong.