Video agent use cases
Can an AI agent colour-match shots from different generations?
Partly Automatable, with a human reviewing the result. Last checked 2026-10-14
Partly. An agent can measure and correct exposure, white balance and contrast automatically, which fixes most of the mismatch between independently generated clips. What it cannot do is invent a look, or rescue shots whose lighting direction genuinely disagrees — that has to be fixed at generation time.
What you are actually trying to do
You generated ten shots for one scene and they arrive at different brightness, warmth and contrast, so the sequence reads as ten unrelated clips rather than one scene.
This is a correction problem, not a grading problem. The goal is to make shots belong together, not to apply a creative look.
What works today
Measurement is fully automatable. Average luminance, black and white points, and colour balance per shot are things ffmpeg will tell you.
Matching to a reference shot works well for exposure and white balance — pick the best-looking shot as the anchor and bring the others to it.
Normalising to a common contrast curve removes most of the 'this clip is flat and that one is crushed' effect.
Lighting direction cannot be fixed in post. If one shot is lit from the left and the next from the right, correction makes it worse, not better.
A creative grade should be a fixed LUT applied after matching, not something a model decides per video.
Where it breaks
Auto-correcting each shot independently makes the mismatch worse, because every shot lands on a different 'correct'. Always match to one reference.
Correcting before normalising format. Mixed pixel formats and colour ranges produce corrections that are mathematically right and visually wrong.
Limited versus full colour range mismatch causes washed-out or crushed output — this is the most common invisible cause.
Letting a vision model guess numeric corrections. It will produce plausible values that are wrong. Measure, then compute.
Full symptom page: colour and exposure jump between AI clips.
How to wire it up
Normalise format first — same resolution, frame rate, pixel format and colour range for every shot — before measuring anything.
Pick one anchor shot. Usually the one you would keep if you could only keep one, or a mid-sequence shot so corrections stay small.
Measure every shot's signal statistics and compare against the anchor.
ffmpeg -i shot.mp4 -vf "signalstats,metadata=print:key=lavfi.signalstats.YAVG" \
-frames:v 60 -f null - 2>&1
Compute per-shot corrections from the measurements and apply them deterministically. Do not ask a model for the numbers.
ffmpeg -i shot.mp4 -vf "eq=brightness=0.03:contrast=1.04:saturation=0.98,\
colorbalance=rs=-0.02:bs=0.03" -c:a copy matched/shot.mp4
Flag rather than correct any shot whose required correction exceeds a threshold — that shot should be regenerated, not fixed.
Apply your creative LUT last, once, to the whole matched sequence.
What to use
- ffmpeg signalstats — per-shot measurement, free and exact.
- ffmpeg eq / colorbalance / lut3d — correction and final look.
- DaVinci Resolve shot match — far better results if a human is in the loop; scriptable via its Python API.
- A vision model — useful only as a final sanity check on stills, never as the source of correction values.
Related questions
Why not just ask the video model for consistent lighting?
You should, and it helps — locking lighting language in the frozen character or scene block reduces the spread a lot. It will not eliminate it, so you still need the matching pass.
Can a vision model grade a shot?
It can describe what is wrong usefully. It cannot produce correction values you should trust. Use it to decide *whether* to act, and maths to decide *how much*.
When should I regenerate instead?
When lighting direction differs, when a shot is clipped at either end of the range, or when the correction needed is large enough to introduce banding. Regenerating is cheaper than a shot that looks wrong forever.