Video agent use cases
Can an AI agent generate and burn in captions?
Yes Reliable to automate today. Last checked 2026-10-14
Yes, completely, and it is the highest-return thing to automate first. Transcribe the finished audio, write SRT or ASS, burn in with ffmpeg. The only rule that matters: caption the audio that exists, never the script you wrote.
What you are actually trying to do
You want accurate, well-timed captions on every video without typing them, including the karaoke-style word-highlight look that short-form formats expect.
It is also an accessibility requirement and a measurable retention factor on silent autoplay, so there is no argument for skipping it.
What works today
Word-level transcription is accurate and cheap, and word-level timing is what makes modern caption styles possible at all.
SRT is universal and fine for upload-as-sidecar. ASS gives you per-word highlighting, positioning and styling for burn-in.
ffmpeg burns either in with one filter, and hard-burned captions survive every platform's re-encode.
Translation of a finished caption file is reliable and gives you multi-language reach at near-zero cost.
Where it breaks
Captioning the script instead of the audio. The read never matches the written line exactly, and the mismatch is obvious on screen. Always transcribe the rendered audio.
Technical terms and names get mis-transcribed. Pass a vocabulary hint list, and keep a find-and-replace map for the terms your content uses constantly.
Unescaped paths break the filter. Windows paths and colons inside subtitles= are the classic ffmpeg failure here.
Captions sit under the platform UI. Keep burn-in inside the safe area — roughly the middle 70% vertically for vertical video.
Style is applied per-caption instead of globally, producing flicker between lines. Set styling once in the ASS header.
How to wire it up
Render the final audio first. Captions are a post-audio step, always.
Transcribe the rendered audio with word-level timestamps, passing a vocabulary hint list of names and jargon.
Apply a correction map for the terms the transcriber reliably gets wrong, before writing the caption file.
Write SRT for sidecar upload and ASS when you want burn-in with per-word styling.
Burn in with ffmpeg, escaping the path carefully.
ffmpeg -i in.mp4 -vf "subtitles=caps.ass:fontsdir=./fonts" \
-c:a copy -c:v libx264 -crf 18 out.mp4
Visually QC one frame from a captioned section — extract a still and check the text is inside the safe area and readable at phone size.
What to use
- Whisper or a hosted transcriber — word-level timestamps with vocabulary hints.
- ffmpeg — burn-in, styling via ASS, frame extraction for QC.
- A text model — translation of the finished caption file, and cleanup of obvious transcription errors.
- Platform sidecar upload — keep the SRT too; search engines and platform transcripts both use it.
Related questions
Burn in or sidecar?
Both. Burn in for short-form, where viewers watch muted and platform captions are off by default. Upload the SRT as well so the platform has searchable text.
How accurate is automated transcription?
High on clean narration, noticeably worse on crosstalk, accents and domain jargon. The correction map is what gets you the last few percent, and it pays for itself after a handful of videos.
Can the agent decide caption style?
Let it choose nothing. Style belongs in a fixed template per format — a model picking fonts per video destroys the visual consistency that makes a channel recognisable.