Video agent use cases

Can an AI agent cut a long video into short clips?

Yes Reliable to automate today. Last checked 2026-10-14

What you are actually trying to do

You have a 40-minute recording, stream or interview and you want six to ten short clips out of it without scrubbing a timeline yourself.

The work splits cleanly in two: deciding where to cut, which needs judgement, and performing the cut, which does not. Agents are good at the first and should never be trusted with the second.

What works today

Transcription with word-level timestamps is solid and cheap. Word-level matters — segment-level timestamps make cuts land mid-syllable.

A text model reading a timestamped transcript picks segments well, because it is a text task wearing a video costume. Ask for strict JSON: start, end, hook, why.

ffmpeg does the extraction. With -c copy it is a stream copy: no re-encode, no quality loss, seconds per clip.

Reframing 16:9 to 9:16 with speaker tracking is the weak link. Hosted tools do it well; a DIY agent doing face-detect crop is visibly worse than a human.

Where it breaks

The agent invents timecodes. It will confidently return a moment that is not there, especially past the 20-minute mark. Validate every start and end against the transcript and the real duration before cutting anything.

Cuts land mid-word. Snap every boundary to the nearest word gap from the word-level timestamps rather than using the model's numbers raw.

-c copy seeks to the nearest keyframe, so a clip can start up to one GOP early. If the start must be exact, re-encode that clip only.

Silent failure. ffmpeg exits 0 and writes a black or zero-byte file more often than you expect — see the pipeline produced a black or 0-byte file.

How to wire it up

1. Transcribe with word-level timestamps and keep the JSON. Everything downstream reads the transcript, not the video.

2. Send the timestamped transcript to a text model and demand strict JSON back, with a hard cap on clip count and a duration range.

   Return ONLY JSON:
   [{"start": 123.4, "end": 167.9, "hook": "...", "why": "..."}]
   Rules: 20-75s each, max 8 clips.
   start and end MUST correspond to words in the transcript.

3. Validate before you cut. Reject any segment where start >= end, where end exceeds the real duration, or where the span does not map onto actual words. This single check kills most failure modes.

4. Snap each boundary to the nearest word gap, then pad 0.25s either side so the first word is not clipped.

5. Cut with ffmpeg, seeking before the input for speed.

   ffmpeg -ss 123.40 -to 167.90 -i in.mp4 -c copy \
     -avoid_negative_ts make_zero out/clip01.mp4

6. Verify every output: non-zero bytes, duration within 0.5s of what you asked for, and a decodable first frame. Fail loudly rather than publishing.

What to use

• Whisper or a hosted transcription endpoint — word-level timestamps. The whole pipeline rests on this being accurate.
• Any text model — segment selection. A cheap text call, not a video call.
• ffmpeg — the actual cut. Free, deterministic, the only part you can fully trust.
• Opus Clip, Vizard, Descript — hosted equivalents that handle reframing and captions for you. Faster to a result, less control, priced per minute.

Related questions

Should the agent run ffmpeg itself?

Let it compose the arguments, but run them through a validated template with numeric values type-checked — not a free-form shell string. An agent writing raw shell is how you lose a source file.

Why not ask a video model to find the good bits?

You pay vision pricing for a question the transcript already answers. Video understanding earns its cost when the moment is visual — a reaction, a whiteboard — not when it is something somebody said.

How good is the model at picking clips?

Good at finding self-contained statements, poor at judging what will perform. Treat its ranking as a shortlist to skim, not a decision.