Multimodal Video Ingestion
To make a video legible to a text+vision LLM you reduce it to the two channels the model can read: frames (sampled still images = the visual channel) and a transcript (aligned text = the audio channel). Ingestion is then two jobs: get a good transcript, and choose good frames — then fuse them so the model sees "what was on screen while this was said" rather than two disconnected streams. The transcript is the cheap half (pull free platform captions when they exist; fall back to ASR — Whisper/Groq/OpenAI — when they don't, as with Loom or Zoom recordings). The frames are where the real design choices live, because you have a fixed attention/token budget and must decide which stills earn a place — see Scene-Change Frame Sampling. The fused product is a structured, timestamp-keyed report, not a raw dump.
This is the exact contract firehose consumes: claude-watch's
/scry emits a manifest.json of
frames[] (each with a t_s) and
transcript.segments[] (each with
start_s/end_s), and the distiller fuses each
frame with the transcript segment whose span contains its timestamp.
firehose never touches ffmpeg/yt-dlp — it owns the manifest contract and
consumes it only.
Claims
- A video, to an LLM, is two things — frames and transcript; reduce it to both. principle — durable framing independent of tools; the whole ingestion problem decomposes along this axis.
- The transcript is the easy channel; frame selection is the hard one. observation — captions/ASR are cheap and linear; picking frames is a budget-allocation problem (Scene-Change Frame Sampling).
- Fuse frames with their aligned transcript segment — align each frame to the caption span that contains its timestamp (nearest preceding on a gap). (best practice) — context: building illustrated notes where the value is "what was shown as this was said"; unaligned frames + text lose the cross-modal signal that justified sampling the frame.
- Get transcripts from free platform captions first; fall back to ASR (Whisper/Groq) only when none exist. (best practice) — context: cost/latency-sensitive ingestion of web/meeting video; captions are free and instant, ASR is the paid fallback for caption-less sources.
- Own the frames+transcript contract; don't re-touch the raw
video stack downstream. (best practice) — context: a
pipeline where extraction and analysis are separated (as in firehose's
/scryseam); the manifest is the single source of truth, so ffmpeg/yt-dlp run exactly once, at the seam.
Related
- Scene-Change Frame Sampling — the sub-problem inside the "frames" half: which frames to keep given a fixed budget.
- Distillate: This Claude Skill Watches Videos So You Don't Have To.