Scene-Change Frame Sampling
When you sample frames from a video for an LLM you have a fixed frame budget, so the policy that picks which frames decides how much of the visual channel survives. The naïve policies — a fixed count spread over the whole video, or a fixed interval (one frame every N seconds) — allocate the budget by clock time, which is the wrong axis: information in edited video is bursty and often front-loaded, so uniform sampling wastes frames on static stretches and misses the cuts, cards, and motion graphics where the content actually lives. On long content it collapses (100 frames over a 10-hour course ≈ one frame / 6 min — "the transcript plus a few random screenshots"). Scene-change sampling instead spends a frame each time the picture actually changes (e.g. FFmpeg scene detection above a threshold), so coverage tracks visual information density instead of the clock. The robust version pairs it with a coverage floor — a guaranteed frame every K seconds regardless of scene activity — so long quiet segments are never fully starved. It is content-adaptive sampling applied to the frame-budget problem inside Multimodal Video Ingestion.
Claims
- Fixed-interval / fixed-budget frame sampling allocates by clock time, so it misses bursty and front-loaded visual information. observation — the failure mode that motivates the whole idea; the sample rate is uncorrelated with where content changes.
- Sample on scene change, not on a timer — spend the frame budget where the picture actually changes. (best practice) — context: edited video (talks, courses, tutorials, anything with cuts and graphics) where visual change correlates with information; for a static single-shot recording (locked-off webcam, screen with no motion) the correlation breaks and uniform coverage is as good or better. This is a contingent "best," not an invariant — its worth is exactly its context.
- Pair scene detection with a coverage floor so long static
stretches still get sampled. (best practice) —
context: real videos mix bursty edits with long quiet segments; pure
scene detection can under-sample the quiet parts, so a guaranteed floor
(firehose uses one frame / 30 s,
origin: coverage-floor) backstops it. Tune the scene threshold and floor to the frame budget. - Frame sampling is budget allocation, not extraction — the scarce resource is the model's attention, not disk. principle — durable: you are choosing what a bounded reader gets to see; the policy's job is to maximize information per frame spent.
Related
- Multimodal Video Ingestion — the parent problem; scene-change sampling is how you produce the "frames" half of the contract.
- Distillate: This Claude Skill Watches Videos So You Don't Have To.