This Claude Skill Watches Videos So You Don't Have To

TL;DR

A short creator explainer (Taoufik, 3:56) for claude-watch — a Claude Code skill (fork of bradautomates/claude-video) that lets Claude "watch" any linked video by reducing it to the two things a model can actually read: frames + transcript. The load-bearing idea is the frame-sampling policy. The transcript is the easy half (yt-dlp pulls free YouTube captions; falls back to Whisper/Groq when there are none, e.g. Loom/Zoom). Frames were the hard half: the naïve policy grabs a fixed budget of frames across the whole video (~100), which on a 1-hour video is one frame / 36 s and on a 10-hour course one frame / 6 min — so the model is "basically just reading the transcript with a few random screenshots" and misses everything on-screen (motion graphics, edits, b-roll). The fix is scene-change frame sampling (FFmpeg scene detection): spend frames where the picture actually changes, not on a timer, so the sample tracks information density instead of clock time. The output is a structured "watch report"; the skill's own differentiator is that it optionally auto-saves into a cross-linked knowledge graph ("second brain" in Obsidian) — the note doesn't land in a folder, it wires into the pages you already have. Thin as media, but architecturally it is an independent re-derivation of firehose's own /scry-manifest → vault-graph pipeline.

Concepts introduced

• Multimodal Video Ingestion — new page. The frame that makes video legible to an LLM: a video is frames + transcript; reduce to both, fuse them, and hand the model aligned "what was said while this was on screen" pairs. This is exactly the contract firehose consumes from claude-watch /scry (manifest.json).
• Scene-Change Frame Sampling — new page. The sharp, transferable nugget: sample frames on scene change, not on a fixed timer or fixed budget, so coverage follows where the content changes. The video's real technical contribution.

Held, not dropped (themes touched that don't warrant their own page yet — spin out on demand):

• Second-brain / cross-linked PKM auto-save. The skill's self-described differentiator (visible verbatim in a leaked production beat-sheet at t≈01:58: "Beat 5 — Auto-save — YOUR ORIGINAL MOVE #2, the differentiator … watches AND files, same skill"): after watching, it asks one question — save to your knowledge base? — and if yes, cross-links the note into existing pages rather than dropping it in a folder. Held, not promoted: this is firehose's own thesis (markdown-truth vault + constructive dedup into the concept graph), so for this operator it corroborates rather than teaches — see stance prose below.
• Transcript-source fallback ladder (free YouTube captions → Whisper → Groq → OpenAI). An implementation detail of the ingestion contract; folded into Multimodal Video Ingestion as a claim, not its own page.
• "Watch it so you don't have to" / attention as the bottleneck. The video's whole pitch — offload watching, get a report, do other work meanwhile — is firehose's north star restated by a third party. Held as corroboration, not a concept.
• Fork lineage & framing (taoufik/claude-watch ← bradautomates/claude-video; the "0–10 s hook microscope" extra frames near the open). Provenance, not a concept.

Key claims

• A video, to an LLM, is two things: frames and transcript — reduce it to both. (principle — Multimodal Video Ingestion) — the durable framing; everything else is how well you produce each half.
• The transcript is the easy half; the frames are where the hard choices live. (observation — Multimodal Video Ingestion) — captions are cheap and linear; frame selection is a budget-allocation problem with no free lunch.
• Fixed-budget / fixed-interval frame sampling degrades to "transcript + a few random screenshots" on long content. (observation — Scene-Change Frame Sampling) — concretely: 100 frames over a 10-hour video is one frame / 6 min; the visual channel is effectively lost.
• Sample frames on scene change, not on a timer — spend the frame budget where the picture actually changes. (best practice — Scene-Change Frame Sampling) — context: reading edited video (talks, courses, tutorials) where visual information is bursty and front-loaded; for a static single-shot recording, uniform coverage is fine and scene detection buys little. State the context: it is "best" precisely when visual change correlates with information, which most produced video satisfies but a locked-off webcam does not.
• Keep a coverage floor alongside scene detection so long quiet stretches still get sampled. (best practice — Scene-Change Frame Sampling) — context: pure scene detection can starve a long static segment of frames; firehose's own manifest pairs origin: scene frames with origin: coverage-floor frames every 30 s for exactly this reason.
• Ingestion should end by cross-linking the note into an existing knowledge graph, not filing it in a folder. (best practice — held; folded into stance) — context: personal knowledge management where the value is in connections; this is firehose's constructive- dedup thesis, corroborated here.

Why this is novel (and where it strongly corroborates)

The graph so far is two regions — an evals/continual-improvement spine and an agent-architecture cluster. This capture opens a third, orthogonal region — multimodal ingestion — that attaches to none of the existing concepts, so the dominant stance is novel: two new pages, nothing duplicated.

The high-value thread is corroboration of firehose's own architecture by an independent builder. Taoufik, with no connection to this project, converged on the same pipeline firehose is built around:

1. Frames + transcript as the ingestion contract. claude-watch's frames+transcript report is the /scry manifest.json seam firehose already consumes — an independent witness that "own the frames+transcript contract, never touch the raw video stack twice" is the right seam.
2. Scene-change sampling + coverage floor. The video argues for scene detection over fixed intervals; firehose's manifest already encodes exactly this (origin: scene with scene_score, plus a 30 s coverage-floor). Convergent design, recorded as a backlink from Scene-Change Frame Sampling, not a duplicated concept.
3. Cross-linked graph ingestion = constructive dedup. The "save to your knowledge base, wire it into existing pages" move is firehose's markdown-truth vault + concept-graph thesis. Same idea, independently arrived at — which is why it is held here rather than promoted: for this operator it is confirmation, not new knowledge.

No contradicts tension surfaced. One caveat worth logging: the on-screen "watch report" example (t≈00:00) and the leaked beat-sheet (t≈01:58) show this is a polished, partly staged marketing artifact — the ideas are sound and independently useful, but treat the capability demo as advertisement, not benchmark.

Illustrated walkthrough

• t=00:00 — the output, shown first. Frame f0001: a finished watch report open in Claude Code beside the source video (a Dwarkesh × Karpathy interview). The report leads with a coverage note — "the frame sampler grabbed scene changes… every cut lives in the first ~29 minutes… the production front-loads its edits" — then lists the two on-screen graphics it actually caught (a research-paper card, a GitHub-repo card). This is the product's real deliverable: an illustrated distillate, honest about what the sampler could and couldn't see.
• t=00:30 — the gap the skill closes. Frame f0002: "the old way" — a solid black bar (the transcript, fully captured) beside a hatched empty box (the visual channel, missing). Transcript-only tools left the picture on the floor.
• t=01:00 — what "visual" means. Frame f0003: three labeled cards — motion graphics · edits · b-roll — the on-screen information a transcript can never carry.
• t=01:30 — the fix. (f0004, scene-detection segment) The switch to FFmpeg scene detection: "instead of grabbing frames on a timer, it grabs one frame every time the scene actually changes."
• t=01:58 — the differentiator (and a leak). Frame f0005: title card "save to your knowledge base?" — but the background is the video's own production beat-sheet, legibly showing "Beat 5 — Auto-save — YOUR ORIGINAL MOVE #2 (THE differentiator)… watches AND files, same skill" and the transcript fallback ladder (captions → Whisper on Groq → OpenAI). The scaffolding of a marketing video, accidentally sampled.
• t=02:18 — install. Frame f0006: the GitHub repo taoufik123-collab/claude-watch — "Give Claude the ability to watch any video — frames + transcript + structured report. Fork of bradautomates/claude-video with scene-change frames, 0–10 s hook microscope, and optional Obsidian auto-save." The chapter rail (install → /watch → clone → run) confirms the architecture in one line.
• t=02:31–02:33 — the demo. Frames f0007/f0008: pasting a long course / a Karpathy interview URL and running /watch while doing other work.
• t=03:30 — where it lands. Frame f0010: an Obsidian "Second brain" graph view — folders raw/watched, wiki/{concepts,content,entities,sources} and a dense node graph — the new note connected, not filed. Structurally the same shape as firehose's own vault (captures/, concepts/, distillates/).
• t=03:56 — outro. Frame f0011: end card; promises a follow-up on building the second brain ("took me a month to build correctly").

Linked from

• AI Second Brain
• Every Level of a Claude Second Brain Explained
• This Week
• Multimodal Video Ingestion
• Scene-Change Frame Sampling