video youtu.be Watch ↗

How AI Agents Search Their Memory — Hybrid Retrieval, in Practice (OpenClaw)

TL;DR

Storing memory is only half the problem; the other half is finding the right entries at the right time without drowning the context window. This talk walks the full read path and lands on one thesis: keyword and semantic search each have the other's blind spot, so the practical default is Hybrid Retrieval — run BM25 keyword search and vector search in parallel and merge them with Rank Fusion — optionally refined by a model-scored Reranking pass over the shortlist. Semantic search alone breaks on exact identifiers (error codes, useState), which is why the keyword channel earns its place; tellingly, the Claude Code team dropped a vector DB for plain grep because it was better and easier to maintain (a concrete "no pain, no climb" for Agentic Simplicity). The second load-bearing idea is a context-discipline one: Search-Then-Get splits retrieval into a lean search (path, line numbers, score, ~700-char preview, citation) and a targeted get (fetch an exact line range), so only confirmed-relevant content reaches the window. The OpenClaw implementation grounds all of it — single SQLite DB with FTS5 + sqlite-vec, weighted fusion at 0.7 vector / 0.3 keyword, a 4× candidate over-fetch, a 0.35 score floor, and Incremental Indexing that hashes files and chunks to avoid re-embedding anything unchanged.

Concepts introduced

Hybrid Retrieval — keyword + semantic run in parallel and fused, because semantic search is unreliable on exact identifiers and keyword search is blind to paraphrase.
Rank Fusion — merging two ranked lists: weighted score fusion (preserves match strength) vs reciprocal rank fusion (position-only); plus candidate over-fetch and a score floor.
Reranking — a model-scored relevance pass over the fused shortlist; cheap fast recall first, expensive accurate model last (why not just use an LLM for the whole search).
Search-Then-Get — the two-tool split (lean search → targeted get) that keeps the context window lean; the attention-budget discipline expressed as a tool contract.
Incremental Indexing — hash files and chunks to skip re-embedding anything unchanged; full reindex only on a config change, done via temp-DB-and-atomic-swap.

Held, not dropped

Themes the capture touches that don't warrant their own concept page yet (spin out on demand):

BM25 internals — the term-frequency × inverse-document-frequency intuition. Currently a claim under Hybrid Retrieval; promote if a distillate goes deep on keyword ranking.
Embedding-provider auto-selection & graceful degradation — the local→OpenAI→Gemini→Voyage fallback chain, and disabling memory search when no key exists. A provider-portability pattern, not yet a concept.
QMD (Toby Lütke's alternative memory backend) — named as OpenClaw's optional non-default backend; a pointer, not developed here.
Single-file SQLite as the retrieval substrate — FTS5 + sqlite-vec + cache co-located in one DB (the same SQLite+FTS5 pattern firehose reuses from richard-chatgpt). Held as an architecture note.
Approximate nearest neighbor (ANN) — the accuracy-for-speed index at scale. Folded as an observation into Semantic Retrieval rather than spun out.

Key claims

Semantic search is unreliable on literal strings and exact identifiers (error codes, function names). principle → Hybrid Retrieval, Semantic Retrieval — a structural limit of retrieval-by-meaning, and the whole reason the keyword channel is kept.
Keyword and semantic search have complementary blind spots — run both and fuse. (best practice) → Hybrid Retrieval — context: memory whose queries mix natural language with exact identifiers; unnecessary if queries are reliably one shape.
The Claude Code team started with a vector DB, then dropped it for grep + agentic search — better and easier to maintain. observation → Agentic Simplicity, Retrieval Maturity Levels — an independent, concrete "no pain, no climb": the fancier retrieval level lost.
Weighted fusion preserves match strength; RRF uses only rank position. observation → Rank Fusion — the tradeoff that picks between the two combine methods.
Over-fetch candidates (4×) before fusing, then apply a minimum score threshold. (best practice) → Rank Fusion — context: hybrid pipelines; richer pool for fusion, at more compute.
Rerank only a refined shortlist, never the whole corpus — cheap recall first, expensive precision last. principle → Reranking — a model can't scan thousands of candidates cheaply; the two-stage split is what makes model-quality relevance affordable.
A retrieval tool should return just enough to judge relevance, not the full content. principle → Search-Then-Get — progressive disclosure keeps the context window (the real bottleneck) lean.
Split memory access into a preview-returning search and a line-range get; make recall mandatory in the prompt. (best practice) → Search-Then-Get — context: agent memory where loading whole files would blow the window.
Hash to skip work: unchanged files (content hash) and unchanged chunks (embedding cache) never hit the embedding API. (best practice) → Incremental Indexing — context: embedding pipelines where API calls are the cost; safe because an embedding is a pure function of (text, provider, model).
Full re-index only on a config change that invalidates embeddings (provider/model/chunk size), via temp DB + atomic swap. (best practice) → Incremental Indexing — the one case a rebuild is actually required, done without a half-built index going live.
Chunks carry their source line range, enabling exact citation and targeted fetch. observation → Search-Then-Get, Semantic Retrieval — the metadata that makes get precise.

Why this builds on the retrieval graph (and corroborates the simplicity thesis)

This is the read-path companion to the existing second-brain material, which stopped at Semantic Retrieval as "level 3" of Retrieval Maturity Levels. It builds on that spine rather than duplicating it: it sharpens level 3 by naming semantic search's exact-identifier failure mode and introducing the practical siblings that patch it — Hybrid Retrieval, Rank Fusion, Reranking — plus two production concerns the earlier material never reached, Incremental Indexing (keeping the index fresh cheaply) and Search-Then-Get (spending context only on confirmed-relevant content). The last one is the most firehose-aligned idea in the talk: it's the same "human attention / context window is the bottleneck" discipline, enforced by the shape of the tools rather than by prompt exhortation.

Two secondary stances worth flagging. First, a strong corroboration: the report that the Claude Code team abandoned a vector database for grep + agentic search is an independent, concrete instance of Agentic Simplicity's "complexity climbs, capability doesn't" — a fourth source now converges on it, and from the retrieval angle specifically. Second, a mild internal tension the video never resolves: its headline thesis is "hybrid is the answer," yet its most memorable data point is a team removing the semantic half entirely and being better off. The reconciliation is Query-Shaped Storage — for a code-shaped corpus queried by exact tokens, keyword alone fits; for a mixed-query memory store, you want both. The channel count is a query-shape decision, not a universal upgrade.

Illustrated walkthrough

Keyed by t=MM:SS, fusing each kept frame with the transcript segment it fell in.

t=00:00 — the setup (talking head). The presenter (Damian Galarza) frames it: his prior video covered storing memories, "but this is only half the problem… these files aren't useful if you don't have an effective way to load them into context at the right times." Four pillars announced: keyword vs semantic, why hybrid is the answer, reranking, and a real OpenClaw example.
t=00:57 — keyword search, illustrated. A slide of five markdown memory files (meeting-notes.md, api-requirements.md, session-2024-03.md, architecture.md, planning.md) with literal token matches highlighted (limit / limits / UPLOAD / File / upload). This is exactly the BM25/Semantic Retrieval-contrast the narration is making: keyword search is literal word matches, ranked by term frequency × rarity — "what powers full text search in databases like SQLite's FTS5." The Claude-Code-dropped-vectors-for-grep aside lands just before this.
t=01:37 / t=02:50 — embeddings & nearest-neighbor (NOT sampled). The narration explains embeddings ("How do I speed up my app?" ≈ "Tips for improving application performance" produce similar vectors; "Best restaurants near me" is far), then nearest-neighbor search as "plotting coordinates on a map." The sampler returned blank transition frames here — the actual vector and map diagrams were never captured. Treat this segment as audio-only in the notes.
t=04:12 — the hybrid split. A two-column card, KEYWORD SEARCH (BM25) | SEMANTIC SEARCH (VECTOR) — the visual statement of Hybrid Retrieval. Narration: run both in parallel, then "use a technique called fusion to combine the results," and the two methods (weighted score fusion vs RRF) follow. OpenClaw uses weighted, ~70% vector / 30% keyword.
t=05:23 — reranking (title-card only). The fused list is "based on purely math… it doesn't have the ability to understand nuance." Reranking passes the shortlist to a model (an LLM or a specialized model like Cohere) for nuanced relevance — and the key argument for why not start with the model: speed and context-window limits (scan thousands of embeddings fast, then rerank only the refined set).
t=06:20 → t=07:16 — the ~56 s blind gap. OpenClaw intro: two memory backends (default = SQLite + vector embeddings; optional = QMD, an open-source library by Toby Lütke), then the start of the four embedding-provider options. No usable frames in this window.
t=07:16 — embedding providers (blank frame). Audio: four providers — local via Ollama, OpenAI text-embedding-3-small, Google gemini-embedding-001, Voyage voyage-4-large — with an auto mode that prefers a local model, then falls back OpenAI → Gemini → Voyage, and disables memory search entirely if none are configured (graceful degradation). Slide not sampled.
t=07:57 — "Virtual Search Tables." A title card for the schema section. The single SQLite DB holds a files table (path, content hash, mtime → enables Incremental Indexing); a chunks table (~400-token chunks, 80-token overlap, each storing text, the embedding vector, and its source line range); two virtual tables — chunks_fts (FTS5 → BM25 keyword) and chunks_vec (sqlite-vec → float32 cosine); and an embedding cache. The table diagram itself was not sampled — only its heading.
t=~09:30 — the search pipeline (audio). Query embedded with the same provider that indexed; two searches run in parallel; BM25 ranks flattened via 1/(1+rank), cosine distance flipped via 1 − distance; both over-fetch a 4× candidate multiplier; weighted fusion 0.7·vector + 0.3·text; results present in both channels get both scores, one-channel results take a zero for the other; finally sort, apply a 0.35 minimum threshold, cap to requested count. (See Rank Fusion.)
t=~10:54 — the two memory tools (audio). memory_search is a mandatory recall step (prompt tells the agent to search before answering about prior work, decisions, dates, people, preferences, to-dos); returns snippets with path, line numbers, score, ~700-char preview, and citation. memory_get is the follow-up: fetch a specific line range. This is Search-Then-Get — "keeps the context window lean and efficient."
t=11:12 — "Incremental Sync Flow." Title card for the sync system. File watcher (debounced ~1.5 s, dirty flag) → compare each file's content hash against the files table → skip unchanged files → re-chunk changed files → hash each chunk against the embedding cache → reuse on hit, else embed (concurrency-limited). Full re-index only on a provider/model/chunk-size change, built into a temp DB and atomically swapped; session transcripts sync on byte/message deltas. (Incremental Indexing.)
t=13:06 — wrap-up. "From storing memories to searching them to keeping them in sync… a masterclass in practical AI architecture," then like/subscribe and a pointer to running OpenClaw and building with Claude Code.