Incremental Indexing
Keeping a search index fresh without re-embedding everything
on every change. The engine of it is hashing to skip
work at two granularities. A file watcher (debounced ~1.5 s,
dirty-flag) triggers a sync; the sync lists memory files and compares
each file's content hash against a files
table — if the hash matches, that file is skipped entirely, so only
genuinely-changed files get re-chunked. When a file does change, it's
split into chunks (OpenClaw: ~400 tokens with 80-token overlap) and
each chunk's text is hashed against an embedding cache
— a cache hit reuses the stored vector and skips the embedding API call.
Embedding calls cost money and add up, so the cache is a direct cost
optimization. Re-embedding runs under a concurrency
limit so it doesn't overwhelm the provider.
A full re-index is reserved for changes that invalidate existing embeddings — a different embedding provider, model, or chunk-size config, detected as a mismatch in a metadata table. Even then it's done safely: build the new index into a temporary database, then atomically swap it into place, so a half-built index is never live. Session transcripts get a lighter delta-tracking trigger (sync once accumulated bytes/messages cross a threshold). Net effect: memory stays searchable without an expensive full rebuild on every edit — the same "spend work only where the input actually changed" discipline that content-addressed caches everywhere rely on.
Claims
- Hash to skip: unchanged files (content hash) and unchanged chunks (embedding cache) never hit the embedding API. best practice — context: embedding pipelines where API calls are the cost; "best" because an embedding is a pure function of (text, provider, model), so identical text can safely reuse a cached vector.
- Full re-index only on a config change that invalidates existing embeddings (provider, model, or chunk size). best practice — context: an index whose vectors are only comparable within one embedding config; changing the config makes old vectors incompatible, which is the one case a rebuild is actually required.
- Rebuild into a temp database and swap atomically so a half-built index is never live. best practice — context: any full reindex; "best" when queries must keep working during the rebuild and a partial index would return wrong results.
- Debounce the file watcher and use a dirty flag so bursts of edits collapse into one sync. observation — factual: avoids redundant syncs while a file is being actively edited.
Related
- Semantic Retrieval — incremental indexing is how the vector store it queries stays current.
- Search-Then-Get — the chunks
(with line ranges) this produces are what
getfetches. - Query-Shaped Storage — chunk size / overlap are storage-shape choices set by how you'll query.
- Distillate: How AI Agents Search Their Memory — Hybrid Retrieval, in Practice (OpenClaw)