Embedding Deduplication

When a document is indexed, each chunk is converted to a vector embedding via an external API (e.g., OpenAI). Embedding API calls are the most expensive part of indexing -- both in cost and latency. Brain uses chunk-level deduplication to skip embedding calls for chunks that have already been embedded with the same model.

Why chunk-level?

A naive approach hashes the entire document text and skips re-embedding if the hash matches. The problem: a one-character edit in a 50-page PDF invalidates the entire cache and forces re-embedding of every chunk.

Chunk-level dedup hashes each chunk individually. When a document is re-indexed after a small edit, only the chunks whose text actually changed need to be sent to the embedding API. Unchanged chunks reuse their existing embeddings.

How it works

The indexing pipeline follows this flow for each document:

Extract text
    |
Split into chunks
    |
Hash each chunk (normalized text + embedding model)
    |
Batch-lookup hashes in chunk_hash table
    |
    +-- Cache hit: fetch embedding from source chunk
    |
    +-- Cache miss: call embedding API
    |
Insert all chunks with embeddings
    |
Register new hashes in chunk_hash table

Hash computation

Each chunk is hashed using BLAKE3 over the concatenation of:

normalize(chunk_text) + \0 + embedding_model

The null byte separator prevents ambiguous collisions between chunk text and model name.

Normalization (applied only for hash computation, not for the embedding input):

• Collapse all whitespace runs (\s+) to a single space
• Trim leading and trailing whitespace
• Lowercase

This means trivial formatting changes (extra spaces, different line endings, case changes) produce the same hash and reuse the cached embedding. The original chunk text is always sent to the embedding API unchanged.

The embedding model is part of the hash key, so the same chunk text embedded with different models produces different hashes. This is correct -- embeddings from different models are not interchangeable.

Cache lookup

Hashes are looked up in the chunk_hash table, which maps each hash to a source pool and chunk ID. When a hit is found, the embedding is fetched directly from the source chunk's database row rather than calling the embedding API.

Cache entries are shared across projects. If two projects index the same document with the same embedding model, the second project reuses embeddings from the first.

Partial cache hits

The system handles partial matches naturally. If a 100-chunk document is re-indexed after editing one paragraph:

• ~98 chunks match the cache (same normalized text + model)
• ~2 chunks are cache misses (the edited paragraph and possibly its neighbor due to overlap)
• Only 2 embedding API calls are made instead of 100

The force flag

When a document is re-indexed with force: true, the chunk hash cache is bypassed entirely. All chunks are sent to the embedding API regardless of whether they've been embedded before. The new embeddings still get registered in the cache for future use.

This is useful when you suspect cached embeddings may be stale or want to ensure a clean re-index.

Chunking settings and the cache

Chunk size and chunk overlap are not part of the hash key. Two chunks with identical normalized text and embedding model will share an embedding regardless of the chunking settings that produced them.

However, the chunking settings are recorded as metadata alongside each cache entry. This allows stale entries to be identified if a project changes its chunking configuration.

Extraction cache

Separate from embedding deduplication, Brain also caches extracted text from uploaded files. If the same binary file (identified by its raw file hash) is uploaded to multiple projects, the text extraction step (PDF parsing, HTML conversion, etc.) is performed only once. This is a document-level cache and is unrelated to the chunk-level embedding cache described above.