Comprehensive Guide to RAG Vector Chunking
Retrieval-augmented generation, or RAG, grounds a language model's answer in documents the model did not train on. The pipeline has three stages: split documents into chunks, embed each chunk into a vector, and on every query retrieve the chunks closest to the question. Of the three, chunking is the most consequential and the most underestimated. Get it wrong and the retriever can never find the answer, no matter how good the embedding model is.
Fixed-size chunking is the baseline everyone starts with. The text is walked left to right and cut at every N characters, ideally snapping to a word boundary so the cut lands at a space. Its virtues are speed and predictability — the chunk count is deterministic and the storage cost is trivial to forecast. Its weakness is semantic blindness: a fixed cut can sever a sentence, a definition, or a numbered list mid-item, leaving neither half sufficient on its own.
The overlapping sliding window fixes the boundary problem at a price. The window advances by chunk size minus overlap characters, so the tail of one chunk reappears at the head of the next. If a question depends on words that straddle a cut point, the overlap makes sure at least one chunk contains the full phrase. The price is redundancy: a 25% overlap means roughly 33% more characters are stored and embedded for the same source text.
Semantic chunking respects the document's natural structure instead of arbitrary offsets. Paragraphs become chunks, and oversized paragraphs are split on sentence boundaries so meaning stays intact. Semantic chunks are denser and more coherent, which usually lifts retrieval precision, but they cost more to construct and can be unpredictable in size when paragraphs vary wildly.
Chunk size is the first dial most people turn, and it interacts with the embedding model. Many embedding models are trained on text snippets of a specific length, and chunk sizes far from that training distribution degrade retrieval. A common starting point is 200 to 500 characters, tuned against the model's documented context window and the length of the questions users actually ask.
Token accounting keeps the economics honest. The rule of thumb is four characters per token, so a 200-character chunk is roughly 50 tokens, and a source of 2,400 characters is roughly 600 tokens. The simulator reports total chunks, per-chunk tokens, and average size so you can compare strategies on storage and embedding cost before you ever pay an embedding bill.
Retrieval quality is the true test. A chunk that is too small may miss context the answer needs; too large and it dilutes the relevant passage among noise, which confuses semantic similarity scoring. The ideal chunk is the smallest unit that is self-contained — a paragraph that states a complete idea, with any references it depends on either inside it or consistently nearby.
Metadata is the silent amplifier of good chunking. Attaching source, heading, page, and section to each chunk lets a retriever filter by document and lets the generator cite its sources. Chunking is where that metadata is assigned, so plan the fields while you still control the split, not after the vectors are written.
The practical workflow is iterative: simulate a strategy, inspect the boundary highlights, read a few retrieved chunks against real questions, and adjust. The color-coded preview in the simulator exists precisely for this — you see the cuts and the repeated overlap text before you commit them to an index. Most teams converge on a hybrid: semantic chunking for structured prose, with a character cap and a small overlap for safety.
Chunking is a cost lever as much as a quality lever. Re-embedding a corpus because you chose poorly is expensive and slow, so prototyping with a simulator before re-embedding saves both. Start with your real document, run all three strategies, and let the chunk counts and token estimates — not intuition — pick the default.