Applied AI·Embeddings and retrieval
you cut at four hundred tokens and lost the sentence that straddled the boundary, so you make adjacent chunks overlap and try again.
Chunk size and overlap
Draft summary, pending review
The two basic knobs: how many tokens per chunk (commonly a few hundred to around a thousand) and how much adjacent chunks overlap so sentences straddling a boundary survive in one piece. There is no universal best; it is an empirical, per-corpus decision.