Short answer: It embeds the full document first, then splits, so each chunk vector still carries surrounding context.
Normal pipelines chunk first and embed each piece alone, which breaks pronouns and cross-references like as mentioned above. Late chunking reverses that order: whole-document token representations form with awareness of neighbors, then boundaries are chosen and pooled into chunk vectors. It is not the same as late interaction, which keeps per-token vectors at search time. For memory, it helps when early and later mentions of the same fact would otherwise become unrelated fragments.
Hierarchical chunking solved the granularity tradeoff by deliberately keeping several fixed levels of chunk size available at once. Late chunking approaches the same underlying problem from a genuinely different angle, changing not how many chunk sizes exist, but the actual order in which chunking and embedding happen in the first place.
What Order Do Chunking and Embedding Normally Happen In, and Why Does That Order Matter?
Every chunking approach covered so far in this Part follows the same basic sequence: split a document into pieces first, then embed each piece independently, one at a time. This ordering means each chunk gets embedded in isolation, with no awareness of what came before or after it in the original document. A chunk referencing something established several paragraphs earlier, “this approach,” “the same issue,” “as mentioned above,” loses the connection to whatever it was actually referring to, because the chunk containing that earlier reference is nowhere in view when this later chunk gets its own, separate embedding.
What Does Late Chunking Actually Reverse About This Process?
Late chunking flips the sequence: embed the entire document first, using a model capable of processing long spans of text at once and producing a contextualized representation for every individual token across the whole thing, and only afterward decide where the chunk boundaries should actually fall. Because every token’s representation was computed while the model could see the entire surrounding document, a token appearing near the end of a passage still carries some trace of context from everything that came before it. Only once this full, context-aware set of token representations exists does the actual splitting happen, pooling together the relevant token representations for each intended chunk to produce that chunk’s final embedding.
Why Does This Reordering Actually Produce Better Chunk Embeddings Than Chunking First?
Because each individual chunk’s embedding is now derived from token representations that were computed with the whole document in view, that chunk’s final embedding retains a trace of the surrounding context, even though the chunk itself, as a standalone piece of text, might look exactly the same as it would have looked under ordinary, chunk-first processing. This means a query about a concept discussed across two genuinely separate but related passages has a better chance of finding both of them, since each one’s embedding carries some awareness of the other, rather than the two passages being embedded as if they existed in complete isolation from each other.
Does This Approach Require Fundamentally Different Infrastructure to Actually Use?
Late chunking requires a long-context embedding model capable of processing an entire document at once and capable of producing token-level representations rather than only a single pooled output, but the actual chunking logic applied afterward can be as simple as ordinary, fixed-size splitting. Once the context-aware token representations exist, deciding where chunk boundaries fall is a comparatively minor decision, since the more important context-preserving work already happened during that initial, whole-document embedding pass, not during the splitting step itself. The resulting chunk embeddings can then be stored and searched exactly the same way ordinary chunk embeddings would be, without requiring any change to how retrieval itself works downstream.
Is Late Chunking the Same Idea as Late Interaction, Despite the Similar Name?
The two are related in spirit but structurally distinct. Late interaction, covered earlier in this Part, keeps a separate vector per token permanently, comparing every query token against every document token individually at search time. Late chunking still pools its context-aware token representations down into one vector per chunk, the same single-vector-per-piece structure as ordinary chunking, it just makes sure that pooling happens after the model has already seen the whole surrounding document, rather than embedding each isolated chunk on its own from the start. The two techniques share the underlying goal of preserving more context than naive, isolated chunking would, but they achieve it through genuinely different mechanisms.
Where Does This Kind of Cross-Reference Preservation Matter Most for Memory Specifically?
Content where an idea introduced early genuinely gets revisited, built on, or referenced later benefits most from this kind of context-preserving approach, since exactly this pattern is what naive, chunk-first embedding handles poorly. Consider a media company’s podcast transcript search assistant helping producers find moments where a guest’s earlier point gets picked back up later in a long, hours-long conversation:
from engram import EngramClient
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
client.memories.add(
"Early in the interview, the guest mentioned struggling with imposter syndrome after their first major promotion. Much later, when discussing mentorship, they circled back to that exact experience, explaining it's precisely why they now make a point of checking in with newly promoted team members.",
properties={"episode_id": "episode-4471-leadership"},
)
A producer searching for this specific connection benefits from an embedding process aware that these two moments are actually linked:
results = client.memories.search(
query="Did the guest ever connect their own imposter syndrome experience to how they mentor others?",
properties={"episode_id": "episode-4471-leadership"},
)
Because Engram’s extraction step captured this connection explicitly as one coherent memory rather than treating the early mention and the later callback as two isolated, disconnected facts, this search correctly surfaces the link between them as a single, coherent finding. This is exactly the kind of cross-referential connection that naive, isolated chunking of a raw, hours-long transcript would risk losing entirely, splitting the early mention and the later callback into two separate chunks with no embedded awareness that they were ever related, precisely the failure mode late chunking, and Engram’s own context-aware extraction, are both built to avoid.
Late chunking preserves cross-references within a single piece of content by changing when pooling happens relative to embedding. A related but different concern is how a retrieval system balances finding everything genuinely relevant against avoiding everything that isn’t, a tension that shows up regardless of which specific chunking or embedding technique a system happens to use. Our next chapter, What is recall vs precision in memory retrieval?, takes up exactly that tension.