What is hierarchical chunking?

The previous chapter established the basic tension in chunking: too large and a chunk dilutes its own meaning, too small and it loses the context needed to make sense on its own. Hierarchical chunking is one specific way of resolving that tension for genuinely long, structured documents, not by picking one single chunk size, but by deliberately keeping multiple sizes at once, each serving a different purpose.

What Does It Actually Mean to Chunk a Document at Several Levels Simultaneously?

Hierarchical chunking breaks a long document into layers rather than one flat set of equally-sized pieces. A top layer might capture broad sections or overall themes, a middle layer might capture individual paragraphs within those sections, and a bottom layer might capture individual sentences or fine-grained details within those paragraphs. Each layer exists at once, covering the same underlying content at a different level of granularity, rather than one layer replacing another. A search can then work with whichever layer actually suits the question being asked, broad and thematic, or narrow and specific.

Why Would Keeping Several Layers at Once Actually Beat Picking One Single, Fixed Chunk Size?

A single fixed chunk size forces every query into the same tradeoff regardless of what that specific query actually needs. A broad, thematic question gets served reasonably well by a large chunk but poorly by a tiny one, since the tiny one lacks the broader context the question is really asking about. A narrow, specific question gets served well by a small, precise chunk but poorly by a large one, since the large one dilutes the one specific detail the question actually needs among everything else bundled into that same oversized chunk. Keeping multiple layers available at once means a system doesn’t have to guess in advance which granularity a future query will need, it can serve either kind of question well by reaching for whichever layer actually fits.

How Does a Search Actually Decide Which Layer to Use for a Given Query?

A common pattern, sometimes called small-to-big retrieval, searches against the smallest, most precise layer first, since fine-grained chunks tend to produce the most accurate matches for a specific query. Once that precise match is found, the system then pulls in the broader, parent-level chunk that small piece belongs to, supplying the model with enough surrounding context to actually make sense of the specific detail it matched on. This gets the precision benefit of searching small, tightly-focused chunks while still avoiding the context-loss problem that searching only at that narrow level would otherwise create.

Does This Pattern Apply Only to Static Documents, or Does It Show Up in Memory as Well?

The same underlying idea shows up throughout memory as this knowledge base has already covered it, just under different names. An atomic fact behaves like the small, precise, bottom-layer chunk, retrievable with sharp precision for a narrow question. A running summary or a broader profile behaves like the higher-level, parent chunk, supplying the surrounding context a narrow fact alone wouldn’t convey. Neither shape replaces the other, exactly the same non-competing relationship already established when this knowledge base covered atomic facts and summaries earlier. Hierarchical chunking is really the same principle, generalized to documents with more than just two levels of structure.

What Actually Determines How Many Layers a Given Document Genuinely Needs?

Not every document benefits from many layers. A short, simple document with little internal structure gains little from being artificially split into several hierarchical levels, since there’s no meaningful distinction between a broad theme and a narrow detail in something that short to begin with. A genuinely long, structured document, with real sections, subsections, and fine-grained detail nested within them, benefits considerably more, since it actually has that structure to preserve in the first place. Adding hierarchy where a document’s actual structure doesn’t call for it adds complexity without adding a corresponding benefit.

How Does Weaviate Engram’s Combination of Bounded Summaries and Atomic Facts Reflect This Same Hierarchical Principle?

Weaviate Engram already implements this small-to-big pattern through its combination of atomic, unbounded facts and bounded, summary-level topics, letting a search reach for narrow precision or broader context depending on what a specific question actually needs. Consider a manufacturing plant’s equipment-maintenance assistant helping technicians work from a long, structured equipment manual, where a technician might need either one exact spec or the broader procedure that spec belongs to:

from engram import EngramClient

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])

client.memories.add(
    "Torque specification for the drive-shaft coupling bolts on this press model is 85 Nm, part of the quarterly preventive-maintenance procedure covering the full drive assembly.",
    properties={"equipment_id": "press-line-3"},
    topics=["MaintenanceSpecs"],
)

A technician needing the exact spec retrieves the precise, narrow fact directly:

spec_result = client.memories.search(
    query="What's the torque spec for the drive-shaft coupling bolts?",
    properties={"equipment_id": "press-line-3"},
    topics=["MaintenanceSpecs"],
    retrieval_config=HybridRetrieval(limit=3),
)

procedure_context = client.memories.search(
    query="quarterly preventive maintenance procedure",
    properties={"equipment_id": "press-line-3"},
    topics=["MaintenanceProcedures"],
    retrieval_config=FetchRetrieval(limit=1),
)

The first search finds the exact torque figure with sharp precision, exactly what a technician working with a wrench actually needs in the moment. The second search, fetching the broader, bounded procedure context this specific spec belongs to, supplies the surrounding detail a technician might need if they’re not just tightening one bolt but running the full quarterly procedure and need to understand where this specific step fits within it. Neither retrieval alone would serve both needs equally well, which is exactly the same small-to-big principle hierarchical chunking applies to long documents, just expressed through Engram’s own topic structure instead of document-level layers.

Hierarchical chunking preserves structure by deliberately keeping several fixed levels of granularity available at once. A related but different technique achieves a similar goal without committing to any fixed set of levels ahead of time, by embedding an entire document first and only deciding how to divide it afterward. Our next chapter, What is late chunking?, takes up exactly that alternative approach.