When should you use templates vs custom memory pipelines?

Groups and topics give you configuration knobs. Pipelines are the machinery those knobs steer. Weaviate Engram processes stores through an asynchronous pipeline—a DAG of extract, transform, buffer, and commit steps—that turns raw string, conversation, or pre-extracted input into durable memories. You do not have to design that DAG on day one. Project templates seed common use cases with ready topic sets and working pipelines. When the default shape stops matching the work, you deepen control: first by editing topic descriptions, then—where your plan allows—by configuring the pipeline itself.

This chapter contrasts starting from templates with moving to fully custom pipelines, explains what the pipeline steps and runs actually do, shows a practical escalation path that avoids premature custom DAGs, and walks a stained-glass came-bending desk from a personalization template into a buffered daily summary path. After pipelines comes the broader build-versus-managed decision for memory as a service.

What do templates buy you before you touch the DAG?

Templates exist so teams can ship memory without becoming pipeline engineers on week one. Engram’s personalization-style templates typically provision a default group with starter topics such as UserKnowledge, wired to a pipeline that already extracts from conversations, transforms against existing context, and commits scoped memories. Continual-learning and multi-agent-oriented templates follow the same idea for their use cases: sensible topics, sensible processing, hybrid search on the way out.

That is enough for many products. You send conversations with user_id, wait for runs to complete, and search. The highest-leverage customization is still the topic layer: rewrite descriptions so extraction magnets pull the facts your domain cares about. Bounded topics give you single-profile or single-summary cardinality without redesigning commit logic. Groups let you split personalization from shared technique when one seeded bundle is no longer honest.

Templates are not a trap if you treat them as a starting contract. They become a trap when the product’s real need is a different processing shape—batching a day’s memories into one summary, delaying commit behind a buffer, or routing pre-extracted facts differently from free text—and you keep stuffing that need into topic descriptions alone.

What does a custom pipeline change that topics cannot?

A pipeline is a directed acyclic graph with content-type entrypoints. ExtractFromString, ExtractFromConversation, and ExtractFromPreExtracted feed transform steps such as TransformWithContext, TransformOperations, TransformConcatenate, and TransformAggregate. Those steps deduplicate, merge, resolve conflicts, and honor bounded topics. Buffer steps pause until a count or time trigger fires, aggregating work across inputs that share a scope. Commit finalizes create, update, and delete operations into storage.

Custom pipelines rearrange that graph. A daily-summary design might extract, transform, and commit immediately, then buffer committed memories for twenty-four hours, then transform again into one aggregated activity memory and commit a second time. Topics still decide what categories exist. The pipeline decides when consolidation happens, how buffers batch work, and whether different content types share downstream steps. Configurable pipelines are available on Engram enterprise plans; until you need that control, templates plus topic and group configuration remain the productive path.

Every memories.add creates a run with states such as running, in_buffer, completed, or failed. Custom pipelines make run semantics more visible because buffers and multi-stage commits are intentional. Observing committed_operations after completion is how you verify the DAG did what you designed—not what you hoped the LLM inferred from a topic blurb.

How should you escalate from template to custom without thrashing?

Escalate in order. First, fix topic descriptions and bounds when the wrong facts are stored or duplicates compete. Second, split groups when use cases diverge. Third, use pre-extracted input when your application already has structured facts and you want transform/commit without another free-text extraction pass. Fourth, customize the pipeline when timing, buffering, or multi-stage aggregation are first-class product requirements.

That order protects velocity. Many “we need a custom pipeline” tickets are actually “our UserKnowledge description is vague” tickets. Conversely, forcing a twenty-four-hour rollup into a single unbounded topic with client-side cron jobs reimplements buffers poorly. Match the lever to the failure. If extraction content is wrong, change topics. If extraction timing and aggregation are wrong, change the pipeline. Skipping that diagnosis is how teams burn a quarter rebuilding a DAG that still extracts the wrong facts.

Keep the application contract stable across the escalation. Still pass scopes. Still choose groups deliberately. Still search with hybrid retrieval and optional topic filters. Template versus custom is about processing configuration inside Engram, not about abandoning Engram for a one-off memory service you maintain alone—until a later chapter’s build-versus-managed tradeoffs say otherwise.

What does escalation look like at a stained-glass came desk?

A studio assistant starts on a personalization template for artists at a came-bending desk, then adds a custom buffered rollup for daily shop notes. Scenario id: stained-glass-came-bending-desk-9.

from weaviate.engram import EngramClient
from weaviate.engram.retrieval import HybridRetrieval

engram = EngramClient()
user_id = "glazier-iona"
scenario = "stained-glass-came-bending-desk-9"

# Phase 1 — template path: default group, seeded topics, stock pipeline
turn = [
    {"role": "user", "content": "I prefer wider lead came on outdoor panels and slower heat on tight curves."},
]
run = engram.memories.add(
    content=turn,
    group="default",
    scopes={"user_id": user_id, "properties": {"scenario": scenario}},
)
# poll run until status == completed; inspect committed_operations

prefs = engram.memories.search(
    query="lead came width heat preference curves",
    group="default",
    topics=["UserKnowledge"],  # or refined custom topic names after you edit config
    retrieval=HybridRetrieval(alpha=0.5, limit=5),
    scopes={"user_id": user_id},
)

# Phase 2 — custom pipeline concept (enterprise configuration on the group):
# [extract] -> [transform] -> [commit] -> [buffer 24h by shop scope] -> [transform aggregate] -> [commit]
# shop_daily = engram.memories.add(
#     content="Tight curves on cobalt panels needed lower heat today.",
#     group="shop_daily_rollup",
#     scopes={"properties": {"shop_id": "north-glaze", "scenario": scenario}},
# )
# while shop_daily.status == "in_buffer": wait for buffer trigger

Phase one ships Iona’s preferences through the template pipeline with topic edits as needed. Phase two only appears when the shop wants an automatic daily technique rollup—exactly the shape Engram buffers and second-stage transforms are for. Until that requirement is real, custom DAG work would be ceremony. Ceremony feels like progress in architecture reviews and stalls shipping for the artists who only needed clearer topic magnets. When it is real, you configure the pipeline on a dedicated group instead of inventing an external batch job that fights Engram’s run model.

Start with Engram templates and topic/group configuration; move to custom pipeline DAGs when buffering, multi-stage aggregation, or content-type routing become product requirements—not when a description tweak would have fixed extraction. Our next chapter, When should you build vs use a managed memory service?, widens the lens from pipeline shape to whether memory infrastructure itself should be owned in-house.