Short answer: Assemble extract, transform, buffer, and commit around each distinct purpose, often as separate configurations.
Personal recall and shared task knowledge need different topics and search scopes. Start simple, then split configurations when purposes diverge so searches and topic names do not collide. Separate configs can still share the same step sequence while differing in topics. Engram supports structuring distinct use cases cleanly instead of one undifferentiated pipeline.
Every stage covered throughout this Part, extraction, transformation, buffering, commit, exists as a building block a specific pipeline can use, skip, or arrange differently. None of these blocks tells a team how to actually assemble them for their own particular application. This chapter looks at that design question directly: how should a team actually decide what shape its own pipeline should take?
Why Doesn’t One Single Pipeline Design Serve Every Use Case Equally Well?
Different applications need genuinely different things from memory. One application mainly needs to recall an individual person’s preferences across separate sessions. Another mainly needs to accumulate general, shared knowledge about how to perform a task well, regardless of which specific person happened to be involved in any given interaction. A third might need both at once, personal recall for one purpose and shared, accumulated experience for another, kept carefully separate from each other. A single, universal pipeline design can’t serve all of these well simultaneously, which is exactly why pipeline configuration exists as a genuine design decision rather than a fixed, one-size-fits-all default.
What’s the Simplest Starting Point for a Team That Doesn’t Yet Know Exactly What Its Memory Needs Will Look Like?
Starting from a prebuilt template covering a common, well-understood pattern, personalization or continual learning, gives a team a working pipeline immediately, with sensible topic definitions already in place, while still leaving room to adjust the specific natural-language descriptions that control what actually gets extracted. This approach defers the harder architectural decisions until a team has actual, real usage to learn from, rather than forcing those decisions upfront before anyone genuinely knows what the application’s memory needs will actually look like in practice.
How Should a Team Decide Whether It Genuinely Needs More Than One Separate Configuration Rather Than Just One?
The right signal is whether an application has genuinely distinct use cases that call for different topic definitions or different processing logic entirely, rather than simply different pieces of content flowing through the exact same underlying purpose. A single chatbot recalling one person’s preferences across sessions is one coherent use case, even if it involves several different kinds of preference. An application that both recalls personal user preferences and separately accumulates shared, project-wide operational knowledge is actually two genuinely distinct use cases bundled into one application, and forcing both through a single, undifferentiated configuration risks tangling two purposes that would be far clearer, and far easier to reason about, kept apart.
Why Does Keeping Genuinely Distinct Use Cases in Separate Configurations Actually Matter in Practice?
Separate configurations let each use case define its own topics without worrying about naming collisions or unintended overlap with an entirely different purpose’s own topics, and they let each one be searched independently, retrieving only what’s actually relevant to whichever specific purpose a caller currently cares about. Bundling two genuinely distinct purposes into one shared configuration risks searches surfacing irrelevant results from the wrong purpose, or topic descriptions that have to awkwardly serve two different intentions at once rather than being written cleanly for just one.
Does Choosing Between Separate Configurations Always Mean Choosing Entirely Different Pipelines, Step by Step?
Not necessarily. Two distinct use cases might genuinely benefit from the exact same underlying pipeline structure, extract, transform, commit, applied to two entirely different sets of topics, or they might need meaningfully different pipeline behavior, one benefiting from buffering and batch consolidation that the other has no use for at all. The topics a configuration defines and the pipeline steps it uses are two separate design choices, and a team designing for multiple distinct use cases should weigh each independently rather than assuming that different topics automatically require an entirely different pipeline shape, or the reverse.
How Does Weaviate Engram Let a Team Actually Structure Separate Configurations for Genuinely Distinct Use Cases?
Weaviate Engram organizes topics and pipeline behavior into named groups, letting a team keep genuinely distinct use cases cleanly separated while sharing the same underlying platform. Consider a veterinary telehealth platform that needs both personal recall for individual pet owners and separately accumulated, shared knowledge about effective triage practices:
from engram import EngramClient
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
client.memories.add(
"This pet owner mentioned their dog has a known sensitivity to a specific flea prevention medication.",
user_id="owner-449921",
group="personalization",
)
client.memories.add(
"When a dog presents with sudden lethargy and pale gums, prioritize checking for signs of internal bleeding before other differentials.",
group="triage_knowledge",
)
The pet owner’s medication sensitivity lives in the personalization group, scoped to that specific owner and never surfaced for anyone else’s consultation, while the triage insight lives in a completely separate, project-wide group shared across every veterinarian using the platform:
owner_context = client.memories.search(
query="Any known medication sensitivities for this pet?",
user_id="owner-449921",
group="personalization",
)
triage_guidance = client.memories.search(
query="What should be checked first for sudden lethargy and pale gums?",
group="triage_knowledge",
)
Keeping these two genuinely distinct purposes in separate groups means a search against one never accidentally surfaces content meant for the other, and each group’s own topics can be described precisely for exactly the purpose they actually serve, rather than compromising between two different intentions at once. This is exactly the value deliberate pipeline design delivers: matching a system’s actual, distinct needs rather than forcing everything through one undifferentiated configuration that serves neither purpose particularly well.
Designing pipelines around genuinely distinct use cases keeps each one’s memory clean, well-scoped, and searchable on its own terms. One of these use cases deserves particular attention on its own, memory that doesn’t just recall facts but actively improves how an agent behaves over time based on its own accumulated experience. Our next chapter, What are continual learning pipelines for agents?, takes up exactly that pattern.