When should you cache retrieved memories?

The previous chapter looked at keeping any single retrieval fast. A different, complementary opportunity sits just beside that one: some searches ask for essentially the same thing more than once, and a search that’s already been answered recently doesn’t always need to be run again from scratch. This chapter looks at caching retrieved memories, what makes a memory search cacheable in the first place, and where that convenience actually breaks down.

Why Would the Same Underlying Search Ever Get Run More Than Once in a Short Span of Time?

A popular product’s support documentation gets searched by many different customers asking effectively the same question, a frequently reused system prompt triggers the same background lookup on every single turn, or a dashboard refreshes the same summary query every time a page loads. None of these repeated searches are wasteful mistakes, they’re simply a natural consequence of certain queries recurring often, sometimes from different callers who never know they’re asking something someone else just asked moments earlier. A cache exists to notice this repetition and skip redoing work that already has a good, still-valid answer sitting nearby.

What Actually Makes a Vector Search Result Harder to Cache Than a Traditional Database Lookup?

A traditional cache keyed on an exact request string works cleanly because identical requests produce identical cache keys, but two genuinely equivalent memory searches rarely arrive with identical wording. “Helpful for joint pain” and “good for joint pain” ask essentially the same thing, but a cache keyed on exact text would treat them as two completely unrelated requests, missing an opportunity to reuse an answer that’s already sitting right there. A cache built for memory search needs to recognize equivalence in meaning, not just in exact phrasing, which is a genuinely different kind of matching than a traditional cache was ever built to do.

How Does a Cache Actually Recognize That a New Query Is Close Enough to an Older One to Reuse Its Answer?

A semantic cache stores each query it has already answered as its own vector, alongside the result that query produced, and checks a new incoming query against those stored query vectors the same way an ordinary memory search checks a query against stored memories. If a new query lands close enough to a previously cached query, within whatever similarity threshold the cache is configured to accept, the cached result gets reused directly, skipping the full underlying search entirely. This is exactly the similarity-threshold judgment covered several chapters back, applied here to queries themselves rather than to the memories those queries are trying to find.

What Happens When the Underlying Memories a Cached Result Was Based On Actually Change After That Result Was Cached?

This is where caching a memory search gets genuinely riskier than caching most other kinds of data. A cached result that was accurate at the moment it was stored can silently become stale the instant an underlying memory gets updated, added, or removed, and nothing about the cache entry itself signals that anything has changed. Serving a stale cached answer isn’t just a minor inconvenience here, it risks handing back information a system has already moved past, which is a particularly bad failure mode for anything built around keeping memory current in the first place.

How Should a System Actually Decide When a Cached Retrieval Result Has to Be Thrown Away?

The safest default ties a cache entry’s lifetime to how quickly its underlying data is actually expected to change, giving frequently updated memories a short-lived cache entry and giving comparatively stable, rarely updated memories a longer one. A cache entry can also be tied more directly to specific memories rather than to a fixed duration, so that an update to any of those specific memories invalidates the cached result immediately rather than waiting for its arbitrary time limit to run out. Choosing the more direct option costs a little more bookkeeping but avoids ever serving a result that’s already been quietly invalidated by newer information.

How Does Weaviate Engram Support Caching Retrieved Memories Without Risking Stale Results?

Weaviate’s object time-to-live feature lets a collection expire entries automatically after a configured duration, a mechanism well suited to holding cached query results whose freshness window is known in advance. Consider a knowledge-base search widget for a software product, where the same handful of setup questions get asked repeatedly across many different customer sessions, and where the underlying documentation itself only changes a few times a month:

from engram import EngramClient

client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])

client.memories.add(
    "Cached answer for 'how do I reset my two-factor authentication device': "
    "Go to Account Settings, select Security, then choose Reset 2FA Device.",
    topics=["query_cache_setup_docs"],
    properties={"cache_ttl_hours": 12},
)

cached_hit = client.memories.search(
    query="how do I turn off two factor authentication on a lost phone",
    topics=["query_cache_setup_docs"],
    retrieval_config="hybrid",
)

A new customer phrasing the same underlying question in noticeably different words still lands close enough to the cached entry to reuse it directly, skipping a full search over the entire documentation collection. Because the setup documentation itself only changes occasionally, a twelve-hour cache lifetime comfortably balances freshness against the real savings of not re-running the same underlying search for every customer who happens to ask a version of the same common question. This is exactly the value caching delivers for a use case like a documentation search widget, where a large share of traffic clusters around the same handful of recurring questions, and where reusing a recent, still-valid answer costs far less than treating every customer’s question as if no one had ever asked anything similar before.

Caching pays off precisely because some searches repeat, and it stays safe only when a system is deliberate about when a cached answer has to be thrown out. A different kind of repetition matters at a different layer of a system entirely, one where several agents working together all need to draw from, and sometimes contribute to, a shared pool of memory rather than each searching in isolation. Our next chapter, How does retrieval work for multi-agent coordination?, takes up exactly that shared setting.