Why does resending chat history get expensive and slow?

The first few messages of a conversation with an agent barely register as a cost anywhere. A short question, a short answer, done in well under a second, for a fraction of a cent. It’s easy to assume that pattern just continues as the conversation gets longer, scaling in some gentle, predictable way. It doesn’t. Resending the full conversation history on every single call, the pattern that makes an agent feel continuous within one session, creates a cost and a latency curve that gets steep much faster than intuition suggests, and understanding exactly why is what makes the alternative worth taking seriously rather than treating as a minor optimization.

What Actually Gets Billed and Processed on Every Single Call?

When an application calls a language model, it isn’t charged or timed based on just the newest message. It’s charged and timed based on everything included in that request: the system instructions, every prior turn of the conversation still being resent, and the new message on top of all of it. None of the earlier turns get a discount for having already been sent once before. Each one is processed again, in full, as if it were being seen for the first time.

The numbers get large faster than most people expect. A conversation that has gone fifty turns back and forth, which sounds like a perfectly ordinary, moderately long conversation rather than an extreme one, can easily exceed ten thousand input tokens on a single request by that point, just from the accumulated back-and-forth. That’s ten thousand tokens processed and billed for one message, and the fifty-first turn will cost even more, because it inherits everything the fiftieth turn was already carrying plus two more messages on top.

Why Does the Cost Compound Rather Than Just Add Up?

It helps to actually trace where the tokens from an early turn go. The first exchange gets sent once, on turn one. Then it gets sent again on turn two, alongside the second exchange. Then again on turn three, alongside the second and third. By the time the conversation reaches turn fifty, that first exchange has been resent and reprocessed roughly fifty separate times, and every other exchange in between has been resent a similar number of times relative to how early it happened.

Add that up across an entire conversation and the total number of tokens processed doesn’t grow in a straight line with the number of turns. It grows closer to the square of the number of turns, because each new turn doesn’t just add its own size to the total, it adds its own size multiplied by everything already accumulated before it. A conversation twice as long doesn’t cost twice as much to run end to end; it costs closer to four times as much, because both the number of turns and the average size of each request roughly double at the same time.

Why Does Latency Rise Along With Cost?

Cost isn’t the only thing being paid here. A model has to read and process every token in its input before it can begin producing a response, and a larger input takes measurably longer to get through that step. A request carrying ten thousand tokens of resent history doesn’t just cost more than a request carrying a few hundred, it also takes noticeably longer before the response even starts, on top of whatever time the response itself takes to generate.

This matters more for agents than it does for a single person patiently waiting on a chatbot reply, because agents frequently make several model calls in sequence to handle one user request: reasoning about what to do, calling a tool, reasoning about the result, and producing a final answer. If each of those calls is dragging along the same growing history, the slowdown doesn’t happen once, it happens at every step of that chain, and the delays stack on top of each other before the user ever sees a response.

Doesn’t Trimming or Summarizing Older Messages Just Fix This?

A reasonable next idea is to just stop letting the history grow unbounded: periodically summarize older turns into something shorter, or drop the oldest messages once the conversation passes some length. This does help, and it’s a real improvement over doing nothing, but it doesn’t actually solve the underlying problem, it just changes how quickly it comes back.

Summarizing has its own cost, since producing a good summary usually means another model call, run repeatedly as the conversation keeps growing, which eats into the savings it’s supposed to provide. It also risks losing exactly the kind of detail that mattered, since deciding what’s safe to compress requires knowing in advance what will matter later, which usually isn’t obvious at the time it’s being summarized away. And even a well-tuned trimming strategy still ties the size of every request to how long the conversation has been going, just with a gentler slope than sending everything in full. The size sent on request two hundred will still be noticeably larger than the size sent on request twenty, because the summary itself keeps growing to account for everything it needs to represent.

How Does Weaviate Engram Keep Context Size Flat Regardless of History Length?

The fix that actually breaks this relationship is to stop scaling the request size with conversation length at all, rather than scaling it more slowly. Instead of resending history or an ever-growing summary of it, an agent can keep only the last couple of exchanges for immediate conversational continuity, and search for anything older that’s actually relevant to the current message. Weaviate Engram is built around exactly this pattern, and the size of what gets searched and returned doesn’t grow just because the conversation has been running longer.

Picture a subscription billing support bot that’s been helping a customer across dozens of separate conversations over several months, about invoices, plan changes, and failed payments. Rather than carrying that entire history forward, the application keeps only the last couple of exchanges in the active thread, and pulls in anything else through a search:

from engram import EngramClient

engram = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
recent_messages = []  # only the last couple of exchanges live here

def handle_message(user_input, user_id):
    results = engram.memories.search(
        query=user_input,
        user_id=user_id,
        retrieval_config=HybridRetrieval(limit=5),
    )
    memory_context = "\n".join(f"- {m.content}" for m in results)

    recent_messages.append({"role": "user", "content": user_input})
    response = call_model(
        system_prompt=f"Relevant billing history:\n{memory_context}",
        messages=recent_messages[-6:],  # last few exchanges only
    )
    recent_messages.append({"role": "assistant", "content": response})

    engram.memories.add(recent_messages[-2:], user_id=user_id)
    return response

Whether this is the customer’s third message or their three-hundredth, spread across a single sitting or eight months of occasional tickets, the request built this way stays roughly the same size: a handful of recent messages plus a small set of relevant search results, rather than a transcript that keeps getting longer the more the relationship continues. The cost of the tenth conversation with this customer isn’t meaningfully higher than the cost of the first, because nothing about this pattern depends on how much history exists overall, only on how much of it is actually relevant to the message in front of it right now.

None of this addresses a slightly different temptation, though, which is to assume the entire problem could have been avoided from the start simply by using a model with a much larger context window, one big enough to hold all of this history without needing to trim or search anything at all. Our next chapter, Why isn’t a bigger context window enough for memory?, takes that idea seriously and works through exactly why it falls short even when the window is large enough to technically fit everything.