Short answer: Cap accept rate and in-flight Engram runs together—memory writes spend extract, reconcile, and commit work, not just HTTP request counts.
High availability keeps memory searchable when nodes fail; rate limiting keeps it writable when agents succeed too loudly. Engram accepts memories.add quickly and processes extract, transform, and commit in the background—good for latency, dangerous if every turn dumps unbounded history. Ordinary API limits count requests; memory pipelines spend different currencies. This chapter covers throttling accept rate versus in-flight runs, how Engram’s async model changes backpressure design, and how to pace writes with token buckets and concurrency caps in front of the client. During bulk migrations throttle harder than interactive traffic. Watch failed runs so throttles protect quality instead of hiding outages. Prefer backpressure to blind drops; tighten local buckets for noisy tenants rather than punishing everyone globally.
High availability keeps memory searchable when nodes fail. Rate limiting keeps memory writable when agents succeed too loudly. Engram accepts memories.add quickly and processes extract, transform, and commit in the background. That asymmetry is a gift for latency. It is also a trap if every agent turn dumps unbounded history into the pipeline. This chapter explains what you are actually throttling, how backpressure differs from hard rejection, how to pace Engram writes with token buckets and concurrency caps, and how to watch failed runs so throttles protect quality instead of hiding outages.
Why Do Memory Write Pipelines Need Their Own Limits?
Ordinary API rate limits count HTTP requests. Memory write pipelines spend different currencies. Each accepted add may trigger LLM extraction, reconciliation searches, and vector commits. Ten chatty agents can produce a quiet HTTP graph and a red extract queue. The store stays up. The memories arrive late, collide, or fail.
Weaviate’s own import tooling shows the same idea at a lower layer. Batch clients offer dynamic sizing, fixed batches, and explicit rate_limit modes so vectorizer APIs are not overrun. Engram sits above that world as a managed memory service. Your application still owes an admission policy for how fast raw conversation enters the pipeline.
Without that policy, HA clusters spend their redundancy absorbing self-inflicted load. Throttles are part of availability design, not a separate politeness layer.
What Should You Throttle: Accept Rate, In-Flight Runs, or Both?
Separate the gates. An accept-rate limit caps how many memories.add calls start per user, tenant, or process each minute. An in-flight cap limits how many pipeline runs may be outstanding before new work waits. Rate alone is not enough. A hundred requests per minute with long extract steps can still flood workers. Concurrency alone is not enough. A brief burst can still exceed provider quotas downstream.
Token buckets fit bursty agent traffic. They allow a short surge after quiet periods, then refill at a sustained rate. Pair the bucket with a semaphore. The bucket smooths average intake. The semaphore bounds simultaneous pipeline pressure. Per-tenant buckets prevent one workspace from starving the rest. Evict idle bucket state so the map of tenants does not become its own leak.
Prefer backpressure over silent drops. Block or queue inside your service until a slot opens. When you must reject at an edge, return a clear retry hint. Blind retries from every agent at once recreate the stampede you just stopped.
How Does Engram’s Async Model Change Throttle Design?
Engram returns a run_id immediately and continues work asynchronously. Hot paths should not call runs.wait on every turn. That keeps user latency low. It also means accept success is not commit success. Your throttle dashboard must include run outcomes: completed, failed, and lingering running counts, not only HTTP 200 rates.
Use waits in drills, backfills, and CI. There you can confirm commits before opening the next batch. For live chat, fire and forget under the admission policy, then sample run health in the background. If failed runs climb while accept rate is flat, tighten topic scope or slow intake. The pipeline is telling you the content or the model path is unhealthy.
Buffer steps inside Engram can pause work until a trigger continues it. Application-level queues should respect the same idea. Do not keep shoving new conversation into a group whose runs are already failing. Pause that scope, alert, and resume after a clean sample completes.
How Do You Implement Throttled Writes With Weaviate Engram?
Put the limiter in front of the Engram client, not after a giant in-memory list of pending turns. Bound the queue. Convert depth into expected wait using recent service times so you reject work that cannot meet your latency budget. Log shed decisions with scope identifiers so operators see which greenhouse bay or customer cohort is flooding the system.
During bulk migrations or historical replay, throttle harder than on interactive traffic. Replays lack natural human pacing. They will saturate extract workers unless you cap objects per minute the way Weaviate batch rate_limit caps import against vectorizer quotas. Interactive traffic can keep a small burst capacity so a single rich turn is not punished.
Here is an orchid house desk that paces Engram writes with a simple token bucket and concurrency gate before calling memories.add. The scenario is a humidity alert storm that would otherwise enqueue duplicate notes for every sensor tick.
import os
import time
import threading
from engram import EngramClient
client = EngramClient(api_key=os.environ["ENGRAM_API_KEY"])
group = "orchid_house"
grower = "grower-sam"
bay = "orchid-bay-3"
class WriteGate:
"""Token bucket for accept rate + semaphore for in-flight Engram runs."""
def __init__(self, rate_per_sec: float, burst: int, max_in_flight: int):
self.rate = rate_per_sec
self.tokens = float(burst)
self.burst = float(burst)
self.updated = time.monotonic()
self.lock = threading.Lock()
self.slots = threading.Semaphore(max_in_flight)
def acquire(self, timeout: float = 5.0) -> bool:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
with self.lock:
now = time.monotonic()
self.tokens = min(self.burst, self.tokens + (now - self.updated) * self.rate)
self.updated = now
if self.tokens >= 1.0 and self.slots.acquire(blocking=False):
self.tokens -= 1.0
return True
time.sleep(0.05)
return False
def release(self) -> None:
self.slots.release()
gate = WriteGate(rate_per_sec=2.0, burst=6, max_in_flight=3)
def note_humidity(reading_pct: float) -> dict:
if not gate.acquire():
return {"accepted": False, "reason": "throttled", "bay": bay}
try:
run = client.memories.add(
[
{
"role": "user",
"content": (
f"Bay {bay}: humidity reading {reading_pct:.1f}%. "
"If leaf tips brown, raise misting only after substrate "
"surface has dried for twenty minutes."
),
},
{
"role": "assistant",
"content": "Queued orchid humidity note for memory extraction.",
},
],
user_id=grower,
group=group,
properties={"bay_id": bay, "cost_center": "greenhouse"},
)
return {"accepted": True, "run_id": run.run_id, "status": run.status, "bay": bay}
finally:
gate.release()
# Sensor storm: many ticks; gate keeps pipeline intake bounded
outcomes = [note_humidity(72.0 + (i % 5) * 0.4) for i in range(12)]
accepted = sum(1 for o in outcomes if o.get("accepted"))
print({"accepted": accepted, "shed": len(outcomes) - accepted, "sample": outcomes[:3]})
Tune rate and burst from measured extract capacity, not from wishful QPS. If canary searches stay healthy while accept sheds rise, the gate is doing its job.
Which Signals Tell You the Throttle Is Wrong?
Watch accept rejects beside failed-run rate and search empty-hit rate. Rising sheds with flat failures means you are protecting a healthy pipeline. Rising failures with low sheds means you are admitting poison or an overloaded model path. Rising queue wait with happy HTTP clients means backpressure is missing at the edge.
Alert on sustained saturation of the semaphore. That is early warning before Engram run lag becomes user-visible forgetfulness. Review per-scope hot spots. A single bay or tenant that dominates admits usually needs a tighter local bucket, not a global slowdown that punishes everyone.
Rate limiting memory writes is admission control for expensive asynchronous work. Cap accept rate and in-flight runs together. Prefer backpressure to blind drops. Measure Engram run health, not only request counts. Then agents stay memorable without trampling the pipelines that make memory possible. Clean API shapes make those limits easier to enforce consistently.
Our next chapter, How should you design APIs for memory services?, turns from protecting the write path to shaping the interfaces agents and applications use to store and recall memory through Engram.