RAG is not a memory system. It's a policy engine.

How to actually use Vector Databases in production AI agent systems

When I first built a RAG system (a knowledge retrieval tool for internal videos) at my previous firm, my mental model was simple: embed documents, embed the query, find the closest match, stuff it in the prompt. The LLM answers.

The above process isn't wrong. It's just incomplete in a way that only reveals itself when the retrieval is for something that can have real measurable effects. For example, something like policy enforcement, where being wrong has consequences.

In a production AI agent system, the Vector DB isn't there to help the model remember things. It's there to be the authoritative, retrievable source of business rules that the model must reason over, never around. The difference in framing matters enormously for how you build it.

What You're Actually Storing (and Why)

Taking the example of an e-commerce agent system I have been designing, the Vector DB holds two categories of documents:

FAQs: Customer questions and corresponding answers. "What is your return policy?" "Can I exchange a sale item?" "What happens if my item arrives damaged?" These are semi-stable, mostly static and benefit from semantic search. A customer asking "can I send back something I bought last month" should retrieve the return window policy even though they didn't use the word "return window."

Policies: The actual business rules. Return window (30 days, good condition). Refund threshold (50€ auto/manager). Exchange eligibility by product category. Human escalation triggers.

Here's the crucial design decision: the Agent retrieves policy before it takes any action. It doesn't assume or rely on something it was told at system prompt time. Every time it's about to process a return, it fetches the relevant policy chunk from the Vector DB and reasons over it fresh.

Why? Because policies change. A brand might extend its return window for holiday purchases. Product categories get added or removed from exchange eligibility. If your policy is baked into the system prompt, you have a deployment every time a policy changes. If it's in the Vector DB, you update the document, reindex and the agents pick it up immediately.

The RAG Query in an Agent Context

Here's where agent-context RAG gets different from the "chatbot" use case.

In a typical RAG chatbot, the retrieval is driven by the user's question. User asks about returns → retrieve return policy → answer.

In an agent system, the retrieval is driven by the agent's current reasoning state, not the user's words.

The Agent, mid-workflow, might issue a retrieval query that sounds nothing like what the customer said. The customer said "I need to return my jacket." The agent queries the Vector DB with something like: "exchange eligibility rules for outerwear category" because it has already looked up the order, seen it's been 25 days and determined the customer is within the return window but wants to understand exchange options.

This means your Vector DB needs to handle retrieval that is semantically accurate for agent-generated queries, not just user-generated ones. The documents need to be chunked at the right granularity. Not too coarse (retrieving an entire 5-page policy document when you need one rule), not too fine (retrieving a single sentence with no context).

For a policy-heavy system, I would chunk by rule or sub-rule, not by paragraph. "Items must be in original packaging" is a complete retrievable fact. "For exchanges, the following product categories are eligible: [list]" is a complete retrievable fact. A 500-token chunk containing five different rules is going to confuse the retrieval.

When RAG Fails (And It Will)

Retrieval failure in a regular Q&A chatbot means the user gets a suboptimal answer and clicks away mildly annoyed. Retrieval failure in an agent processing a return means the agent might make a decision based on no policy or wrong policy.

The failure modes to plan for:

No match found. The query returns nothing above your similarity threshold. The agent needs to have a defined fallback: flag for human review, ask the customer for more information or default to a conservative policy interpretation (when in doubt, don't auto-approve the 400€ refund).

Wrong chunk retrieved. The cosine similarity was high enough, but the retrieved chunk is about the exchange policy when the agent needed the return policy. Hybrid search (combining dense vector search with sparse keyword matching like BM25) reduces this significantly.

Outdated content. Someone updated the policy PDF but forgot to trigger a reindex. Now the model is enforcing last season's rules. This is an operational problem as much as a technical one as every policy update needs to be a documented process that includes a reindex step.

Hallucination over the retrieved context. The model retrieves the correct policy chunk but then reasons about it incorrectly. "Items must be returned within 30 days of purchase" becomes "this item is returnable" when it has actually been 32 days. This is where read-back confirmation (stating the decision and its basis before executing) catches things that retrieval quality can't.

Scratchpad vs. Long-Term Memory vs. Vector DB

One of the things that tricks people up is conflating three different types of memory in an agent system.

Scratchpad (short-term session memory): Ephemeral, in-memory, per-session. The customer mentioned their order number in turn 2. The agent should not ask for it again in turn 5. This is not in a database but in the session state, flushed when the conversation ends.

Relational DB (operational memory): Customer records, order history, return requests, transaction logs etc.. Structured, queryable by ID, the source of truth for what happened. When the agent needs to know if a customer has already made a return this quarter, this is where it checks.

Vector DB (semantic retrieval): FAQs, policies, product knowledge. This is queried by meaning, not by key. When the agent needs to know what the rules say, this is where it checks.

In a system I was building previously, we initially tried to put everything in the Vector DB because it felt elegant with one store, semantic search for everything. The problem: querying "has this customer made a return in the last 30 days" is not a semantic search problem. It's a SQL query. Forcing it through vector search was slower, less accurate and architecturally backwards.

The "Policy Engine" Framing

I want to come back to the framing I opened with, because it has real engineering implications. If you think of the Vector DB as memory, you might chunk loosely, embed the whole policy PDF and accept occasional retrieval noise. If you think of the Vector DB as a policy engine, you:

• Chunk aggressively at rule boundaries
• Version your policy documents with timestamps
• Test retrieval accuracy on a golden set of agent queries (not user queries)
• Monitor retrieval latency separately from LLM latency
• Have a defined behavior for retrieval failures

The second version is more work but it's also the version that is more robust and operational.

The entire pattern generalizes further than e-commerce. Any system where an AI agent must operate within constraints that change faster than model weights can be updated (which is most real-world systems) needs a retrieval layer that's treated as an engineering problem, not an afterthought.