RAG Knowledge Freshness: Stop AI Assistants Using Retired Documents

RAG knowledge freshness requires a verified path from a source change to the evidence an assistant retrieves. Give documents stable identities, handle withdrawals explicitly, measure when changes become searchable, and check that answers stop relying on retired versions. A scheduled import alone cannot prove that path works.

For a business assistant, outdated instructions can look convincing because they still have a citation. Consider a support guide that changes where customers should send a return request. Updating the original page is only the first step. The searchable copy, its extracted passages and any reused answer must also reflect the change.

1. Define what current means for each source

Start with the decision the assistant supports. An internal glossary may tolerate a slower refresh than a troubleshooting guide for a recently changed application. Ask the content owner how long an outdated answer would remain acceptable, then turn that answer into a measurable freshness requirement.

Record the authoritative location, document identity, current revision and publication status. Keep the source modification time separate from the time your system last fetched it. A recent fetch can retrieve an old export; neither timestamp alone proves that the assistant has the latest approved content.

Also distinguish publication from effectiveness. A guide prepared for a future product release should not automatically replace today's instructions. Your application needs a rule for selecting the version applicable to the question.

This is an operational extension of building RAG systems over enterprise knowledge bases. Retrieval design establishes how evidence reaches the model. Knowledge freshness establishes which evidence should still be eligible.

2. Choose an update method with explicit deletion behavior

An incremental import usually suits a collection with frequent small changes, provided the system can identify existing documents reliably. A complete replacement can suit a controlled collection whose export represents every document that should remain available. Neither approach removes the need to verify the result.

Google Cloud's data refresh documentation distinguishes incremental imports, which add or replace documents with matching IDs, from full reconciliation, which can remove documents absent from the supplied source. It also warns that automatically generated IDs might change across imports. Those details apply to the documented Google import workflows, not every search connector.

Use the following criteria when reviewing your own ingestion process. These are proposed operating choices, not platform performance comparisons.

Full reconciliation deserves particular care. An incomplete export can make valid documents appear withdrawn. Compare the incoming inventory with the expected collection and investigate unexpected omissions before proceeding.

3. Treat document withdrawal as its own workflow

Deleting the original file does not necessarily delete its searchable representation. Microsoft's Azure Storage change and deletion documentation explains that storage indexers detect changes through timestamps, while deletion requires a configured strategy. It says to establish deletion detection from the first indexer run; adding it later does not repair documents already orphaned by earlier deletions.

That requirement is specific to the documented Azure Storage integration. Its broader operational lesson is useful elsewhere: identify how your connector discovers that something no longer exists. Do not infer deletion support from successful update support.

For application-managed ingestion, keep a relationship between each source document and the passages derived from it. A withdrawal should address every searchable child passage. Check any separate keyword index, vector index or answer cache your application actually uses.

Microsoft's document deletion guide describes manual deletion for push indexing and orphaned search documents, using document keys. This makes identity mapping part of recovery planning. A filename search is a weak substitute when several documents share similar titles.

Where appropriate, retain a withdrawal record long enough for ingestion workers to process it. Confirm removal before discarding the information needed to locate indexed copies. If an urgent removal is still processing, use an application-controlled exclusion for that source rather than asking the model to disregard it in a prompt.

4. Measure freshness at retrieval, not at job start

Track the stages separately: source publication, ingestion start, processing completion and verified retrieval. The interval between publication and the first successful current retrieval is a useful measure of propagation delay. A job's start time tells you much less.

Azure AI Search's indexer scheduling documentation states that jobs can queue and may not begin at the scheduled time. An execution that overlaps the next scheduled occurrence can also postpone that occurrence. A schedule therefore expresses an intended cadence, rather than an assurance that every change becomes searchable within that interval.

Build a small set of freshness checks around recognizable content. After an edit, retrieve a passage containing the new instruction. After a withdrawal, check the retired document's identity and questions that previously retrieved it. Then inspect an assistant answer and its citations.

These checks answer different questions. A document lookup tests whether indexed content exists. A representative query tests whether retrieval selects it. An answer check tests whether the assistant uses the current evidence appropriately. Keep these cases alongside your AI workflow evaluation tests, so ingestion changes receive the same attention as prompt changes.

5. Set useful behavior when freshness is uncertain

Decide what users should see when a source exceeds its freshness requirement or a withdrawal remains unresolved. For a low-impact reference question, showing the source revision and a warning may be acceptable. For instructions that depend on a recent application change, route the user to the maintained source or a responsible person until verification succeeds.

Make that decision in application logic using trusted ingestion metadata. A model cannot establish that an index is current merely by reading a document that calls itself the latest version.

Existing conversations need attention too. An earlier turn may already contain a retired passage. For questions requiring current instructions, retrieve fresh evidence and avoid treating an earlier generated answer as the authority. Assess any answer caching separately from document indexing.

When current evidence is unavailable, the assistant should explain the gap rather than reconstruct instructions from an obsolete version. The related guidance on RAG answer abstention covers how to make that response useful.

Freshness also has limits. A newly indexed document may contain an editorial error. A current document may apply to another product version. Content ownership and applicability checks remain necessary even when synchronization works correctly.

6. Run a withdrawal drill before expanding the assistant

Common mistakes include changing document IDs with every export, checking only total document counts, assuming a successful job processed every item, and testing replacement without testing deletion. Another is displaying a global "last updated" time when different sources completed ingestion at different times.

Use this practical checklist for one bounded collection:

• Assign an owner and authoritative source for each document type.
• Define an acceptable propagation delay and the response to a missed target.
• Preserve document identity and map derived passages back to their source.
• Test additions, edits, withdrawals and incomplete exports separately.
• Inspect processing errors and verify representative retrieval results.
• Check citations, existing conversations and any reused answers after withdrawal.
• Record who investigates failures and how the affected source is temporarily excluded.

The next step is a controlled drill with non-sensitive test content. Publish a guide, revise one instruction, then withdraw it. Record what the search layer retrieves and what the assistant answers at each stage. Include a question phrased differently from the guide's title.

Expand the collection after the team can demonstrate replacement and removal, explain any delay, and recover from a failed update. That gives business owners concrete evidence for deciding which workflows the assistant can support.