AI document chunking should split long files into passages that retain enough meaning to answer a business question. Start with document structure, preserve the relationship between rules and exceptions, and test retrieval before choosing a chunk size. A file fitting within a model's input limit does not establish that the assistant can use it reliably.
For business teams, the practical question is straightforward: when an assistant finds a passage, does it also receive the conditions needed to interpret it? A support procedure separated from its exception can produce a confident but incomplete answer. This guide focuses on that boundary problem, rather than the wider architecture covered in SBS's article on building RAG systems over enterprise knowledge bases.
1. Define what each retrieved passage must explain
Before adjusting a splitter, collect representative questions and identify the source passages a competent colleague would need. Include questions that require an exception, a table heading or a reference to another section.
For example, an internal application support guide might describe how to restart a failed import, then explain later that imports already awaiting review must follow a different procedure. A useful retrieved passage needs either both instructions or a dependable route to the related section. Merely finding the words "restart import" is insufficient.
Microsoft's RAG chunking guidance explains that oversized passages can exceed model limits or introduce irrelevant material, while undersized passages can omit necessary context. That tradeoff makes document structure and the questions users ask more useful starting points than a universal word count.
Create a proposed review sheet with the question, required evidence, relevant exception and expected source location. Have the document owner check it. This gives developers a concrete target and helps distinguish a retrieval failure from a poor answer generated from adequate evidence.
2. Choose boundaries that match the document
Different documents need different treatment. A troubleshooting guide has procedures and warnings. A product catalogue has repeated records. A comparison table depends on its row and column headings. Applying identical boundaries to all three is convenient, but should earn its place through testing.
Azure AI Search documents fixed-size, structure-based, semantic and combined chunking approaches. It also describes adding a document title to passages taken from the middle of a file to prevent context loss. These are available approaches, not evidence that one will suit every collection.
Use the following proposed decision criteria when reviewing your documents. Each row identifies a starting approach and the evidence needed to keep it.
| Document pattern | Starting approach | What to inspect before accepting it |
|---|---|---|
| Procedures with headings and warnings | Split by procedure or subsection | Keep prerequisites and relevant warnings with the steps |
| Repeated product or service records | Keep each record together where feasible | Retain the product identity and attribute names |
| Tables spanning several pages | Preserve table structure, or create contextual row groups | Repeat applicable column headings and retain units and footnotes |
| Long narrative guidance | Try bounded passages with overlap | Check whether boundary passages retain the subject and qualifications |
| Very large sections covering several topics | Split within the section and retain its heading path | Verify that each passage has a clear subject |
Treat these choices as hypotheses. A heading can contain several unrelated subjects, and a short paragraph can depend on material several pages away. Someone must inspect the resulting passages, not just the splitting configuration.
3. Preserve context without copying everything
Give each passage a stable connection to its source. A practical record should carry a document identifier, version, heading path and location such as a page or section. Keep the original available so reviewers can examine surrounding material.
Distinguish metadata stored for filtering from context actually shown to the model. Storing a useful section title in the index will not help the answer if the retrieval response omits it. Review the complete evidence package the assistant receives, including its source labels.
Overlap can help where adjacent passages share meaning. Microsoft's Azure AI Search guidance describes overlap as a way to maintain continuity and notes that suitable amounts depend on the content. Its suggested settings are starting points for that implementation, not a universal production standard.
A proposed rule for your team is to add context deliberately: repeat a relevant heading, preserve a qualification, or retrieve a neighbouring passage when needed. Avoid automatically attaching entire chapters to every match. That makes it harder to tell whether the search found useful evidence.
Anthropic's context engineering guidance recommends curating relevant information within the model's finite context. It supports a broader design principle here: more supplied material should have an identifiable purpose. Larger context capacity does not remove the need to select evidence carefully.
4. Test retrieval separately from answer quality
Use the same questions to compare candidate splitting approaches. First inspect the returned passages without reading the generated answer. Does the evidence include the necessary condition? Can a reviewer identify the source? Are repeated passages crowding out a distinct exception?
Then assess the answer against that evidence. This separates two problems that need different fixes. If retrieval misses the exception, rewriting the answer prompt may leave the underlying defect intact. If retrieval includes the exception but the answer ignores it, investigate how the assistant uses the evidence.
Build a small review set around awkward boundaries:
- A question whose answer sits immediately after a split.
- A procedure with an exception in the next subsection.
- A table row that needs its column headings and footnote.
- Similar instructions for two different products.
- A question requiring evidence from separate sections.
- A question the documents cannot answer.
Record evidence completeness, answer correctness and whether the source location supports the answer. Track processing time and retrieved volume as operational observations, without treating shorter or faster responses as automatically better.
Use SBS's AI workflow evaluation guide to structure comparisons before release. Keep the question set stable while changing the chunking approach so the comparison remains interpretable.
5. Avoid fixes that hide missing evidence
A common mistake is choosing the largest permitted passage because it avoids an input error. Capacity establishes what can fit; it does not establish what users need. Another is shrinking passages until search results look precise, while removing the conditions that make those results usable.
Watch for repeated headers, navigation and footers after document parsing. Inspect whether they contribute meaning before retaining them throughout the collection. Microsoft's architecture guidance explicitly includes preprocessing tasks such as removing unwanted elements, reformatting tables and identifying document structure.
Do not repair broken table extraction solely by changing chunk size. If values have lost their headings during parsing, a larger passage may still contain ambiguous data. Correct the representation before comparing retrieval approaches.
Also avoid using a generated summary as the only surviving evidence. If your proposed workflow uses summaries to locate relevant sections, keep a route back to the original passages and check important statements against them. Summaries deserve their own evaluation because omissions can be difficult to spot after the source disappears from view.
When required evidence remains missing, let the assistant report the gap or request clarification. SBS's article on RAG answer abstention explains how to handle that outcome.
6. Run a bounded pilot before changing the whole collection
Choose one document family with a clear owner, such as application troubleshooting guides. Compare the current approach with one alternative, inspect the evidence, and expand only after reviewing the failures. Keep the original documents and record which parser and splitting configuration produced the candidate index.
Use this practical checklist before approving the pilot:
- Confirm that parsed text preserves headings, reading order and table meaning.
- Check passage sizes against the actual embedding and generation models in use.
- Retain source identifiers, versions and navigable locations.
- Inspect rules, exceptions and table boundaries manually.
- Compare retrieval and answer quality separately on the same questions.
- Test questions that require multiple passages and questions with no answer.
- Assign someone to review failed cases and approve the next configuration.
The next step is to select a handful of difficult documents and write the evidence review sheet before changing settings. That turns "make the assistant more accurate" into a reviewable task: demonstrate that each retrieved passage contains, or reliably leads to, the information needed for a correct answer.
India
UAE
KSA
USA
UK
Canada
Oman
Kuwait
Australia
Thailand