Skip to content

docs: add practical RAG troubleshooting checklist #2487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
monkeyjack123 wants to merge 1 commit into
base: main
Choose a base branch
from
+39 −0
Open

docs: add practical RAG troubleshooting checklist #2487

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@ Most code examples are written in Python, though the concepts can be applied in

For other useful tools, guides and courses, check out these [related resources from around the web](https://cookbook.openai.com/related_resources).

## RAG troubleshooting

If you're shipping retrieval systems, see the quick
[RAG troubleshooting checklist](./articles/rag_troubleshooting_checklist.md).

## License

MIT License
34 changes: 34 additions & 0 deletions articles/rag_troubleshooting_checklist.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# RAG Troubleshooting Checklist
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P0 Badge Register new article in publication registry

This commit adds articles/rag_troubleshooting_checklist.md but does not add a matching registry.yaml record, and this repo’s publication workflow requires new content to be registered for static-site inclusion; without that entry, the checklist will be present in the repo but omitted from cookbook.openai.com. Please add a registry item for this path (including its metadata) in the same change.

Useful? React with 👍 / 👎.


Use this quick checklist when a retrieval pipeline underperforms in production.

## 1) Confirm retrieval quality before tuning prompts
- Check whether relevant chunks are returned in top-k for a small labeled test set.
- Inspect chunk boundaries (too small loses context, too large lowers precision).
- Verify embedding model and index settings match your content type.

## 2) Verify context assembly
- Ensure reranking/order is deterministic and sensible.
- Remove duplicate or near-duplicate chunks from final context.
- Include source metadata (doc id, section, timestamp) to aid debugging.

## 3) Control stale or conflicting data
- Validate that index refresh jobs run as expected.
- Add recency or version filters when sources evolve frequently.
- Prefer authoritative sources when conflicts are detected.

## 4) Reduce hallucination risk
- Instruct the model to abstain when evidence is insufficient.
- Require citations to provided context where possible.
- Evaluate with adversarial/no-answer test cases.

## 5) Instrument and evaluate continuously
- Track retrieval recall@k and answer quality separately.
- Log failures by category (retrieval miss, grounding miss, synthesis miss).
- Re-test after each change to retrieval, ranking, or prompting.

## Related Cookbook examples
- [Evaluate RAG with LlamaIndex](../examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb)
- [RAG with graph DB](../examples/RAG_with_graph_db.ipynb)
- [How to call functions for knowledge retrieval](../examples/How_to_call_functions_for_knowledge_retrieval.ipynb)
- [Parse PDF docs for RAG](../examples/Parse_PDF_docs_for_RAG.ipynb)