Retrieval Tester
Test retrieval quality with MRR, Hit Rate, NDCG, and Precision metrics.
How It Works
- Configure the test: Enter your query and the list of documents to test.
- Define ground truth: Indicate which documents are actually relevant for this query.
- Analyze metrics: Get MRR, Hit Rate, NDCG and Precision to evaluate your retrieval.
Frequently Asked Questions
- What is MRR (Mean Reciprocal Rank)?
- MRR measures the average position of the first relevant result. An MRR of 1.0 means the right document is always in first position. An MRR of 0.5 means it's on average in position 2.
- What Top K value should I use?
- For factual Q&A, Top K = 3-5 is usually sufficient. For synthesis, increase to 10-20. Too high Top K dilutes relevancy and increases LLM context costs.
- How can I improve my Hit Rate?
- A low Hit Rate indicates a retrieval problem. Solutions: 1) Improve chunking, 2) Test different embedding models, 3) Add reranking, 4) Enrich document metadata.
- What is NDCG?
- NDCG (Normalized Discounted Cumulative Gain) measures ranking quality considering order. A relevant document in position 1 counts more than in position 5. Ideal score: > 0.8.
- Should I use a reranker?
- A reranker (like Cohere Rerank or cross-encoder) improves precision by re-ranking Top K results. Recommended if your MRR < 0.7 or for complex queries.
- How do I test retrieval with real data?
- Create a test set with 50-100 pairs (question, expected relevant documents). Use this tool to calculate metrics. Repeat after each pipeline modification.
