Evaluating Retrieval Quality in RAG Pipelines: Precision@k, Recall@k, and F1@k Explained

Evaluating retrieval quality in RAG (Retrieval-Augmented Generation) pipelines is essential to ensure that the system retrieves relevant documents before passing them to the language model for answer generation. Without accurate retrieval, even the most advanced LLM will struggle to produce correct or meaningful responses. The most widely used metrics for assessing retrieval performance are Precision@k, Recall@k, and F1@k. These are binary, order-unaware measures that evaluate how well the top-k retrieved chunks match the ground truth — the set of documents that actually contain the answer to a given query. Precision@k measures the proportion of retrieved chunks that are truly relevant. It answers the question: “Out of the top k results, how many are correct?” A high precision means the retrieved results are mostly relevant, minimizing false positives. However, it doesn’t guarantee that all relevant documents are included. Recall@k measures the proportion of all relevant chunks that appear in the top-k results. It answers: “Out of all the relevant documents, how many did we retrieve?” A high recall indicates that the system captures most of the relevant information, even if it also includes some irrelevant ones. F1@k combines both precision and recall into a single harmonic mean score. It provides a balanced view of retrieval performance. A high F1@k means the system performs well on both accuracy and completeness — it finds most relevant documents without flooding the results with noise. HitRate@k is a simpler metric that checks whether at least one relevant document appears in the top-k results. It’s useful as a basic pass/fail indicator: if the system fails to retrieve any relevant chunk, the answer generation step is unlikely to succeed. In practice, these metrics are computed across a test set of queries, each with its own ground truth. For example, in the War and Peace case study, the query “Who is Anna Pávlovna?” has three relevant text chunks. After retrieving and reranking the top 10 chunks, the system achieved a HitRate@10 of 1 (it found at least one relevant chunk), Precision@10 of 0.4, Recall@10 of 0.67, and F1@10 of 0.5. These results suggest the system performs reasonably well but could improve. The recall of 0.67 indicates that about two-thirds of the relevant chunks were retrieved, while precision of 0.4 means 60% of the top results were irrelevant. The F1 score of 0.5 reflects a moderate balance between the two. It’s important to note that these metrics are order-unaware, so reranking doesn’t change the values as long as the same set of top-k chunks is retrieved. However, reranking can still improve the quality of the final context passed to the LLM by placing the most relevant documents higher. Ultimately, monitoring these metrics over time helps identify issues in the embedding model, chunking strategy, or vector search setup. They serve as key indicators for tuning and improving RAG systems, ensuring that the foundation of retrieval is solid before generating answers.