LLMs are getting stronger every day, and building a RAG pipeline has never been easier. Knowing whether it actually works is not. Most teams ship a RAG system, see decent-looking answers, and call it done, until users hit hallucination, missing context, or irrelevant chunks.
That’s where evaluation frameworks come in. RAGAS, TruLens, and DeepEval are three of the most widely used tools for measuring RAG quality. In this article, I’ll break down how each one works and when to reach for it.
A RAG system has two moving parts: the retriever, which fetches context, and the generator, which writes the answer using that context. If either half fails, the final answer fails, but they fail in different ways. A bad retriever pulls irrelevant or incomplete chunks. A bad generator ignores good context and hallucinates anyway, or writes something technically correct but unhelpful.
Traditional NLP metrics like BLEU or ROUGE don’t capture any of this. They compare word overlap between the generated answer and a reference answer, useful for translation, not for judging whether an LLM stayed grounded in facts or whether the retriever did its job. RAG evaluation needs metrics that check both halves separately and together, without always requiring a handwritten correct answer for every query. This is exactly the gap RAGAS, TruLens, and DeepEval were built to fill, and each one approaches it with a different philosophy.
Before diving into each tool, it helps to know that most RAG metrics fall into three buckets:
Every framework below scores some combination of these three. The difference is in how automated scoring is whether it needs ground truth answers, and if the tool is meant for one-time evaluation or continuous monitoring.
It is worth understanding the classic information – retrieval metrics that all three build on. These come from traditional search and recommendation systems, and they measure retrieval quality directly using labeled relevance data, no LLM judge needed.
What it measures: Out of the top K chunks your retriever returned, how many are actually relevant?
Formula: Precision@K = (Relevant chunks in top K) / K
Example: You retrieve the top 5 chunks for a query. 3 of them are relevant to answering it.
Then, Precision@5 = 3 / 5 = 0.6
A high Precision@K means your retriever isn’t pulling in noise. A low score means the LLM must sift through junk context, which increases hallucination risk since irrelevant chunks can confuse the generator.
What it measures: Out of all the relevant chunks that exist in your knowledge base, how many did you actually retrieve in the top K?
Formula: Recall@K = (Relevant chunks retrieved in top K) / (Total relevant chunks in the corpus)
Example: There are 4 relevant chunks in your entire corpus for a query. Your top 5 retrieved chunks contain 3 of them.
Then, Recall@5 = 3 / 4 = 0.75
Precision and Recall usually trade off against each other. Increasing K i.e increasing more chunks tends to raise Recall but lower Precision, since you’re pulling in more noise along with the useful chunks. This is why RAGAS reports context precision and context recall as two separate scores instead of one.
What it measures: How high up was the first relevant chuck in your ranked results? This matters because LLMs tend to pay more attention to earlier context, so burying the one useful chunk at position 8 is worse than having it at position 1
Formula: MRR = (1 / N) × Σ (1 / rank of first relevant chunk for each query)
Example: For Query A, the first relevant chunk is at rank 1 is
reciprocal rank = 1 / 1 = 1
For Query B, the first relevant chunk is at rank 3 then, the reciprocal rank = 1 / 3 = 0.3333
MRR = (1.0 + 0.33) / 2 = 0.665
Higher MRR means your retriever consistently surfaces the best chunk near the top, not buried deep in the results.
What it measures: A more nuanced version of the above, it accounts for graded relevance, i.e some chunks are more relevant than others, not just relevant/irrelevant, and penalize relevant chunks that appear lower in the ranking.
How it works:
Formula:
An NDCG of 1 means your retriever ranked chunks exactly as well as theoretically possible. This metric matters more when relevance isn’t binary. For example: A chunk might be highly relevant, somewhat relevant or irrelevant rather than just yes or no.
Keep these four in mind as you read the next sections, each framework re-implements a version of them, just scored differently:
The main difference is that classic Information Retrieval (IR) metrics require pre-established relevance labels (“which chunks are relevant“) before you can compute anything. RAGAS, TruLens, and DeepEval swap in an LLM for that relevance judgment, avoiding the requirement. This needs little setup but adds more noise than a proper IR benchmark. To check each method’s trustworthiness, label 200 to 300 query/chunk pairs and compute Precision@K, Recall@K, and NDCG as a baseline.
Read more: 12 Important Model Evaluation Metrics
Here we’ll put the 3 frameworks to test:
RAGAS is a python framework built specifically for RAG pipelines. Its biggest selling point is that most of its metrics don’t need human labelled ground truth, it uses an LLM as a judge to score outputs against the retrieval context itself, which makes it fast to set up on an existing pipeline.
You pass RAGAS a dataset of {questions
, retrieved_contexts
, generated_answer
} and optionally a reference answer, and it runs each metric using an LLM under the hood.
A typical run looks like:
Code:
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
results = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision])
Each metrics returns a score between 0 and 1, and RAGAS aggregates them into a summary table you can log or compare across pipeline version.
Synthetic test set generation: One of RAGAS’s most useful features that often gets overlooked, it can auto generate a test dataset directly from your documents instead of you writing questions and reference answers by hand. It samples chunks from your knowledge base, then uses an LLM to generate realistic questions, reference answers, and even harder variants. This solves the biggest bottleneck in RAG evaluation: building a labeled test set is normally the slowest part, and this cuts most of that manual work.
from ragas.testset import TestsetGenerator
generator = TestsetGenerator.from_langchain(llm, embeddings)
testset = generator.generate_with_langchain_docs(
documents,
testset_size=50
)
This gives you a ready-made {question, reference_answer, contexts} dataset to run every metric above against, without hand-labeling a single row.
Strengths: Minimal setup, works well for quick benchmarking, plays nicely with LangChain and LlamaIndex out of the box.
Limitations: Since most metrics rely on an LLM judge, scores can vary slightly run to run, and you’re paying for extra LLM calls on every evaluation pass.
Best for: Teams wanting fast, automated, reference-free scoring right after building a RAG prototype, or for A/B testing two versions of a retriever or prompt.
TruLens takes a different angle, it’s built for observability, not just one-off scoring. Instead of running an evaluation once and getting a report, TruLens instruments your RAG app so it logs every step: what was retrieved, what the LLM saw, and what it generated. That log becomes the basis for its scores.
Core concept: the RAG Triad
What sets TruLens apart is the tooling around these three scores. It ships with a dashboard which is built on Streamlit where you can:
A basic setup looks like wrapping your existing RAG chain with a TruChain or TruLlama recorder, then letting it log automatically as real queries comes in.
Code:
from trulens.apps.langchain import TruChain
tru_recorder = TruChain(
rag_chain,
app_id="rag_v1",
feedbacks=[groundedness, context_relevance]
)
with tru_recorder as recording:
response = rag_chain.invoke(query)
Strengths: Best-in-class visibility into why a RAG system failed on a specific query, not just that it failed. Great for ongoing monitoring rather than a single benchmark run.
Limitations: More setup overhead than RAGAS if you just want a quick score. The value really shows up once you’re running many queries over time, not on a single evaluation batch.
Best for: Teams running RAG in production who need to track quality over time, not just at build time. If you’re iterating on prompts or retrievers and want to see exactly what changed and why, TruLens’s tracing is the stronger fit.
DeepEval is a testing-first framework, it treats RAG evaluation like unit testing for LLM outputs, and plugs directly into Pytest. If your team already runs automated tests before every deployment, DeepEval slots RAG quality checks into that same pipeline instead of living as a separate notebook or dashboard.
Core metrics:
How it works in practice: You define test cases the same way you would write a unit test, set a pass/fail threshold, and run it through Pytest:
Code:
from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase
test_case = LLMTestCase(
input=query,
actual_output=generated_answer,
retrieval_context=retrieved_chunks
)
faithfulness_metric = FaithfulnessMetric(threshold=0.7)
assert_test(test_case, [faithfulness_metric])
If faithfulness drops below the threshold, the test fails, the same way a broken function would fail a unit test. This makes it straightforward to block a deploy if a change to your retriever or prompt quietly degrades RAG quality.
Strengths: Fits naturally into existing CI/CD and testing workflows. Clear pass/fail gates instead of just a score to interpret. Custom metrics via G-Eval add flexibility for niche use cases.
Limitations: Being test-oriented, it’s less suited for open-ended exploration or visual debugging, for that, TruLens’s dashboard is a better fit.
Best for: Engineering teams that want RAG evaluation baked into their existing test suite and deployment pipeline, with pass/fail gates rather than exploratory dashboards.
| Framework | Primary Use Case | Ground Truth Needed | Integration Style | Output Format |
|---|---|---|---|---|
| RAGAS | Fast automated scoring | Mostly No | LangChain/LlamaIndex | Score Table |
| TruLens | Observability and monitoring | No | Dashboard+tracing | Live Dashboard |
| DeepEval | CI/CD testing | Optional | Pytest-style | Pass/fail tests |
Case 1:
Just built your first RAG prototype and want quick scores use RAGAS. Run it once against a sample set of queries, get faithfulness and relevance scores, and know within minutes if your retriever or prompt needs work.
Case 2:
Running RAG in production and need to catch quality drift use TruLens. Instrument your app once, then keep an eye on the dashboard as real traffic flows through. If groundedness starts dropping after a model or embedding change, you’ll see it immediately.
Case 3:
Want RAG quality checks inside your deployment pipeline use DeepEval. Write it once as a test suite, and every PR that touches your retriever or prompt gets automatically checked before it ships.
Many teams don’t pick just one. A common pattern is using RAGAS or DeepEval for structured scoring during development, and TruLens for live monitoring once the system is in production. They solve overlapping problems but at different points in the lifecycle, prototype, ship, and monitor.
A few things to watch for regardless of which framework you pick:
RAG evaluation isn’t optional once you move past a demo. RAGAS gives you fast, reference-free metrics for quick iteration. TruLens gives you visibility into production, query by query. DeepEval gives you testing discipline that fits a normal engineering workflow.
None of them is objectively better; the right pick depends on whether you’re prototyping, monitoring, or shipping. Start with RAGAS for a baseline, add DeepEval once you have a CI/CD pipeline worth protecting, and bring in TruLens once real users are hitting your system. That combination will catch quality issues long before your users do.
A. Mostly no. RAGAS and TruLens use an LLM judge to score against retrieval context, so most metrics run reference-free. DeepEval treats it as optional.
A. Use RAGAS for fast scoring on a new prototype, DeepEval for pass/fail checks in your CI/CD pipeline, and TruLens for monitoring live production traffic.
A. They only measure word overlap against a reference answer. They can’t tell whether the model stayed grounded in facts or whether the retriever fetched the right chunks.