RAG Basics

Building a Retrieval-Augmented Generation (RAG) System

A Retrieval-Augmented Generation (RAG) system combines two things:

1. Retrieval: finding relevant documents for a question.
2. Generation: giving those documents to a language model so it can generate an answer.

Instead of asking a language model to answer a question using only what it learned during training, we first retrieve relevant information and include it in the model's input.

The basic workflow is:

Question ↓
Retrieve relevant documents to the question ↓
Build a prompt with the documents and the question ↓
Generate an answer

In this tutorial, we build a simple RAG pipeline using HotpotQA, Sentence-Transformers, FAISS, BM25, and T5-small. , The implementation has four main stages:

1. Prepare the documents
2. Create an index for retrieval
3. Retrieve relevant documents
4. Generate an answer from the retrieved documents

We then evaluate how well both retrieval and generation work.

1. Data preprocessing

What are we doing?

Before we can retrieve documents, we need a collection of documents to search.

We use the HotpotQA dataset. Each example contains a question, an answer, supporting facts, and a collection of article titles and sentences.

We can first look at what information is available in one example:

hotpot["train"][0].keys()

Output:

dict_keys(['id', 'question', 'answer', 'type', 'level', 'supporting_facts', 'context'])

The ‘context’ contains article titles and their sentences.

For example:

hotpot["train"][0]['context']['title']

gives a list of article titles such as:

['Radio City (Indian radio station)',
 'History of Albanian football',
 'Echosmith',
 "Women's colleges in the Southern United States",
 'First Arthur County Courthouse and Jail',
 "Arthur's Magazine",
 '2014–15 Ukrainian Hockey Championship',
 'First for Women',
 'Freeway Complex Fire',
 'William Rast']

The corresponding sentences are stored separately:

hotpot["train"][0]['context']['sentences']

So we need to turn this structure into something that is easier to search.

Creating the document corpus

We create a dictionary where:

• the key is the article title
• the value is the article text

corpus = {} # {title : text}


for example in hotpot["train"]:
   title_list = example["context"]["title"]
   sentences = example["context"]["sentences"]


   for title, article_sentences in zip(title_list, sentences):
       if title not in corpus:
           corpus[title] = " ".join(article_sentences)

For example, several sentences belonging to an article are joined together into one text:

Arthur's Magazine (1844–1846) was an American literary periodical  
published in Philadelphia in the 19th century. Edited by T.S. Arthur,  
it featured work by Edgar A. Poe...

We can check the size of the resulting corpus:

len(corpus)

Output:

482021

So the training corpus contains of 482,021 unique documents.

2. Creating document embeddings

What is an embedding?

An embedding represents text as a list of numbers - a vector in a semantic space. Semantic spaces aim to create representations of natural language that capture meaning. We can say that (good) word embeddings form a semantic space and will refer to a set of word vectors in a multi-dimensional space as "semantic space".

The idea is that texts with similar meanings should have similar embeddings. The embeddings are usually trained specifically to reflect word similarity. One interesting thing that happens because of this is that many semantic and syntactic relationships between words are (almost) linear in word vector space created by these vectors.

There is a very popular example: the difference between king and queen is (almost) the same as between man and woman. Or a word that is similar to queen in the same sense that kings is similar to king turns out to be queens.

In this tutorial we will be using the SentenceTransformer model:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")

We then separate the document texts and titles:

texts = list(corpus.values())
titles = list(corpus.keys())

Now we create an embedding for every document:

embeddings = model.encode(
   texts,
   batch_size=32,
   show_progress_bar=True
)

The resulting embeddings have this shape:

embeddings.shape

(482021, 384)

This means:

• there are 482,021 document embeddings
• each embedding contains 384 numbers

So instead of representing a document as only text, we now also have a numerical representation that can be compared with a question.

To find out if two vectors (that represent a text document and a question that needs to be answered) are similar, we are going to use the cosine similarity of those two vectors.

What is cosine similarity?

Suppose our model turns two pieces of text into vectors: A and B. We can think of each vector as an arrow pointing in some direction in a high-dimensional space. For example, in this tutorial the model is converting a text into a 384-dimensional vector.
Cosine similarity asks: How similar is the direction of these two arrows?

The cosine similarity is based on the angle between the two vectors:
1 → same direction → very similar
0 → roughly perpendicular → little similarity
-1 → opposite direction → very different

The important thing about an embedding is not necessarily the absolute size of its numbers. What we want is for similar meanings to have similar direction, so that when a question is asked, the most relevant (“similar”, pointing in roughly the same direction) text documents can be retrieved for the question to be answered.

3. Building a FAISS index

What is FAISS?

FAISS is a library for searching through large collections of vectors.

We use it to find the documents whose embeddings are most similar to the embedding of a question.

First, we normalize the document embeddings:

import faiss
faiss.normalize_L2(embeddings)

Then we create a FAISS index:

index = faiss.IndexFlatIP(384)
index.add(embeddings)
# cos(A, B) = A * B / ||A|| * ||B||
# normalize the embeddings (dividing each by its length)
# inner product == cosine similarity, because we already normalized the vectors

IndexFlatIP uses inner product (generalization of dot product) to compare vectors.

IndexFlatIP means FAISS is using inner product to compare vectors. Normally, inner product and cosine similarity are not exactly the same thing. But if we L2-normalize the vectors first, every vector has the length of 1. For two normalized vectors: inner product == cosine similarity.

We can check how many vectors are in the index:

print(index.ntotal)

Output:

482021

So the index contains all 482,021 document embeddings.

We also save the normalized embeddings so that they do not have to be recreated every time:

import numpy as np

np.save("embeddings_normalized.npy", embeddings)

4. Retrieving documents

Now we can try the retrieval system.

For example, the first training question in the notebook is:

Which magazine was started first: Arthur's Magazine or First for Women?

We first convert the question into an embedding:

question_embedding = model.encode([question])

faiss.normalize_L2(question_embedding)

Then we search for the five closest documents:

scores, indices = index.search(
    question_embedding,
    5
)

The returned indices tell us which documents were retrieved, while scores tell us how similar they were to the question.

The highest-scoring document was:

Arthur's Magazine (1844–1846) was an American literary periodical  
published in Philadelphia in the 19th century...

with a similarity score of approximately:

0.7296

The other retrieved documents included:

Arthur magazine  
First for Women  
Arthur's Home Magazine  
The Ladies' Magazine

This is useful because the system has found documents related to the question without requiring an exact match of all the words.

5. Comparing dense retrieval with BM25

FAISS is not the only way to retrieve documents.

As a baseline, we also implement BM25, a traditional text-based retrieval method.

The main difference is:

• BM25 relies heavily on matching words in the question and document.
• Dense retrieval compares the meaning represented by their embeddings.

Building the BM25 index

We first tokenize the documents by converting them to lowercase and splitting them into words:

from rank_bm25 import BM25Okapi

tokenized_corpus = [text.lower().split() for text in texts]

bm25 = BM25Okapi(tokenized_corpus)

For the question, we do the same tokenization:

tokenized_question = question.lower().split()

bm25_scores = bm25.get_scores(tokenized_question)

Then we select the five documents with the highest scores:

# sorting the text's indices by their scores from min to max, then reversing and taking top-5 max scores
top_indices = np.argsort(bm25_scores)[::-1][:5]

For example, for the question:

Prince William, Duke of Gloucester and Prince George of Denmark,  
were eventually monarchs of which Kingdom?

BM25 returned documents such as:

Dorothy Clement  
Wedding of Prince William and Catherine Middleton  
Prince Georg of Hanover  
Prince Welf Ernst of Hanover  
Prince George of Cambridge

The dense FAISS search produced a different result:

Prince George of Denmark  
Prince William, Duke of Gloucester  
Prince William of Gloucester  
Monarchy of Denmark  
Frederick, Hereditary Prince of Denmark

This illustrates an important difference.

BM25 is strongly affected by exact word overlap, while dense retrieval can use the semantic similarity between the question and documents.

6. Comparing retrieval quality

Looking at individual examples is useful, but we also want to measure retrieval performance across many questions.

For this evaluation, we create a separate validation corpus containing 66,581 documents.

validation_texts = {}

for example in hotpot["validation"]:
   for title, sentences in zip(
       example["context"]["title"],
       example["context"]["sentences"]
   ):
       validation_texts[title] = " ".join(sentences)

validation_titles = list(validation_texts.keys())
validation_texts = list(validation_texts.values())

The validation documents are then embedded and added to another FAISS index.

validation_embeddings = model.encode(validation_texts, batch_size=16, show_progress_bar=True)
faiss.normalize_L2(validation_embeddings)
validation_index = faiss.IndexFlatIP(384)
validation_index.add(validation_embeddings)

We also create a BM25 index for the same validation documents.

tokenized_validation_texts = [text.lower().split() for text in validation_texts]
validation_bm25 = BM25Okapi(tokenized_validation_texts)

For each validation question, we check whether at least one of its gold supporting documents appears in the top 2, top 5, or top 10 retrieved documents.

recall_bm25 = {2: 0, 5: 0, 10: 0}

recall_dense = {2: 0, 5: 0, 10: 0}

\# What percentage of questions had all required supporting articles retrieved within the top k=2/5/10 ?

recall_bm25 = {2: 0, 5: 0, 10: 0}
recall_dense = {2: 0, 5: 0, 10: 0}
# What percentage of questions had all required supporting articles retrieved within the top k=2/5/10 ?
for example in hotpot["validation"]:

   question = example["question"]
   gold_titles = set(example["supporting_facts"]["title"])

   # BM25
   question_tokens = question.lower().split()
   bm25_scores = validation_bm25.get_scores(question_tokens)
   bm25_indices = np.argsort(bm25_scores)[::-1][:10]
   bm25_titles = [validation_titles[i] for i in bm25_indices]

   # FAISS
   question_embedding = model.encode([question]).astype("float32")
   faiss.normalize_L2(question_embedding)
   faiss_scores, faiss_indices = validation_index.search(question_embedding, 10)
   faiss_titles = [validation_titles[i] for i in faiss_indices[0]]

   # Measure recall
   for k in [2, 5, 10]:
       if gold_titles.intersection(bm25_titles[:k]):
           recall_bm25[k] += 1
       if gold_titles.intersection(faiss_titles[:k]):
           recall_dense[k] += 1

The results were:

Retriever	Recall@2	Recall@5	Recall@10
BM25	70.22%	81.65%	87.68%
FAISS	82.59%	89.99%	93.44%

The FAISS dense retriever performed better at all three values of k.

For example, at k=2:

• BM25 retrieved a supporting document for 70.22% of questions.
• FAISS did so for 82.59% of questions.

For this experiment, dense retrieval with FAISS performed better than BM25 at finding at least one supporting document in the top-k results.

7. Generation

Retrieval gives us useful documents, but a RAG system still needs to produce an answer.

For generation,we are using T5-small:

from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("t5-small")
generator = T5ForConditionalGeneration.from_pretrained("t5-small")

T5 has an encoder-decoder architecture. Encoder-decoder is the standard modeling paradigm for sequence-to-sequence tasks. This framework consists of two components:

• encoder - reads source sequence and produces its representation;
• decoder - uses source representation from the encoder to generate the target sequence.

It originated from a translation task, where you have a text in one language and need to produce a translation of it in another language using only the original text, so encoder encodes the original and decoder decodes the translation from it. But turns out this kind of architecture also does well at any text-to-text task.

The high-level pipeline of a text-to-text model looks something like this:

• feed source and previously generated target words into a network;
• get vector representation of context from the decoder;
• from this vector representation, predict a probability distribution for the next token.

For a more in-depth understanding of language modeling I would highly recommend Lena Voita’s NLP course, which the illustration above is borrowed from, particularly the “Language modeling" and the “Seq2Seq & Attention” chapters.

We also use the same Sentence-Transformer model that we used before to retrieve documents.

from sentence_transformers import SentenceTransformer

embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

The way that the prompt for generation is constructed is like this:

context = "\n\n".join(
   f"Document {i}: {text}"
   for i, text in enumerate(retrieved_texts, start=1)
)
prompt = f"""question: {question}
context:
{context}
answer:"""

So we put together all the sentences from the ‘context’ into the same string object and present them after the question and the ‘context:’ inside the prompt. Then to help the model understand what it needs to provide next there is a ‘answer:’ that the prompt ends with.

We will be using a helper function for generation:

import torch

def generate_answer(
   generator,
   tokenizer,
   prompt,
   max_new_tokens=32,
):
   inputs = tokenizer(
       prompt,
       return_tensors="pt",
       truncation=True,
   )
   generator.eval()

   with torch.no_grad():
       outputs = generator.generate(
           input_ids=inputs["input_ids"],
           attention_mask=inputs["attention_mask"],
           max_new_tokens=max_new_tokens,
           num_beams=1,
           do_sample=False,
           decoder_start_token_id=generator.config.decoder_start_token_id,
           eos_token_id=generator.config.eos_token_id,
           pad_token_id=generator.config.pad_token_id,
           use_cache=True,
       )

   return tokenizer.decode(
       outputs[0],
       skip_special_tokens=True,
   )

At the generation step is where even a small GPU can significantly speed up the process. The most simple (and free) way to do this in my opinion is to use one at Google Colab. You just need to open your notebook with code in Colab, click to Runtime, then Change runtime type, then choose the ‘T4 GPU’ at the Hardware accelerator section (there might be some other model of GPU available when you do this, the important thing is that it is a GPU).

8. Evaluating retrieval

We evaluate retrieval by comparing the documents returned for each question with the relevant document titles in HotpotQA's supporting facts.

Precision@k

Precision@k measures the proportion of the top k retrieved documents that are relevant:

For example, if 2 out of the top 5 documents are relevant, Precision@5 = 2/5 = 0.4.

# Of the first k documents we retrieved, how many were actually relevant?
def precision_at_k(retrieved_titles, relevant_titles, k):
   retrieved = retrieved_titles[:k]
   return sum(
       title in relevant_titles
       for title in retrieved
   ) / k

We calculate Precision@1, Precision@2, and Precision@5 by setting k to 1, 2, and 5.

Mean Reciprocal Rank (MRR)

MRR measures how highly the first relevant document appears. Its reciprocal rank is 1 / rank. If the first relevant document is third, the score is 1/3; if none is retrieved, the score is 0.

# How high in the results was the first relevant document?
def reciprocal_rank(retrieved_titles, relevant_titles):
   for rank, title in enumerate(retrieved_titles, start=1):
       if title in relevant_titles:
           return 1.0 / rank

   return 0.0

We average the reciprocal ranks across all questions to obtain MRR.

These results average performance across 100 validation questions. Precision@k measures the proportion of relevant documents in the top results, while MRR measures how early the first relevant document appears.

9. Evaluating generated answers

Retrieval quality is only part of the RAG system.

Even when the correct information is retrieved, the generator still needs to produce the correct answer.

We evaluate generated answers using:

• Exact Match (EM)
• F1

Exact Match (EM) checks whether the generated answer exactly matches the expected answer.

# Exact Match (EM): 1 → prediction matches the gold answer, 0 → it doesn't
def exact_match(prediction, gold):
   return int(
       normalize_answer(prediction)
       == normalize_answer(gold)
   )

F1 gives partial credit when the generated answer overlaps with the expected answer.

# F1 compares the individual words/tokens in the prediction and gold answer. Precision — how much of the prediction was correct? Recall — how much of the gold answer did we recover?
# F1 = (2 * Precision * Recall) / (Precision + Recall)
def f1_score(prediction, gold):
   prediction_tokens = normalize_answer(prediction)
   gold_tokens = normalize_answer(gold)

   if not prediction_tokens or not gold_tokens:
       return float(prediction_tokens == gold_tokens)

   common = Counter(prediction_tokens) & Counter(gold_tokens)
   overlap = sum(common.values())

   if overlap == 0:
       return 0.0

   precision = overlap / len(prediction_tokens)
   recall = overlap / len(gold_tokens)

   return 2 * precision * recall / (precision + recall)

This is how to visualize precision and recall that make up the F1 score more easily:

These four terms (false positive/false negative/true positive/true negative) are used to evaluate how well a model makes predictions. They compare the model's predictions with what is actually true.

	Document is relevant	Document is not relevant
Retrieved	True Positive (TP) — correctly retrieved a relevant document	False Positive (FP) — retrieved an irrelevant document
Not retrieved	False Negative (FN) — missed a relevant document	True Negative (TN) — correctly left out an irrelevant document

A simple way to remember them:

• True = the model got the classification right.
• False = the model got the classification wrong.
• Positive = the model predicted “relevant.”
• Negative = the model predicted “not relevant.”

The relatively low generation scores compared with the retrieval results show an important point: good retrieval does not automatically produce good answers. The system may retrieve useful information but still fail when the language model has to turn that information into the final answer.

We can also assess the generated answers by question type:

• Bridge
• Comparison

For example, a bridge question is:

What government position was held by the woman who portrayed  
Corliss Archer in the film Kiss and Tell?

Its gold answer is:

Chief of Protocol

The comparison example is:

Were Scott Derrickson and Ed Wood of the same nationality?

Its gold answer is:

yes

To aggregate statistics by question type, we can do this:

# Add HotpotQA query type to the generation results
generation_df["query_type"] = [
   validation[i]["type"]
   for i in range(len(generation_df))
]

# Generation performance by query type
generation_slices = (
   generation_df
   .groupby("query_type")
   .agg(
       count=("question", "size"),
       EM=("em", "mean"),
       F1=("f1", "mean")]
   )
   .reset_index()
)

10. Error analysis

Good evaluation scores do not tell us exactly why a RAG pipeline makes mistakes. Error analysis means inspecting individual questions, retrieved documents, and generated answers to identify what went wrong.

Missed retrievals

A missed retrieval occurs when the retriever fails to return a document needed to answer the question.

Common reasons include:

• Different wording: The question uses different words from the relevant document.
• Missing context: A question requires multiple documents (like a bridge question), but the retriever finds only one.
• Incorrect ranking: Relevant documents exist in the corpus but do not appear high enough in the results.

To investigate, inspect questions with low retrieval scores and compare the retrieved titles with the expected supporting facts. If the relevant information is missing from the retrieved context, the problem starts at the retrieval stage.

Hallucinations

A hallucination occurs when the generator produces an answer that is incorrect or unsupported by the retrieved context.

Possible causes include:

• The retriever returned irrelevant or incomplete information.
• The generator misunderstood the context provided by the retriever.
• The model produced an answer based on its learned knowledge rather than the supplied documents.

Inspect incorrect answers and check whether the retrieved context actually supports them. If the evidence is missing, improve retrieval. If the evidence is present but the answer is wrong, investigate the prompt or generation model.

Overly long prompts

The generator receives the question together with the retrieved documents. If too much context is included, the prompt can become unnecessarily long.

This can:

• Increase generation time and computational cost.
• Include irrelevant information that distracts the model.
• Exceed the model's input token limit, potentially truncating useful context.

To investigate, compare answers using different numbers of retrieved documents, such as the top 2, 5, or 10. Measure answer quality and prompt length to find a useful balance between providing enough evidence and keeping the prompt concise.

So, when an answer is incorrect, check the pipeline in order:

1. Retrieval: Were the relevant documents found?
2. Context: Did the prompt contain enough useful evidence without unnecessary details?
3. Generation: Did the model produce an answer supported by that evidence?

This helps distinguish retrieval errors from generation errors and identify which part of the pipeline needs improvement.

11. Reporting

The retrieval results by question type were:

Question type	Count	Precision@1	Precision@2	Precision@5	MRR
Bridge	79	0.7595	0.5190	0.2582	0.8181
Comparison	21	0.8571	0.6905	0.4095	0.9048

Comparison questions had better retrieval performance in this evaluation.

However, I also found that comparison questions had much worse generation performance, showing that retrieval and generation can behave differently for different types of questions.

This is why evaluating only the final answer is not enough. Looking separately at retrieval and generation helps us understand where the RAG system is failing.

I also compared two retrieval approaches:

• BM25: traditional word-based retrieval
• FAISS + dense embeddings: semantic retrieval

BM25 (Best Matching 25) is a keyword-based method for ranking documents by how relevant they are to a search query.

It considers:

• Word overlap: How many query words appear in a document.
• Word importance: Rare words contribute more than common words.
• Term frequency and document length: It accounts for how often words appear and how long the document is.

FAISS, on the other hand, searches for documents whose embedding vectors are most similar to the question's vector, using semantic similarity rather than just word overlap.

Example: If you ask “Where was the actor born?”, BM25 favors documents containing those exact words. FAISS may also find a document saying “The actor's birthplace was London” because the meaning is similar.

In the validation experiment, dense retrieval performed better:

Retriever	Recall@2	Recall@5	Recall@10
BM25	70.22%	81.65%	87.68%
FAISS	82.59%	89.99%	93.44%

Finally, I evaluated the complete baseline and obtained:

Retrieval:  
Precision@1 = 0.7800  
Precision@2 = 0.5550  
Precision@5 = 0.2900  
MRR = 0.8363

Generation:  
EM = 0.2100  
F1 = 0.3017

The main lesson from this baseline is that a RAG system has two separate problems to solve:

1. Find the right information.
2. Use that information to produce the right answer.

A system can perform well at retrieval while still struggling with generation, so both parts need to be evaluated separately.