Education Idol

A Complete Step-by-Step Guide to Building a Basic RAG System

This document provides a complete, step-by-step guide to manually building your own local Retrieval-Augmented Generation (RAG) chatbot. By leveraging Python, LangChain, ChromaDB, and Ollama, you can create a tool that chats with your PDFs privately—without sending your sensitive data to third-party APIs like OpenAI.

Here is exactly how to set it up from scratch.

Step 1: System Prerequisites for Your Local RAG Chatbot

Before writing any code, ensure you have the core tools installed on your computer.

Install Python (Version 3.9 to 3.12 Recommended)

Download and install Python from python.org.

• Important Note: As of right now, Python 3.13 has some compatibility issues with certain machine learning libraries like Numpy. To avoid frustrating errors later, Python 3.12 is the safest bet.

Download and Set Up Ollama (LLaMA 3.2 Model)

Download and install Ollama from ollama.com. Once installed, open your terminal (Command Prompt or PowerShell) and run the following command:

ollama run llama3.2

(This might take a few minutes as it downloads a few gigabytes of model data to your local machine).

[Insert Screenshot Here: Terminal showing the successful download of the LLaMA 3.2 model] Alt Text: Terminal window showing the successful local download of the LLaMA 3.2 model via Ollama.

Step 2: Python Project Setup & Dependency Installation

Create a Project Folder & Virtual Environment

Create a new folder anywhere on your computer (e.g., pdf-chatbot) and open your terminal directly inside that folder.

It is highly recommended to create a virtual environment. This keeps your project dependencies isolated and prevents conflicts with other Python projects. Run these commands:

powershell

python -m venv venv
.\venv\Scripts\activate

Install Required Python Packages (LangChain, ChromaDB, etc.)

With your virtual environment active, run the following pip command to install all the specific libraries required for this RAG pipeline:

powershell

pip install langchain==0.3.4 langchain-core==0.3.15 langchain-community==0.3.5 langchain-chroma==0.1.4 langchain-text-splitters==0.3.2 pypdf sentence-transformers chromadb

Prepare Your Local PDF Data Folder

Create a folder named data inside your main pdf-chatbot project folder. Place any PDF you want to chat with inside this data folder.

Step 3: Writing the Python RAG Code (Ingestion & Chat)

To keep the project organized, you will need to create two separate Python files in your main project folder.

1. Create the PDF Ingestion Script (ingest.py)

This script takes your PDF, splits it into small paragraphs (chunks), converts those chunks into numbers (embeddings), and saves them in a local vector database (ChromaDB).

Create a file named ingest.py and paste the following code:

python

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_chroma import Chroma

def ingest():
    # 1. Point to your PDF file
    pdf_path = "data/your_file.pdf" # MAKE SURE TO CHANGE THIS TO YOUR PDF'S NAME!
    print(f"Loading {pdf_path}...")
    
    # 2. Read the PDF
    loader = PyPDFLoader(pdf_path)
    docs = loader.load()
    
    # 3. Split the text into chunks
    print("Splitting document into chunks...")
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
    chunks = text_splitter.split_documents(docs)
    
    # 4. Initialize the embedding model (converts text to vectors)
    print("Downloading/Loading embedding model...")
    embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
    
    # 5. Save everything to ChromaDB
    print("Saving to local Vector Database...")
    db_path = "./chroma_db"
    vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory=db_path)
    print(f"Done! Vector DB created at {db_path}.")

if __name__ == "__main__":
    ingest()

2. Create the Chat Interface Script (chat.py)

This script loads the database you just created, connects to your local Ollama LLM, and provides the interactive chat interface.

Create a file named chat.py and paste the following code:

python

import warnings
warnings.filterwarnings("ignore") # Hides annoying deprecation warnings

from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_chroma import Chroma
from langchain_community.llms import Ollama
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate

def chat():
    print("Initializing embedding model...")
    embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
    
    # 1. Load the database we created in step 1
    db_path = "./chroma_db"
    vectorstore = Chroma(persist_directory=db_path, embedding_function=embeddings)
    
    # Configure it to retrieve the 3 most relevant paragraphs
    retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
    
    # 2. Connect to local Ollama
    print("Connecting to local LLaMA model...")
    llm = Ollama(model="llama3.2")
    
    # 3. Create the prompt instructions
    system_prompt = (
        "You are an assistant for question-answering tasks. "
        "Use the following pieces of retrieved context to answer the question. "
        "If you don't know the answer, say that you don't know. "
        "Use three sentences maximum and keep the answer concise.\n\n"
        "Context:\n{context}"
    )
    prompt = ChatPromptTemplate.from_messages([
        ("system", system_prompt),
        ("human", "{input}"),
    ])
    
    # 4. Tie it all together into a LangChain pipeline
    question_answer_chain = create_stuff_documents_chain(llm, prompt)
    rag_chain = create_retrieval_chain(retriever, question_answer_chain)
    
    print("\n--- PDF Chatbot Ready! ---")
    print("Type 'exit' to quit.\n")
    
    # 5. The Chat Loop
    while True:
        query = input("You: ")
        if query.lower() in ['exit', 'quit']:
            break
            
        print("Bot: Thinking...")
        try:
            # Send question to pipeline
            response = rag_chain.invoke({"input": query})
            print(f"\nBot: {response['answer']}\n")
        except Exception as e:
            print(f"Error: {e}\n")

if __name__ == "__main__":
    chat()

Step 4: How to Run Your Local PDF Chatbot

Run the Ingestion Script to Build the Vector Database

Double-check that your PDF is in the data/ folder and that you have updated the pdf_path filename inside ingest.py.

Open your terminal and run the ingestion script:

powershell

python ingest.py

Note: The very first time you run this, it will download a small (~80MB) embedding model from Hugging Face. Wait for the script to finish completely and print “Done!”.

[Insert Screenshot Here: Terminal output showing the successful creation of the ChromaDB vector database] Alt Text: Command prompt output showing the Python RAG ingestion script successfully saving PDF chunks to a local ChromaDB.

Launch the Chat Script and Start Querying Your PDF

Once the vector database is created, you only need to run the chat script to start talking to your document:

powershell

python chat.py

That’s it! Start chatting with your document. Because of the RAG architecture, the bot will strictly use the information retrieved from your provided PDF to formulate its answers, keeping your workflow entirely local and private. Type exit to close the chatbot when you are done.