DSPy Programming Tutorial: From Signatures to RAG in Python

Overview

DSPy is a programming model and toolkit for building reliable, modular systems on top of large language models (LLMs). Instead of hand-writing brittle prompts, you declare what each step should input and output (signatures), compose those steps as modules, and let DSPy optimize the prompts and few-shot demonstrations automatically. The result is code you can test, version, and ship.

This tutorial walks through core concepts and a practical workflow—from “hello world” to a small retrieval-augmented generation (RAG) pipeline—plus tips for evaluation and production.

Why DSPy instead of ad‑hoc prompting?

• Declarative: You describe the interface (inputs/outputs), not the exact words of the prompt.
• Modular: Reuse building blocks like Predict, ChainOfThought, and Retrieve.
• Optimizable: Given a small labeled dev set, DSPy “teleprompters” can learn better prompts and examples automatically.
• Testable: Programs are regular Python; you can unit test, evaluate, and iterate as with any ML system.

Installation and setup

1. Install the library

pip install dspy-ai

2. Configure your LLM provider

DSPy supports multiple backends. Here are two common setups; use whichever you have credentials for.

import dspy

# OpenAI example (set OPENAI_API_KEY in your environment)
lm = dspy.OpenAI(model='gpt-4o-mini', temperature=0.2, max_tokens=512)

# Or, a local/Ollama/HF-style client (example; adjust to your runtime)
# lm = dspy.HFClient(model='meta-llama/Llama-3-8B-Instruct', max_tokens=512, temperature=0.2)

# Configure global settings
dspy.settings.configure(lm=lm, trace=False)

Tip: keep temperature low (0.0–0.3) for deterministic, evaluable outputs during development.

Key DSPy concepts at a glance

• Signature: A typed schema of a task’s inputs and outputs.
• Module: A reusable component (e.g., Predict, ChainOfThought, Retrieve) wired together into a program.
• Teleprompter/Optimizer: An algorithm that learns effective prompts and demonstrations from a small training/dev set (e.g., BootstrapFewShot).
• Example: A labeled datum used for optimization or evaluation.

Hello, World: a minimal QA module

We’ll build a question–answering module that maps a question to a short answer.

import dspy

# 1) Define a signature: what the model sees and must produce
class AnswerQuestion(dspy.Signature):
    """Answer short factual questions concisely."""
    question = dspy.InputField(desc="A short, factual question.")
    answer = dspy.OutputField(desc="A concise, correct answer.")

# 2) Wrap it in a module
class BasicQA(dspy.Module):
    def __init__(self):
        super().__init__()
        self.predict = dspy.Predict(AnswerQuestion)

    def forward(self, question: str):
        pred = self.predict(question=question)
        return pred  # pred.answer holds the string answer

# 3) Use it
qa = BasicQA()
print(qa("Who wrote The Hobbit?").answer)

Because the signature states the intent, DSPy builds a reasonable prompt automatically. You can later optimize it with a teleprompter.

Building a tiny dev set and a metric

Optimizers need a handful of labeled examples and a metric. We’ll use exact match (EM) for simplicity.

# Dev/train examples (tiny for the tutorial; start with ~20–50 in practice)
trainset = [
    dspy.Example(question="Who wrote The Hobbit?", answer="J. R. R. Tolkien").with_inputs("question"),
    dspy.Example(question="What is the capital of Japan?", answer="Tokyo").with_inputs("question"),
    dspy.Example(question="In what year did Apollo 11 land on the Moon?", answer="1969").with_inputs("question"),
]

# A simple exact-match metric

def exact_match(example, pred, trace=None):
    gold = example.answer.strip().lower()
    got = getattr(pred, 'answer', '').strip().lower()
    return float(gold == got)

Note the with_inputs(“question”): it marks which fields are inputs versus labels.

Prompt optimization with a teleprompter

Now let DSPy search for effective few-shot demonstrations and prompts.

from dspy.optimize import BootstrapFewShot

teleprompter = BootstrapFewShot(
    metric=exact_match,
    max_bootstrapped_demos=6,  # how many demos it can synthesize
    num_trials=30,             # how many prompt/demo variants to try
)

compiled_qa = teleprompter.compile(
    module=BasicQA(),
    trainset=trainset,
)

# Evaluate quickly on the same set (use a held-out set in practice)
scores = [exact_match(ex, compiled_qa(question=ex.question)) for ex in trainset]
print(f"EM on trainset: {sum(scores)/len(scores):.2f}")

The compiled module is still a drop-in BasicQA, but it carries learned demonstrations/prompts internally.

Adding chain-of-thought for multi-step reasoning

When reasoning benefits from intermediate steps, swap Predict for ChainOfThought. The signature remains declarative.

class ReasonSignature(dspy.Signature):
    """Answer the question with brief reasoning then a final answer."""
    question = dspy.InputField()
    answer = dspy.OutputField(desc="Final answer only; be concise.")

class ReasoningQA(dspy.Module):
    def __init__(self):
        super().__init__()
        # CoT encourages structured intermediate thinking under the hood
        self.solve = dspy.ChainOfThought(ReasonSignature)

    def forward(self, question: str):
        return self.solve(question=question)

rq = ReasoningQA()
print(rq("If a train leaves at 3 PM and arrives 2 hours later, what time does it arrive?").answer)

You can optimize ReasoningQA the same way with BootstrapFewShot.

Retrieval-Augmented Generation (RAG) in DSPy

LLMs hallucinate when they lack context. With DSPy, wire a retriever to fetch passages, then answer using those passages.

# 1) Configure a retriever (example: ColBERT server; substitute your retriever)
# Start/point to your retriever service before running this.
retriever = dspy.ColBERTv2(url="http://localhost:2017")  # adjust to your setup

dspy.settings.configure(lm=lm, retriever=retriever)

# 2) Define a RAG module
class RAGQA(dspy.Module):
    def __init__(self, k=4):
        super().__init__()
        self.retrieve = dspy.Retrieve(k=k)
        self.answer = dspy.ChainOfThought("question, context -> answer")

    def forward(self, question: str):
        ret = self.retrieve(question)
        # ret.passages may be a list of strings or objects depending on retriever
        context = "\n\n".join(map(str, getattr(ret, 'passages', [])))
        pred = self.answer(question=question, context=context)
        return pred  # pred.answer is the final string

rag = RAGQA(k=4)
print(rag("What is the PageRank paper's publication year?").answer)

To optimize the whole pipeline, simply pass RAGQA into the teleprompter with a small labeled set of (question, answer) pairs.

rag_teleprompter = BootstrapFewShot(metric=exact_match, max_bootstrapped_demos=6, num_trials=30)
compiled_rag = rag_teleprompter.compile(module=RAGQA(k=4), trainset=trainset)

Behind the scenes, DSPy learns effective demonstrations/prompts that teach the module how to use retrieved passages.

Evaluating your program

For quick experiments, loop through a dev set with your metric. For larger runs, parallelize or log traces.

def evaluate(module, devset):
    total = 0.0
    for ex in devset:
        pred = module(question=ex.question)
        total += exact_match(ex, pred)
    return total / max(len(devset), 1)

heldout = [
    dspy.Example(question="Who painted Starry Night?", answer="Vincent van Gogh").with_inputs("question"),
]

print(f"Held-out EM: {evaluate(compiled_rag, heldout):.2f}")

Tips for robust evaluation:

• Keep temperature low for scoring; raise it later if you want diverse outputs.
• Prefer task-specific metrics (F1 for span extraction, BLEU/ROUGE for generation, or custom validators).
• Store predictions and traces for error analysis.

Controlling outputs with stricter signatures

Signatures can steer formatting and content. You can encode style and constraints in descriptions.

class JSONIntent(dspy.Signature):
    """Extract a user intent as a compact JSON object with fields: intent, entities."""
    utterance = dspy.InputField(desc="A single user message.")
    answer = dspy.OutputField(desc="A minified JSON object with keys: intent, entities (list). No prose.")

class IntentExtractor(dspy.Module):
    def __init__(self):
        super().__init__()
        self.predict = dspy.Predict(JSONIntent)
    def forward(self, utterance: str):
        return self.predict(utterance=utterance)

ie = IntentExtractor()
print(ie("Book a flight from NYC to LA tomorrow").answer)

If outputs drift, optimize with a small labeled set and a stricter metric that validates JSON structure.

Debugging and transparency

• Tracing: set trace=True in dspy.settings.configure to inspect prompts, intermediate predictions, and retrievals.
• Narrow the search: reduce num_trials during prototyping to save time/cost, then scale up.
• Seed and reproducibility: keep temperature low and control seeds in your backend where possible.

Practical patterns you can compose

• Multi-hop QA: Use Retrieve twice with different questions, or stack modules where the first produces sub-questions.
• Routing: A light classifier module chooses between two downstream solvers (e.g., CoT vs. direct Predict).
• Tool use: Wrap a function call (e.g., calculator, SQL) as a module and feed its outputs to a reasoning step.

Cost and latency tips

• Smaller models first: optimize prompts on a smaller, cheaper model, then validate on your target model.
• Cache: memoize inputs/outputs during development to avoid repeated calls.
• Retrieve narrowly: tune k and index quality; fewer, better passages reduce tokens and distractions.

From notebook to production

• Version control your modules and the small dev/validation sets you use for optimization.
• Persist artifacts: keep the compiled module parameters (e.g., selected demos/prompts) alongside code.
• Guardrails: define stricter signatures and metrics; run canary evaluations before deploying updates.

What to try next

• Swap BootstrapFewShot for other teleprompters (e.g., search-based or iterative improvement techniques) and compare results.
• Enforce output schemas more strictly using JSON-oriented signatures and validators.
• Add a reranker between Retrieve and Answer for higher-quality contexts.

Recap

DSPy lets you program LLM systems with clear interfaces, reusable modules, and automated prompt/demo optimization. Start with small, labeled dev sets; wire modules for reasoning and retrieval; optimize and evaluate; then iterate. The result is not just a prompt—it’s a testable, maintainable program you can confidently move toward production.