DSPy: program the pipeline, compile the prompts (stop hand-tuning)

The Stack

Steps

1. 1

   Write the DSPy program and structure-check it

   Write program.py: a Signature (the I/O contract), a Module (the pipeline), a metric function, and an optimizer. CI compiles it and asserts all four are present. Running the optimizer needs dspy, a key, and examples, so that step is fenced.

   cat > program.py <<'PY'
   import dspy
   
   
   class QA(dspy.Signature):
       "Answer the question concisely."
       question = dspy.InputField()
       answer = dspy.OutputField()
   
   
   class Pipeline(dspy.Module):
       def __init__(self):
           super().__init__()
           self.gen = dspy.ChainOfThought(QA)
   
       def forward(self, question):
           return self.gen(question=question)
   
   
   def metric(example, pred, trace=None):
       return example.answer.lower() in pred.answer.lower()
   
   
   optimizer = dspy.BootstrapFewShot(metric=metric)
   PY
   python3 - <<'CHECK'
   import py_compile, sys
   src = open("program.py").read()
   try:
       py_compile.compile("program.py", doraise=True)
   except py_compile.PyCompileError:
       print("BAD: program.py does not compile"); sys.exit(1)
   def need(tok, msg):
       if tok not in src:
           print("BAD: " + msg); sys.exit(1)
   need("dspy.Signature", "no DSPy Signature")
   need("dspy.Module", "no DSPy Module")
   need("def metric", "no metric function for the optimizer")
   if "Bootstrap" not in src and "MIPRO" not in src and "optimizer" not in src:
       print("BAD: no optimizer (the prompts are not being compiled)"); sys.exit(1)
   print("config OK: DSPy program compiles with a Signature, a Module, a metric, and an optimizer (prompts compiled, not hand-tuned)")
   CHECK

2. 2

   Compile / optimize (the model step, not checked by CI)

   pip install dspy, supply a model key and a handful of labeled examples, and run the optimizer. It searches prompts and few-shot demos against your metric. This costs tokens and is fenced.