Skip to content
← back to projects
2026·shipped·Forward Deployed Engineer

Tenacious-Bench: Sales Agent Evaluation & LoRA Fine-Tuning

A 200-task benchmark built through multi-LLM synthesis with judge filtering. Trained LoRA adapters with SFT, DPO, and a process-reward model to target a specific failure mode.

PythonLoRASFTDPOProcess Reward ModelHuggingFace
Code ↗Link ↗

Pipeline

1
Synthesis
Multi-LLM task generation
↓
2
Judge filtering
Top decile survives
↓
3
Human sign-off
Hand-annotated
↓
4
SFT + DPO
LoRA training
↓
5
Process reward model
Scores reasoning steps

Highlights

  • ▸200-task benchmark synthesized by multiple LLMs and filtered through a judge model for quality.
  • ▸LoRA adapters trained with SFT + DPO + a process-reward model.
  • ▸Targeted a specific failure mode found in the prior week's Conversion Engine.
  • ▸Published dataset, datasheet, and model card on HuggingFace.

The full loop

Week eight found a specific failure mode in the Conversion Engine: the sales agent kept generating confident-sounding outreach that ignored a disqualifying signal. Week nine was to build a benchmark that measures that failure, then train an adapter that fixes it.

Data pipeline

  1. Synthesis: multiple LLMs generate candidate tasks under a structured schema.
  2. Judge filtering: a separate model rates each task; only the top decile survives.
  3. Human sign-off: I read every retained task and hand-annotated the reference answer.

Training

  • SFT on the annotated set, teaching the model the shape of a correct answer.
  • DPO on paired preference data, teaching it which of two candidate answers is better.
  • Process reward model, a lightweight critic that scores intermediate reasoning steps, not just final answers.

Publishing

Dataset, datasheet, and model card live on HuggingFace so the failure mode and its fix are both reproducible: the dataset and the trained judge model.

Trade-offs

A process-reward model is overkill if you just need a pass/fail eval — SFT + DPO alone would have caught the disqualifying-signal failure mode; the PRM earns its complexity only because I wanted visibility into which reasoning step went wrong, not just the final verdict. For a benchmark this size (200 tasks), the judge-filtering step is also doing a lot of the real work; if I rebuilt this for a larger corpus, I’d invest more in inter-rater agreement between judges before trusting a single judge’s top-decile cut.

What I would change now

The benchmark is tightly coupled to the Conversion Engine’s sales domain. If I rebuilt it, I’d extract the synthesis-judge-train pipeline into a reusable harness that takes a domain config and a failure-mode specification, so the same loop could generate benchmarks for other agent tasks without rewriting the scaffolding each time.