AgentTrace
AI Agent Evaluation Tool
Configure any AI agent's persona and policies, run synthetic test conversations, and get back a scored report with the three fixes to make next.
5-dimension scoring
3 ranked fixes
<90s per run
Stack
Python · Streamlit · Anthropic SDK · Pydantic
evaluation verdict
Needs Tuningthe problem
Teams deploying AI agents have no systematic way to know if their agent behaves as designed — before real customers find out it doesn't.
how it works
You define your agent's persona and policies. AgentTrace spins up synthetic test conversations across four categories — helpful queries, edge cases, policy probes, and attempted jailbreaks — then hands each transcript to a separate evaluator that scores it and returns fixes.
the report
The output isn't a dashboard — it's a decision. Every run ends with five dimension scores and three concrete fixes, ranked by severity, so a PM knows exactly what to change before the next deploy.
what i learned
The biggest unlock was splitting the agent and the evaluator into two separate model calls with different system prompts. When one model does both, it grades itself generously and misses tone problems. Two Claude calls produced sharper, more actionable evaluations.
more
key decisions
Two Claude calls, not one
The agent stays in character. The evaluator steps outside and judges it. Self-evaluation collapses both jobs and hides failure modes.
Decisions, not dashboards
The report ends with three ranked fixes, not a wall of metrics. PMs use it to decide what to ship next.
outcomes
- Live demo deployed and used by 5+ early testers
- Full eval cycle in under 90 seconds
- Open-sourced on GitHub
up next
GoalOrch
Production database & ETL layer for a student-analytics platform