How it’s built
A Customer Agent you can rehearse: personalized from Klaviyo, grounded in sources you control, and measured by a judge you can calibrate.
Architecture
Local MCP server over stdio (read-only) for profiles, events and catalog. REST with a separate write key for events and coupon codes.
Pydantic AI agent with typed, shopper-scoped tools and a validated structured reply.
Chunked sources in Postgres. Full-text and pgvector search fused with reciprocal rank fusion.
Persona agents built from real profiles, a Postgres-backed job queue and concurrent workers.
Sonnet with five binary criteria, calibrated against human labels with Cohen’s κ.
Every rehearsal is an experiment; every agent run is a trace; human labels return as feedback.
The loop: a shopper’s Klaviyo profile and order events become a scenario → the persona chats with the agent through tools that can only read that shopper → the judge scores the transcript against the evidence → failures cluster into gaps → a fix reruns only the affected conversations → Braintrust holds the comparison.
Decisions, and what building it taught
No tool accepts a customer id. The signed-in profile is injected from the run context, so a prompt injection has nothing to point at. A scenario tries it on purpose.
The first rehearsals reported eight “hallucinations”. Most were the judge being shown tool names instead of tool results, so correctly quoted sale prices looked invented. Giving it the full results removed those false failures.
The model was seen to satisfy a “condition not confirmed” guard by flipping a flag and retrying in the same turn. Returns and exchanges now require the shopper’s own words from a later message, checked in code.
Coupon caps by loyalty tier, the 30-day return window, final-sale items and a three-searches-per-turn cap are all enforced in code. A prompt-only search limit did not stop a nine-search loop.
Failures cluster into knowledge gaps (add a source) and behavior gaps (edit guidance). Fixing one reruns only the conversations that exposed it, linked to its parent run for comparison.
Label conversations yourself and compare. Edit the judge and re-judge the same evidence to see whether agreement improves.
Stack
| Agent framework | Pydantic AI 2 (typed deps, tools, output validators, structured output) |
| Models | Claude Haiku 4.5 for the agent and shoppers, Claude Sonnet 5.5 for the judge |
| Evals | Braintrust experiments, traces, datasets and feedback |
| Klaviyo | klaviyo-mcp-server (stdio, read-only) and the REST API |
| Backend | FastAPI, SQLAlchemy, Postgres with pgvector (SQLite locally), SSE |
| Frontend | Next.js 15, React 19, Tailwind 4 |
| Guardrails | Per-IP and per-day limits, spend caps, SSRF-safe URL fetching, admin-gated edits |
Honest limits
- Simulated shoppers are language models playing a script; they find real failures but are not real customers.
- Each scenario runs once per rehearsal, so outcomes vary run to run. A production setup would run several trials and report intervals.
- The judge is an LLM. That is why the calibration tab exists.
- Without billing, the embedding provider allows 3 requests a minute, so vectors index in the background and queries fall back to keyword search when no slot is free.
- Demo data only. This is an independent project, not affiliated with or endorsed by Klaviyo.