Agent evaluation tooling is shipping fast, and the message is clear: production quality depends on your evaluation pipeline, not just your prompt. This week brought specialized evaluators, live benchmarking competitions, and CI/CD-style preview environments for agents.
Introducing LangSmith Tuned Evaluators, starting with Perceived Error
LangChain is betting that custom-trained evaluator models will replace frontier-model-as-judge for production teams. Their Perceived Error evaluator matches GPT-4 level accuracy while cutting evaluation costs by 82-98%, making it economically viable to evaluate every production trace instead of sampling. The product ships with no configuration required, which matters when teams want coverage from day one rather than spending weeks building custom eval harnesses.
// Evaluate all production traces, not samples
const evalResult = await langsmith.evaluators.perceivedError({
traceId: trace.id,
// No custom rubrics or few-shot examples needed
});
if (evalResult.hasError) {
await alerting.notify({
severity: 'medium',
context: evalResult.explanation
});
}Designing effective Genie Agents from a single prompt
Databricks is clear about where agent quality comes from: governed data, not clever prompts. Genie Agents pull context directly from Unity Catalog (structured tables, PDFs, internal docs), and the guidance is to curate those sources aggressively before worrying about prompt tweaks. The best practice workflow is focused: pick one use case, establish benchmark questions, validate against known-good answers, then expand scope only after the foundation proves reliable.
Evaluating AI Agents Live at the Grounded Reasoning Cup
Stanford won this live agent competition with 63.3% accuracy by treating the entire pipeline as the unit of optimization, not just the LLM. Baseline frontier models hit under 30% on the same task when dropped in without tuning. The competition setup is worth noting: teams trained on OfficeQA, then competed on unseen Treasury documents to test generalization. The takeaway is that parsing quality, retrieval strategies, reusable skill libraries, and verification layers matter more than model selection, and performance on familiar benchmarks doesn't guarantee transfer to new domains.
LangSmith Preview Builds: Test agent changes before production
Preview Builds bring the Vercel deployment model to agent development. Each GitHub PR gets an isolated staging environment that non-technical reviewers can interact with before merge, complete with trace inspection and auto-cleanup after configurable idle timeouts. This shifts agent validation left in the development cycle and gives product teams a way to sign off on behavior changes without running code locally.
# .github/workflows/agent-preview.yml
on:
pull_request:
types: [labeled]
jobs:
deploy-preview:
if: contains(github.event.pull_request.labels.*.name, 'preview')
runs-on: ubuntu-latest
steps:
- uses: langchain/preview-build-action@v1
with:
idle-timeout: 2h
max-concurrent: 3The pattern is consistent: evaluation infrastructure is moving from post-deployment sampling to comprehensive pre-production coverage. Teams that treat eval pipelines as first-class CI/CD artifacts are shipping higher-quality agents with predictable costs. 🚀