AWS open-sourced awslabs/Agent-EvalKit: run evals inside Claude Code, Kiro CLI, or Kilo Code via slash commands like /evalkit.plan and /evalkit.report. Six phases cover plan, data, OpenTelemetry traces, run, score, and code-linked fixes; a travel-research demo scored Response Quality 83.9%, Tool Parameter Accuracy 64.5%, Faithfulness 32.3%—pretty replies still invented temps and FX rates over empty tool results. An AWS ML blog documents the systematic eval approach.

Key Takeaways

  • ✓Open source: awslabs/Agent-EvalKit with slash commands in Claude Code / Kiro / Kilo.
  • ✓Pipeline: plan → data → OTel traces → run → score → code-linked fixes.
  • ✓Insight: demo Faithfulness only 32.3%; empty tool results still hallucinate facts.
⚡

Key Decision Metrics at a Glance

TL;DR Verdict
🏆 Benchmark RankRank
Pipeline: plan → data → OTel traces → run → score → code-linked fixes.
Full Leaderboard→
💰 Cost & PlansCompare
API & Subscription
Compare Plans↗
🛠️ Ecosystem & IDERules
Claude Code (Anthropic)
Setup IDE Rules↗
🧭

Heavy Claude Code use: compare subscription limits and API bills

Compare 40 dev plans & simulate token costs vs $20/mo subscriptions

⚔️ Compare Model→💰 Compare plans & limits↗
Action HubReady to adopt this in production?

Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.

⚔️ Launch VS Compare→💳 View Plans↗
Up NextAWS: Bedrock AgentCore Runtime hosts coding agents with Firecracker microVMs and persistent workspace
Read Now→