This analysis organizes the evidence behind AI Agent Evaluation Is Becoming a Product Category, then explains the practical implications, trade-offs and current limits.
Read the evidence below as a decision trail: what changed, why it matters, which trade-offs shaped the result, and where the conclusion still depends on context.
Abstract: Within a week, AWS released two AI agent evaluation tools—Strands Evals, which specializes in diagnosing why the agent failed, and Agent-EvalKit, which systematically evaluates the entire execution path—coupled with Langfuse, an open source observation tool that continues to become popular, "evaluation agent" is changing from a subsidiary function to an independent tool category. When there are so many agents that special tools are needed to detect when and why they fail. How do these three work divide the work and how do they choose?
- AWS launched the open source Agent-EvalKit on June 11, 2026, and Strands Evals on June 15, which just complemented both "systematic assessment" and "fault root cause diagnosis."
- Strands Evals compares execution traces against nine major fault categories and uses a two-stage process to separate root causes and downstream symptoms, claiming to reduce diagnosis time from hours to minutes.
- Agent-EvalKit evaluates the entire execution path instead of just looking at the final output to check whether the agent actually responds based on the data returned by the tool.
【In-depth dismantling】
Strands Evals: Automate “why you fail”
Strands Evals is an SDK for failure detection and root cause analysis. It compares the agent's execution trajectory with a set of nine major fault classifications: hallucinations, incorrect actions, choreography errors, failure to comply with task instructions, execution errors, context processing errors, repetitive behaviors, LLM output problems, and setting inconsistencies.
Its detection process is divided into two stages, both driven by LLM: the first stage marks the fault, gives confidence scores and supporting evidence; the second stage determines the causal relationship (divided into PRIMARY/SECONDARY/TERTIARY), separates the root cause from downstream symptoms, and outputs correction suggestions - for example, classified as "the system prompt should be changed" or "the tool description should be changed". It is compatible with OpenTelemetry tracing, can connect to CloudWatch Logs, and can also accept traces generated by frameworks such as Langchain. According to AWS, this can reduce the diagnostic time that originally relied on manual inspection of traces one by one from hours to minutes.
Agent-EvalKit: Evaluate the entire path, not just the final answer
Agent-EvalKit is an open source (Apache 2.0) tool whose core proposition is that "evaluating an agent cannot just look at the final output." What it wants to check is whether the agent really answered based on the data returned by the tool, and whether it correctly selected and introduced the tool parameters, instead of answering beautifully but actually skipping verification or even hallucinating.
The usage is very special - it uses slash command operation through the existing coding assistant (Claude Code, Kiro CLI, Kilo Code), and the assistant runs through six sequential stages: Plan, Data, Trace, Run agent, Eval, Report, and the results produced by each stage are fed to the next stage. It supports frameworks such as Strands Agents, LangGraph, and CrewAI, and uses Amazon Bedrock for inference. The calculated indicators include Faithfulness, Tool Parameter Accuracy, and Response Quality. Finally, it provides priority correction suggestions corresponding to specific code locations.
Langfuse supplements the "continuous observation" aspect
Langfuse is an open source LLM observability platform that performs trace, evaluation, and prompt management. It has approximately 29.2k stars on GitHub. If the first two tools are more focused on evaluation and diagnosis during the "development and testing period", Langfuse is more focused on "long-term tracking after launch." The three together form a line: Agent-EvalKit for systematic evaluation, Strands Evals for digging into root causes of faults, and Langfuse for continuous observation of online behavior.
【My point of view】
I think the fact that "agent evaluation" has become an independent tool category is a clear signal that agentic AI is moving from demo to production. There are many people who can do demos, but there are far fewer people who can clearly explain "why my agent failed and at which step it failed" - and the latter is the real threshold for production. Tools are beginning to grow in this direction, which means that the pain point of the market has shifted from "can be done" to "reliable or not."
When it comes to choosing tools, my judgment is not to do it all at once. First use tools such as Langfuse to connect the traces. Without traces, all subsequent diagnoses are blind; add Agent-EvalKit when systematic evaluation is needed; wait until there are too many faults to be checked manually, and then use Strands Evals for root cause analysis. It should be noted that both Strands Evals and Agent-EvalKit are deeply tied to the AWS/Bedrock ecosystem. Non-AWS users must first think about the lock-in issue before importing.
My pragmatic advice to the Taiwanese team is: treat "agent evaluation" as an engineering discipline at the same level as unit testing, with tools second. Without the evaluation process, no matter which tool you change, it won't save you; with the process, the tool will just automate it.
[Core Insight] When there are too many agents, the question changes from "can you do it?" to "why did you do it wrong?"
The most painful thing after an agent goes into production is usually not that it can't run, but that it runs wrong but doesn't know which step it is. AWS's two releases this week are exactly aimed at this pain point: Agent-EvalKit on June 11 for systematic evaluation, and Strands Evals on June 15 for fault diagnosis. Coupled with Langfuse, which has been popular in the open source circle, it can be seen that "assessment and observation agent" is growing into an independent category.
【FAQ】
What is the difference between Strands Evals and Agent-EvalKit?
Strands Evals specializes in fault detection and root cause analysis, answering "Why did the agent fail and where is the root cause?"; Agent-EvalKit does a systematic evaluation of the entire execution path, answering "Has the agent faithfully answered based on the tool data?" The two can be used together. Agent-EvalKit itself supports calling Strands Evals for indicator evaluation.
Why you can’t evaluate AI agents by just looking at the final output
The agent may give a good answer, but skip the verification step, choose the wrong tool, or suffer from hallucinations. Looking only at the final result will miss the mistakes in the process, which means you don't know whether the success is really reliable or just good luck. So evaluate the entire tool call path, not just the end point.
Do these tools have to use AWS?
Both Strands Evals and Agent-EvalKit are deeply integrated with Amazon Bedrock, but are compatible with open standards such as OpenTelemetry and Langchain; Langfuse is open source and can be deployed independently. Non-AWS users can still use it, but the first two are the most smoothly integrated within the AWS ecosystem.
Original source:AI Agent Failure Detection and Root Cause Analysis with Strands Evals（AWS）、Evaluate AI agents systematically with Agent-EvalKit（AWS）、langfuse/langfuse（GitHub）
What to take away
The article's value is in the evidence and trade-offs behind AI Agent Evaluation Is Becoming a Product Category, not in treating the conclusion as universal.