Suppose you move a customer service agent to a new framework. It responds faster, but does it still call the right tool before answering? If the migration also means rebuilding the tests, it can be hard to tell whether the agent changed or the measurement did.

On November 11, AWS introduced the Universal Agent Evaluation Framework (UAEF), an open-source Python library under the Apache 2.0 license. It lets teams apply a common set of criteria to agents built with tools such as LangGraph, LangChain and Strands Agents by translating their execution records into a shared format.

From execution records to scores

An adapter collects messages, tool calls and handoffs between agents in a record called an AgentTrace. The library can import traces you already have or invoke an agent at an endpoint to collect fresh runs. The evaluator works from the normalized record in either case.

AWS describes 36 metrics across seven dimensions. A team can check which tool the agent chose, what arguments it passed and how well it answered the user. Some metrics score the trace deterministically against expected results. Others use a model on Amazon Bedrock to judge criteria such as relevance. Individual metrics can surface regressions that an average hides. An agent might write a better response while becoming less reliable at retrieving the underlying data.

Comparing two versions of an agent

Start with the same requests, expected results, metrics and evaluator settings before and after a change. With a customer service agent, check whether both versions find the right order and pass valid arguments to the lookup tool. If a model judges part of the evaluation, review a sample of cases where scores moved. What those scores mean depends on the events each adapter captures and the criteria you give the evaluator.

The AWS Labs repository includes the library, examples and infrastructure for an optional deployment. Enabling experiment persistence as described by AWS uses DynamoDB and S3. Because the traces can include messages and tool outputs, decide what can be retained before testing with real requests. AWS says 12 organizations have used UAEF. For your own project, a useful first result is being able to explain a change between two versions using the same tests.

Sources. AWS technical announcement and UAEF repository.