CrewAI 1.15.28 Turns Stopped Agent Runs Into Diagnosable Evaluations

The change

CrewAI 1.15.28, released on 14 October 2026, adds an eval path aimed at a failure mode that ordinary success metrics hide: a run that stops before reaching a verdict. The release notes describe crewai eval as reporting where a stopped model broke, alongside a new --models comparison mode.

This is more than a new score display. In the tagged source, the command reads .crewai/last_run.json (or accepts an execution ID), submits the traced execution to CrewAI AMP, waits for a terminal evaluation state, and prints the returned verdict. The implementation defines a closed set of stop reasons—such as untraced, network error, authorization failure, or evaluation failure—and records which reason ended an attempted evaluation. The same module documents model comparison as running the deployed project once per candidate model and reporting four grades plus cost and time when available.

What I verified

I inspected the immutable 1.15.28 tag rather than relying on a package index. The tag contains lib/cli/src/crewai_cli/experimental/eval_crew.py, lib/cli/src/crewai_cli/evaluate_crew.py, and a dedicated lib/cli/tests/experimental/test_eval_crew.py. The test fixture scripts queued and running responses followed by a terminal verdict, checks that the evaluation URL is opened, and asserts that the printed output preserves the evaluator’s named areas and stop conditions. The source also confirms that the last-run record is local state and that the evaluation service is AMP; this is not a local/offline evaluator.

I did not execute a real evaluation: that would require a CrewAI project with a traced run and AMP authentication, and would send a run to the service. No performance or accuracy result is claimed here.

Why it matters

For dependable agents, “the model stopped” is an operational event, not a single quality score. A reason taxonomy makes the gap between no trace, transport/authentication failure, and a completed but failed evaluation visible. That can improve triage and reproducibility, provided teams retain the trace, project identity, model configuration, and evaluator response alongside the verdict.

There are two important boundaries. First, the release’s eval path is service-backed and authenticated for model comparisons; it should not be confused with a self-contained regression harness. Second, the source and tests show the plumbing and expected behavior, not evidence that the evaluator’s four grades predict task success. That still needs independent, repeatable validation on representative workloads.