CrewAI 1.15.28 Makes Evaluation Failures Safer to Hand to Agents

CrewAI 1.15.28 changes what an unattended evaluator says when one model setup stops: it now emits a bounded, location-aware diagnostic while the rest of the comparison continues. The useful part is not just earlier failure reporting. It is the explicit split between service-authored structure and run-authored text.

What changed

crewai eval --models polls CrewAI AMP while several model setups run. In 1.15.28, a poll response may include a stopped list. The open-source CLI tracks each stop by key and announces it once. If AMP knows the location, the non-interactive output names the path through step, crew, task, and agent; if it does not, the CLI waits up to 20 seconds for that detail, or flushes what it has when the comparison ends. Other setups continue.

The output differs by audience. A person at a terminal gets a short red line. A non-TTY consumer—such as a coding agent—gets a Markdown block with the model, run, case, failure location, plain-language meaning, and a reminder that the comparison is still running.

That distinction matters because evaluator failures are themselves an input channel. The service's structural fields decide where the failure occurred and how the CLI behaves. Text originating in the evaluated run is accepted only from the nested reported object, labelled as data rather than instructions, folded to one line, stripped of control-character effects, and quoted. Names are length-capped and backticks are replaced before they appear in code spans.

Direct verification

I checked out the 1.15.28 tag target (eb15c66cef12073d8d06d6942eecc35e7fd6c783) and inspected the stop parser and announcer in eval_crew.py. The code:

• ignores a non-list stopped value and malformed entries;
• requires location data to be a list before using it;
• reads run-provided reason and error only from reported;
• deduplicates announcements by stop key;
• delays an unplaced stop but emits it at the deadline or final result.

On Windows with Python 3.13, I ran the focused release-tag test file after installing the repository's CLI and core packages in an isolated environment:

189 passed in 16.23s

The suite includes non-TTY and terminal rendering, malformed location shapes, reading run text only from reported, crafted text remaining inside its quote, delayed and final announcements, and older-server behavior.

Why it matters

Agent-facing observability needs two properties at once: enough structure for another agent to act on, and provenance boundaries strong enough that a failed run cannot turn its error text into control input. CrewAI's change is a compact example of both. It turns a late, ambiguous comparison failure into a live diagnostic without granting the diagnostic's least-trusted text authority over the consumer.

There is also a useful compatibility choice: older AMP responses that omit stopped keep the previous behavior rather than being guessed into the new schema.

Caveats

This is not a fully local evaluation path. The comparison and its enriched stop record depend on CrewAI AMP, so the open CLI alone cannot produce the new location data. I did not run a live AMP comparison, trigger an actual provider content-filter refusal, or reproduce the network timing path. The observed result is the focused local test suite and source inspection at the release tag; the operational claims about AMP payloads come from CrewAI's release and merged change description.