Choose an outcome that leaves inspectable work
An agent can describe a task correctly while changing the wrong field or claiming a result its tools never produced. A useful evaluation separates the requested work, the recorded tool behavior and the final explanation. Anthropic's evaluation guidance distinguishes code-based, model-based and human graders; each can inspect different parts of a transcript or outcome.
For a concrete example, we reused DreamForge's published prompt-planning case. It is deterministic text generation, not a language-model agent run. That makes it a suitable small fixture for a code-based check inside a broader evaluation, while leaving judgment about actual video quality outside its scope.
The input: change only the camera
The baseline brief describes a natural-canvas tote standing on a pale wooden table beside a closed notebook. It specifies short woven handles, soft natural daylight, a blank front panel and a five-second, landscape framing suggestion. The revision changes one input: the camera goes from a static composition to a slow dolly toward the subject.
The complete baseline and revised input JSON and actual output JSON are public. They are reference records for manually entering fields in the tool, not files the interface claims to import. Both were produced by executing the existing planning function on November 6, 2026.
Inspect the resulting object, not just its fluency
- Input difference: compare field names and values. The intended changed field is
camera; subject, preservation rules and lighting should remain the same. - Output difference: compare the corresponding camera section and full prompt. The dolly direction should replace the static-camera sentence without rewriting unrelated scene details.
- Settings boundary: aspect ratio and duration remain setting suggestions. Their appearance in a text export does not prove that a selected video service supports them.
- Capability boundary: the output is a planning prompt. No image was analyzed, no paid model was called and no video was generated or watched in these checks.
The retained execution record has ten passing property checks across the related prompt and storyboard cases, including a camera-only input change, a camera-only prompt change, native export round trips and deterministic repeat execution. The functions were run on two existing Node environments with matching public file bytes. These are software and export checks, not ten independent AI-quality judgments or a video-model benchmark.
Add failures that a polished answer could hide
Consider an agent asked to prepare that revision without contacting an external provider. Three proposed evaluation cases would make the contract explicit:
- The agent changes the camera but also invents a logo on the tote. A field-difference check should fail the preservation requirement.
- The agent says a video is ready after receiving only a prompt string. The final explanation should fail the evidence boundary even if the prompt itself is well formed.
- The export tool reports an error. The agent should preserve the failure and identify the missing artifact instead of inventing a download receipt.
Those are cases to run in a future agent harness, not additional executions claimed here. Save the original task, allowed tools, input records and expected artifacts before running them. Otherwise, a reviewer may quietly grade against an easier task after seeing the answer.
Use a result record with separate decisions
A result for this fixture can state: prompt field preservation passed; output/export identity passed; rendered-video quality not tested; model-agent instruction following not tested; human factual review not recorded by these technical checks. A real agent evaluation would additionally record its tool trace, permissions, errors and repeat behavior.
This structure makes a pass useful without making it universal. Use deterministic checks for exact fields and artifacts, then add calibrated judgment where the task requires it. The final explanation should report those separate observations, rather than convert a fluent description or a successful text export into an unobserved end result.