Plan Trajectory Evaluations
The cross-harness runner includes an opt-in deterministic suite for checking how a real agent follows Pretorin’s single-context Plan contract. It does not use an LLM judge, and standard pull-request tests do not call an external model.
uv run python tools/cross-harness-smoke.py --setup
uv run python tools/cross-harness-smoke.py --suite plan --harness codex
uv run python tools/cross-harness-smoke.py --suite plan --harness claude
Every run prints its plan (harnesses × scenarios) and waits for confirmation before
starting. Pass --yes to skip that prompt in a non-interactive context — an unanswered
prompt aborts the run. Narrow a run with repeatable --harness / --scenario flags, and
raise the per-scenario timeout with --timeout <seconds>.
Each scenario starts a fresh harness process. P8 intentionally starts a second process to prove resume behavior across a real fresh-session boundary. The runner captures the harness name, CLI version, model identity when the harness exposes it, ordered Pretorin tool calls, an allowlisted subset of scope inputs, and a deterministic verdict. Raw harness output is not retained for Plan scenarios. Narrative text, evidence content, tokens, and secrets are excluded from the normalized trace.
Scenarios
| ID | Contract checked |
|---|---|
| P1 | A write attributed to a Plan is rejected while that Plan is still draft. |
| P2 | Calls remain inside one system/framework/control scope. |
| P3 | Authoritative state is read before a governed write. |
| P4 | recipe_required causes source preflight and start_recipe, with no live write retry. |
| P5 | Structured errors cause safe recovery or a clean stop. |
| P6 | Completed work has a scoped terminal step record. |
| P7 | Premature completion fails while steps or criteria remain open. |
| P8 | Resume continues without repeating a completed side effect. |
P1-P8 are deliberately single-context. To evaluate another framework or system, switch the active context and start a separate run.
Plan attribution is opt-in on Tier-1 writes. P1 evaluates the state guard once
a write carries plan_id; it does not claim that an unattributed Tier-1 write
is rejected by the current product contract.
Results and safety
Results are written beneath tools/logs/cross-harness/<run-id>/ as YAML, a
Markdown matrix, and privacy-bounded JSON traces. pass and fail represent a
fully observed transition. soft means the harness did not expose enough
structured information for a deterministic verdict and must not be treated as
a pass.
The prompts avoid successful platform-content mutation: guardrail scenarios
stop after the expected rejection, while bookkeeping and resume scenarios use
local Plan state. Only a new Plan whose intent contains the exact scenario
marker (Plan trajectory evaluation P1 through P8) is attributable to the
trial and eligible for cleanup. An attributable draft or active Plan is
cancelled after classification; unrelated or concurrently created Plans are
never changed. Cleanup failures make the trial error and preserve the original
evaluation verdict in the results. Terminal Plan records remain available as
audit history. Raw harness and MCP output is redacted on success, timeout, and
non-zero exit paths. Run the live suite only in a controlled context and review
the generated Plan records after the run. The generic S1-S4 smoke suite remains
the default and is unchanged:
uv run python tools/cross-harness-smoke.py