Testing Recipes

Recipes are real Python modules — test them like any other Python code. pretorin recipe new <id> creates an empty tests/ package (just an __init__.py) next to scripts/; this page covers what to put in it.

Three Layers Worth Testing

A recipe has three layers and each rewards a different test style:

1. Pure helpers inside scripts/ — redaction, normalization, parsers. Plain unit tests with no fixtures. Fastest feedback loop; most coverage per line of test code.
2. run against a fake ctx — the script’s main entry point. Mock ctx.api_client so you don’t hit the network.
3. End-to-end through the recipe runner — load the recipe, call its script through the runner, assert the result. Slower but proves the manifest, the importlib-based dispatch, and the script all line up.

Unit-Testing Helpers

If you’ve factored out helpers into scripts/redact.py or scripts/normalize.py, import them directly:

# tests/test_helpers.py
from scripts.redact import redact_aws_keys

def test_redact_aws_keys_replaces_full_key() -> None:
    text = "AKIAIOSFODNN7EXAMPLE"
    redacted = redact_aws_keys(text)
    assert "AKIA" not in redacted
    assert "[REDACTED:AWS_KEY]" in redacted

The recipe runner adds the scripts/ directory to sys.path. In tests, make sure your pytest.ini or pyproject.toml does the same:

[tool.pytest.ini_options]
pythonpath = ["scripts"]

Testing run with a Fake ctx

The script’s run function takes a ctx argument typed as Any (loose intentionally — see Script contract). A MagicMock with AsyncMock for the I/O methods is enough:

# tests/test_run.py
from unittest.mock import AsyncMock, MagicMock
import pytest

from scripts.run_scan import run

@pytest.mark.asyncio
async def test_run_returns_summary_for_no_rules() -> None:
    ctx = MagicMock()
    ctx.system_id = "sys-1"
    ctx.api_client = MagicMock()
    ctx.api_client.get_test_manifest = AsyncMock(
        return_value={"applicable_stigs": []}
    )

    result = await run(ctx, stig_id="EMPTY_STIG")

    assert result["stig_id"] == "EMPTY_STIG"
    assert result["summary"]["total"] == 0

This shape works because the built-in scanner recipes reach the manifest through ctx.api_client.get_test_manifest, so stubbing that one method is enough to drive run down its no-rules path.

For a script that also shells out to an external tool, stub at the module level instead of through ctx. tests/recipes/test_openscap_run_scan.py does this for openscap-baseline: it replaces the script module’s fetch_test_manifest, rules_for_stig, summarize_results, and OpenSCAPScanner attributes, so no oscap binary is needed and each branch (no rules, scanner unavailable, successful scan) is reachable. tests/recipes/test_builtin_scanner_recipes.py is the complementary manifest-level suite — it asserts that all five scanner recipes load, expose a run_scan script, declare their required CLI, and carry tier: official, without invoking any script.

End-to-End Through the Runner

The strongest test exercises the full path: registry loads the manifest, runner imports the script, script runs against a fake API client. This is what tests/recipes/test_code_evidence_capture.py does for the code-evidence-capture recipe and it’s the regression-test pattern to copy.

Sketch:

import pytest
from pretorin.recipes import loader as loader_module
from pretorin.recipes.loader import clear_cache
from pretorin.recipes.registry import RecipeRegistry
from pretorin.recipes.runner import RecipeScriptContext, run_script
from unittest.mock import AsyncMock, MagicMock

@pytest.fixture(autouse=True)
def _isolate(tmp_path, monkeypatch):
    clear_cache()
    monkeypatch.setattr(loader_module, "_user_recipes_root", lambda: tmp_path / "u")
    monkeypatch.setattr(loader_module, "_project_recipes_root", lambda start=None: None)

@pytest.mark.asyncio
async def test_my_recipe_end_to_end() -> None:
    registry = RecipeRegistry()
    entry = registry.get("my-recipe")
    assert entry is not None

    api_client = MagicMock()
    api_client.create_evidence = AsyncMock(return_value={"id": "ev-1"})
    ctx = RecipeScriptContext(
        system_id="sys-1",
        framework_id="nist-800-53-r5",
        api_client=api_client,
        logger=MagicMock(),
        recipe_id="my-recipe",
        recipe_version="0.1.0",
        recipe_context_id="ctx-test",
    )

    result = await run_script(
        recipe=entry.active,
        script_name="capture",
        ctx=ctx,
        params={"control_id": "ac-2"},
    )

    assert result["evidence_id"] == "ev-1"

For a community recipe outside the pretorin source tree, point the loader at your recipe’s directory:

monkeypatch.setattr(loader_module, "_user_recipes_root", lambda: my_recipe_parent)

Run It Locally

Before wiring the recipe into an agent, exercise it directly:

pretorin recipe run my-recipe --param key=value --param limit=20

pretorin recipe run loads the recipe through the registry (or --path for a not-yet-registered directory), opens a recipe execution context, calls the script, prints the return value, and closes the context. It bypasses the MCP boundary, so:

• Use it for fast iteration on the script itself.
• Pure transformation recipes (return data, don’t write to the platform) work end-to-end.
• Recipes that do write through ctx.api_client need explicit audit_metadata on each call — the MCP boundary stamps automatically; this command does not. See Writer tools.

--no-context skips opening the execution context for recipes that don’t need it.

Validate as a Smoke Test

pretorin recipe validate <id> runs the manifest schema check, the script existence check, and the description-quality check. Add it to your CI as a shell-out smoke test:

- name: Validate recipes
  run: |
    pretorin recipe validate my-recipe
    pretorin recipe validate my-other-recipe

This catches “you renamed the script and forgot to update the manifest” faster than any pytest assertion will.

It also prints warning: lines for anything still holding the pretorin recipe new boilerplate — a description or use_when that starts with TODO, or a script body that still contains the scaffold’s # TODO: implement the tool. stub. Warnings do not fail the command (a fresh scaffold still exits 0), so a recipe you have not finished writing won’t break your CI, but they tell you which parts of the agent-facing contract are still placeholder text. In --json mode they appear under a warnings key alongside valid and issues.

What Not to Test

• Don’t test the platform API. Your recipe is an adapter; testing what create_evidence does is pretorin’s job, not yours. Mock the client.
• Don’t test pydantic validation of the manifest. That’s already covered by pretorin’s loader tests. If your manifest is malformed, pretorin recipe validate will tell you.
• Don’t test redaction patterns. Use pretorin.evidence.redact’s helpers and trust them.