# Bucket-B2 expectations for prompt-evaluator
# Format: per-agent YAML, 5–10 grader-worthy expectations grounded in the agent
# file's explicit promises and/or current matrix failures. This file is consumed
# by the next pass that rewrites stimuli + graders end-to-end; do not treat it
# as a Vally grader file directly.
slug: prompt-evaluator
class: subagent  # subtype: prompt-quality evaluator (sandbox-scoped)
agent_file: .github/agents/hve-core/subagents/prompt-evaluator.agent.md
stimulus_file: evals/agent-behavior/stimuli/prompt-evaluator.yml
latest_result: null
source_review_date: 2026-05-28

expectations:
  - expectation_id: evaluation-log-in-sandbox
    summary: Evaluation log is created inside the provided sandbox folder.
    signal: Response names an evaluation-log.md path under the sandbox folder.
    pass_criteria: |
      Reported evaluation-log path lives in
      `.copilot-tracking/sandbox/<YYYY-MM-DD>-<topic>-<run-number>/` and is
      literally named `evaluation-log.md`.
    failure_modes:
      - Writes evaluation log outside the supplied sandbox folder.
      - Renames the file (e.g., `eval-log.md`, `evaluation.md`).
      - Reports no evaluation-log path at all.
    priority: high
    contract_ref: "agent §Evaluation Log + §Inputs"

  - expectation_id: reads-prompt-builder-and-writing-style
    summary: Pre-requisite reads of prompt-builder and writing-style instructions are evidenced.
    signal: Response references both instruction files by name.
    pass_criteria: |
      Response or evaluation-log description names both
      `.github/instructions/hve-core/prompt-builder.instructions.md` and
      `.github/instructions/hve-core/writing-style.instructions.md` as
      pre-requisite reads informing the evaluation.
    failure_modes:
      - Cites only one of the two instruction files.
      - Substitutes a different style/quality guide.
      - Skips the pre-requisite load and goes straight to findings.
    priority: high
    contract_ref: "agent §Pre-requisite Load Evaluation Context (steps 2–3)"

  - expectation_id: read-only-no-prompt-edits
    summary: Evaluator does not modify the target prompt file(s).
    signal: Response does not claim to edit or rewrite the target prompt.
    pass_criteria: |
      Response contains no statements modifying the target prompt file(s) or any
      other prompt-engineering artifact. Only the evaluation log may be
      created/updated.
    failure_modes:
      - Claims to apply recommended changes to the prompt file.
      - Rewrites a section of the prompt as part of the response.
    priority: high
    contract_ref: "agent §Required Protocol (items 1–2)"

  - expectation_id: response-status-vocabulary
    summary: Evaluation status uses one of the documented values.
    signal: Response includes a status line with Complete, In-Progress, or Blocked.
    pass_criteria: |
      Response reports evaluation status as one of `Complete`, `In-Progress`,
      or `Blocked` (case-insensitive match acceptable).
    failure_modes:
      - Uses non-documented status (`Done`, `Partial`, `Pass`/`Fail`).
      - Omits status entirely.
    priority: high
    contract_ref: "agent §Response Format"

  - expectation_id: findings-have-severity-and-category
    summary: Findings carry severity levels and category tags.
    signal: Response or log describes findings with severity + category.
    pass_criteria: |
      Findings include severity labels (e.g., Critical/Major/Minor or comparable
      tiered severity from the Prompt Quality Criteria) and a category tag
      identifying the evaluation domain (purpose, criteria, style, etc.).
    failure_modes:
      - Findings have no severity assignment.
      - Findings have no category and read as a flat list.
    priority: medium
    contract_ref: "agent §Evaluation Log + §Step 1 Evaluate Execution Log Findings"

  - expectation_id: checklist-ordered-by-severity
    summary: Recommended modifications are a checklist ordered by severity.
    signal: Response contains a checklist of recommendations with severity ordering.
    pass_criteria: |
      Response includes a checklist of recommended modifications targeting the
      specific prompt instruction file(s), ordered by (and including) severity
      labels.
    failure_modes:
      - Recommendations rendered as a paragraph rather than a checklist.
      - Checklist present but severity is missing or ordering ignored.
    priority: medium
    contract_ref: "agent §Response Format"

  - expectation_id: structured-for-parent-agent
    summary: Response is structured for a parent orchestrator, not a user.
    signal: Output omits user-facing chrome and presents the documented summary fields.
    pass_criteria: |
      Response has no end-user salutations, no "let me know" framing, and is
      organized around sandbox path, evaluation log path, status, key details,
      checklist, and clarifying questions. Consistent with
      `user-invocable: false`.
    failure_modes:
      - Greets the user or solicits next-step confirmation.
      - Adds branded headers/emojis implying a user-invocable agent.
    priority: medium
    contract_ref: "agent frontmatter `user-invocable: false` + §Response Format"

  - expectation_id: sandbox-and-evaluation-log-paths-reported
    summary: Both the sandbox folder and evaluation log paths appear in the response.
    signal: Response lists the relative sandbox folder path and the evaluation log path.
    pass_criteria: |
      Response includes (a) the relative path to the sandbox folder under
      `.copilot-tracking/sandbox/` and (b) the relative path to
      `evaluation-log.md` within that folder.
    failure_modes:
      - Mentions only one of the two paths.
      - Uses absolute paths or non-workspace-relative references.
    priority: low
    contract_ref: "agent §Response Format"
