# Bucket-A expectations for security-planner
# Format: per-agent YAML, 5–10 grader-worthy expectations grounded in the agent
# file's explicit promises and/or current matrix failures. This file is consumed
# by the next pass that rewrites stimuli + graders end-to-end; do not treat it
# as a Vally grader file directly.
slug: security-planner
class: planner-coach
agent_file: .github/agents/security/security-planner.agent.md
stimulus_file: evals/agent-behavior/stimuli/security-planner.yml
latest_result: evals/results/agent-matrix/2026-05-28/security-planner.json
source_review_date: 2026-05-28

expectations:
  - expectation_id: state-file-location
    summary: Planning state is written under the security-plans tracking subtree.
    signal: Output names a workspace path matching `.copilot-tracking/security-plans/<project-slug>/`.
    pass_criteria: |
      The response reports a planning-state path beginning with
      `.copilot-tracking/security-plans/` with a project-slug subfolder,
      satisfying the `tracking-file-write` grader
      (`.copilot-tracking/security-plans`).
    failure_modes:
      - State written to `~/.copilot/session-state/` or another absolute path
        (current 2026-05-28 output fails `tracking-file-write` this way).
      - State written to a temp directory.
      - No state location reported.
    priority: high
    contract_ref: "agent §State Management (state under `.copilot-tracking/security-plans/{project-slug}/`)"

  - expectation_id: six-phase-sequence
    summary: The six security phases are enumerated in the documented order.
    signal: Output names Scoping, Bucket Analysis, Standards Mapping, STRIDE Model, Backlog Generation, Handoff.
    pass_criteria: |
      The response lists the documented six phases in order: Scoping → Bucket
      Analysis → Standards Mapping → STRIDE Model → Backlog Generation →
      Handoff.
    failure_modes:
      - Fewer than six phases listed.
      - Phases reordered or relabeled away from the documented sequence.
    priority: high
    contract_ref: "agent §Phases (six-phase security workflow)"

  - expectation_id: phase-markers-present
    summary: Phases are listed with markdown headers or numbered/Phase markers.
    signal: Output uses `##`/`###` headings or `Phase N`/`Step N`/`N.` line starts.
    pass_criteria: |
      The six phases are presented using line-start structural markers
      (`##`, `###`, `Phase N`, `Step N`, or `N.`), satisfying the
      `phase-marker-present` grader.
    failure_modes:
      - Phases listed only inside a table or inline prose with no line-start
        markers.
    priority: high
    contract_ref: "agent §Orchestration (six-phase workflow presentation)"

  - expectation_id: security-caution-block
    summary: Response includes the verbatim security CAUTION disclaimer block.
    signal: Output contains the documented security caution/disclaimer language.
    pass_criteria: |
      The session opens with the security CAUTION block reproduced verbatim,
      acknowledging professional review and the advisory nature of the output.
    failure_modes:
      - Caution block omitted.
      - Caution block paraphrased rather than reproduced verbatim.
    priority: medium
    contract_ref: "agent §Security CAUTION (verbatim disclaimer block)"

  - expectation_id: rai-handoff-for-ai-systems
    summary: ML/LLM/RAG systems set raiEnabled and trigger an RAI handoff.
    signal: Output sets `raiEnabled=true` and references an RAI handoff when the system involves ML/LLM/RAG.
    pass_criteria: |
      When the assessed system involves ML, LLM, or RAG components, the
      response sets `raiEnabled=true` in state and references handoff to the
      RAI planner.
    failure_modes:
      - AI system assessed with no RAI handoff or raiEnabled flag.
    priority: medium
    applies_when: "the assessed system involves ML/LLM/RAG components"
    contract_ref: "agent §RAI Integration (raiEnabled + RAI handoff for AI systems)"

  - expectation_id: bounded-questions-per-turn
    summary: Each turn asks within the documented question range with checklist markers.
    signal: Output asks 3–5 questions using ❓/✅/❌ checklist markers.
    pass_criteria: |
      Per-turn questioning stays within the documented range (3–5) and uses
      the emoji checklist convention.
    failure_modes:
      - More than five questions asked in one turn.
      - Plain questions with no checklist markers.
    priority: low
    contract_ref: "agent §Question Cadence (3–5 per turn, emoji checklists)"

  - expectation_id: no-source-edit
    summary: Security planning does not modify source code or build manifests.
    signal: Output does not name modifications to source-tree files.
    pass_criteria: |
      No occurrences of edit/create verbs paired with `.cs`/`.py`/`.ts`/`.js`/
      `package.json` paths. Writes are confined to `.copilot-tracking/security-plans/`.
    failure_modes:
      - Planning leads to editing the REST API's source code.
    priority: medium
    contract_ref: "agent scope (writes confined to `.copilot-tracking/security-plans/`)"

  - expectation_id: stimulus-topic-fidelity
    summary: Response substantively addresses the security target from the stimulus.
    signal: Stimulus-derived terms appear in the response body.
    pass_criteria: |
      For the `security-planner-class-recipe` stimulus, the response frames the
      six phases around a public REST API and reports where planning state was
      written, rather than emitting generic placeholder content.
    failure_modes:
      - Off-topic plan with no reference to the REST API target.
      - Generic template with placeholder system names.
    priority: medium
    stimulus_scoped: true
    contract_ref: "stimulus design (per-stimulus, not agent-intrinsic)"
