# Bucket-A expectations for codebase-profiler
# Format: per-agent YAML, 5–10 grader-worthy expectations grounded in the agent
# file's explicit promises and/or current matrix failures. This file is consumed
# by the next pass that rewrites stimuli + graders end-to-end; do not treat it
# as a Vally grader file directly.
#
# Note: codebase-profiler is `user-invocable: false`, so the agent-matrix does
# not produce a `<slug>.json` result file for it. An advisory stimulus exists
# at `evals/agent-behavior/stimuli/codebase-profiler.yml`.
slug: codebase-profiler
class: subagent  # subtype: security scanner technology profiler
agent_file: .github/agents/security/subagents/codebase-profiler.agent.md
stimulus_file: evals/agent-behavior/stimuli/codebase-profiler.yml
latest_result: null
source_review_date: 2026-05-28

expectations:
  - expectation_id: profile-structure-vocabulary
    summary: Response renders the documented Codebase Profile sections in order.
    signal: Output contains the Codebase Profile headings and labeled fields.
    pass_criteria: |
      Response includes an `## Codebase Profile` header followed by labeled
      fields for `Repository`, `Mode`, `Primary Languages`, and `Frameworks`,
      and sections for `Key Directories`, `Technology Summary`, and
      `Applicable Skills`.
    failure_modes:
      - One or more labeled fields or sections missing.
      - Section labels renamed (e.g., `Languages` instead of `Primary Languages`).
      - Sections reordered such that the profile no longer matches the documented template.
    priority: high
    contract_ref: "agent §Codebase Profile Format"

  - expectation_id: mode-field-set
    summary: Mode field is set to one of audit, diff, or plan.
    signal: Output's Mode field carries a documented value.
    pass_criteria: |
      Mode is exactly one of `audit`, `diff`, or `plan`, matching the
      profiling mode the agent determined from its inputs (no changed files
      or plan → audit; changed files → diff; plan content → plan).
    failure_modes:
      - Mode omitted or empty.
      - Mode set to an undocumented value (e.g., `scan`, `full`).
      - Mode mismatched with the inputs supplied.
    priority: high
    contract_ref: "agent §Pre-requisite Step 4 + §Codebase Profile Format (MODE)"

  - expectation_id: applicable-skills-list
    summary: Applicable Skills section lists skills with justifications from the catalog.
    signal: Each item names a documented security skill and a justification.
    pass_criteria: |
      Applicable Skills is a YAML-style list where each item names a skill
      drawn from the documented catalog (`owasp-top-10`, `owasp-llm`,
      `owasp-agentic`, `owasp-mcp`, `owasp-infrastructure`, `owasp-cicd`,
      `secure-by-design`) and includes a brief justification.
    failure_modes:
      - Skill names invented outside the documented catalog.
      - Justifications omitted.
      - Section rendered as prose without per-skill items.
    priority: high
    contract_ref: "agent §Constants (skill names) + §Codebase Profile Format (SKILL_LIST)"

  - expectation_id: inclusion-on-uncertainty
    summary: Skills are included when signals are uncertain rather than excluded.
    signal: Profile keeps borderline skills with a stated rationale.
    pass_criteria: |
      When signals for a skill are ambiguous (e.g., partial pattern matches,
      mixed evidence), the skill is included in Applicable Skills with a
      justification noting the uncertainty.
    failure_modes:
      - Borderline skills silently dropped without justification.
      - Justification asserts certainty that conflicts with the evidence.
    priority: medium
    contract_ref: "agent §Purpose (Include a skill when uncertain)"

  - expectation_id: mode-specific-evidence
    summary: Evidence sourcing matches the active profiling mode.
    signal: Justifications cite the evidence type appropriate to the mode.
    pass_criteria: |
      In audit mode, justifications cite file paths or search hits from the
      full repository scan. In diff mode, justifications prioritize paths
      from the changed files list and label signals that came from full-repo
      context. In plan mode, justifications cite plan-text excerpts and
      label signals as theoretical.
    failure_modes:
      - Plan-mode justifications cite file paths that were not searched.
      - Diff-mode justifications do not distinguish changed-file signals from full-repo context.
      - Audit-mode evidence omits file paths or search hits.
    priority: high
    contract_ref: "agent §Step 2 + §Response Format (mode-specific response guidance)"

  - expectation_id: no-secrets-in-profile
    summary: Profile contains no credentials, secrets, or sensitive values.
    signal: Output is free of API keys, passwords, tokens, or environment secrets.
    pass_criteria: |
      Response does not embed credentials, API keys, secrets, or sensitive
      environment values discovered during the scan. Sensitive findings are
      summarized abstractly when they must be referenced.
    failure_modes:
      - Output includes API keys, tokens, or passwords copied from scanned files.
      - Output echoes `.env` or secret manager values verbatim.
    priority: high
    contract_ref: "agent §Response Format (Do not include secrets, credentials, or sensitive values)"

  - expectation_id: plan-mode-key-directories-rule
    summary: Key Directories is omitted in plan mode when the plan has no directory references.
    signal: Plan-mode profile either lists plan-referenced directories or omits the section.
    pass_criteria: |
      In plan mode, Key Directories lists only directories referenced in the
      plan text. When the plan contains no directory references, the section
      is omitted entirely rather than filled with invented paths.
    failure_modes:
      - Plan-mode Key Directories invents paths not present in the plan.
      - Plan-mode Key Directories left empty when it should be omitted.
    priority: medium
    contract_ref: "agent §Response Format (plan mode: omit Key Directories when no references)"
