microsoft/hve-core
Publicmirrored from https://github.com/microsoft/hve-coreAvailable
evals/agent-behavior/expectations/code-review-standards.expectations.yml
126lines · modecode
| 1 | # Bucket-A expectations for code-review-standards |
| 2 | # Format: per-agent YAML, 5–10 grader-worthy expectations grounded in the agent |
| 3 | # file's explicit promises and/or current matrix failures. This file is consumed |
| 4 | # by the next pass that rewrites stimuli + graders end-to-end; do not treat it |
| 5 | # as a Vally grader file directly. |
| 6 | # |
| 7 | # Note: code-review-standards is the language-standards sibling of |
| 8 | # code-review-functional. It enforces coding-standards instructions |
| 9 | # (`.github/instructions/coding-standards/<language>/`) against the diff and |
| 10 | # persists findings to |
| 11 | # `.copilot-tracking/reviews/code-reviews/<branch>/<run>/standards-review.md`. |
| 12 | slug: code-review-standards |
| 13 | class: code-reviewer |
| 14 | agent_file: .github/agents/coding-standards/code-review-standards.agent.md |
| 15 | stimulus_file: evals/agent-behavior/stimuli/code-review-standards.yml |
| 16 | latest_result: evals/results/agent-matrix/2026-05-28/code-review-standards.json |
| 17 | source_review_date: 2026-05-28 |
| 18 | |
| 19 | expectations: |
| 20 | - expectation_id: standards-scope-only |
| 21 | summary: Findings address language-standards compliance, not functional bugs. |
| 22 | signal: Output focuses on style, naming, idioms, lint rules, and coding-standards instructions. |
| 23 | pass_criteria: | |
| 24 | Findings cite coding-standards rules (formatting, naming, idiomatic |
| 25 | usage, linter directives, instruction-file rules). Pure functional |
| 26 | defects (incorrect logic, missing error handling, race conditions) |
| 27 | are absent or explicitly deferred to `code-review-functional`. |
| 28 | failure_modes: |
| 29 | - Findings list logic bugs or behavioral defects as primary findings. |
| 30 | - Mixes functional review into standards review. |
| 31 | priority: high |
| 32 | contract_ref: "agent §Scope (language standards only; functional correctness is owned by code-review-functional)" |
| 33 | |
| 34 | - expectation_id: standards-instruction-reference |
| 35 | summary: Findings reference the relevant coding-standards instruction file. |
| 36 | signal: Output names a `.github/instructions/coding-standards/<language>/*.instructions.md` path or rule. |
| 37 | pass_criteria: | |
| 38 | Each rule-based finding cites either (a) the matching instruction |
| 39 | file path under `.github/instructions/coding-standards/`, or (b) a |
| 40 | named rule from that instruction file. Generic style critiques |
| 41 | without instruction-file backing are flagged as opinion. |
| 42 | failure_modes: |
| 43 | - Findings critique style with no instruction-file reference. |
| 44 | - Names a non-existent instruction file. |
| 45 | priority: high |
| 46 | contract_ref: "agent §Phase 1 (load coding-standards instructions for the changed languages)" |
| 47 | |
| 48 | - expectation_id: severity-per-finding |
| 49 | summary: Each standards finding carries a severity label. |
| 50 | signal: Output applies severity words per finding. |
| 51 | pass_criteria: | |
| 52 | Each standards finding has a case-insensitive severity from |
| 53 | `critical|high|medium|low|info|warning`. Severity is per-finding. |
| 54 | failure_modes: |
| 55 | - Findings unlabeled. |
| 56 | - Severities used only in summary text. |
| 57 | priority: high |
| 58 | contract_ref: "agent §Output Contract (severity per finding); current `severity-vocab` grader" |
| 59 | |
| 60 | - expectation_id: findings-structure-present |
| 61 | summary: Output presents findings in a structured form. |
| 62 | signal: Output contains a severity-labeled table or per-finding sections. |
| 63 | pass_criteria: | |
| 64 | Output uses a markdown table with severity column OR per-finding |
| 65 | sections using `finding|issue|concern|recommendation` language with |
| 66 | each finding tied to a file path and line range when possible. |
| 67 | failure_modes: |
| 68 | - Single paragraph with no per-finding structure. |
| 69 | - Bulleted list with no severity framing. |
| 70 | priority: high |
| 71 | contract_ref: "agent §Output Contract; current `findings-table-present` grader" |
| 72 | |
| 73 | - expectation_id: diff-scoped-findings |
| 74 | summary: Findings are scoped to the reviewed diff. |
| 75 | signal: Findings reference changed files or hunks from the diff. |
| 76 | pass_criteria: | |
| 77 | Findings cite changed files, line ranges, or hunks from the supplied |
| 78 | diff. Findings that step outside the diff are explicitly marked as |
| 79 | out-of-scope context. |
| 80 | failure_modes: |
| 81 | - Findings invented for files not in the diff. |
| 82 | - Bulk findings about unrelated source trees. |
| 83 | priority: medium |
| 84 | contract_ref: "agent §Scope (diff-scoped standards review)" |
| 85 | |
| 86 | - expectation_id: tracking-path-shape |
| 87 | summary: Standards review artifact lives at the documented path. |
| 88 | signal: Output names a path matching `.copilot-tracking/reviews/code-reviews/<branch>/<run>/standards-review.md`. |
| 89 | pass_criteria: | |
| 90 | When the agent reports persisting a standards review, the path starts |
| 91 | with `.copilot-tracking/reviews/code-reviews/`, includes a normalized |
| 92 | branch segment, includes a run identifier, and ends in |
| 93 | `standards-review.md`. |
| 94 | failure_modes: |
| 95 | - Artifact written outside `.copilot-tracking/reviews/code-reviews/`. |
| 96 | - Filename other than `standards-review.md`. |
| 97 | priority: medium |
| 98 | applies_when: "agent reports artifact creation" |
| 99 | contract_ref: "agent §Tracking Artifact (standards-review.md)" |
| 100 | |
| 101 | - expectation_id: verdict-stated |
| 102 | summary: Standards review ends with a verdict from the documented vocabulary. |
| 103 | signal: Output names an overall verdict. |
| 104 | pass_criteria: | |
| 105 | Response concludes with an overall standards verdict drawn from |
| 106 | `approve|approve with changes|request changes|block`. Verdict reflects |
| 107 | the highest-severity finding. |
| 108 | failure_modes: |
| 109 | - No final verdict. |
| 110 | - Verdict expressed only in informal prose. |
| 111 | priority: medium |
| 112 | contract_ref: "agent §Output Contract (standards verdict)" |
| 113 | |
| 114 | - expectation_id: no-source-edit |
| 115 | summary: Review-only — no edits to source code or build manifests. |
| 116 | signal: Output does not reference modifications to source-tree files. |
| 117 | pass_criteria: | |
| 118 | No occurrences of edit/create verbs paired with `.cs`/`.py`/`.ts`/`.js`/ |
| 119 | `.go`/`.rs`/`.java`/`package.json`/`pyproject.toml`/`Cargo.toml` paths. |
| 120 | Proposed fixes appear as recommendations or fenced snippets, not as |
| 121 | claimed edits. |
| 122 | failure_modes: |
| 123 | - Agent claims to apply a style fix during review. |
| 124 | - Edits build manifests while reviewing. |
| 125 | priority: high |
| 126 | contract_ref: "agent scope (review-only); current `no-source-edit` grader" |
| 127 | |