microsoft/hve-core
Publicmirrored from https://github.com/microsoft/hve-coreAvailable
evals/agent-behavior/expectations/brd-builder.expectations.yml
167lines · modecode
| 1 | # Bucket-A expectations for brd-builder |
| 2 | # Format: per-agent YAML, 5–10 grader-worthy expectations grounded in the agent |
| 3 | # file's explicit promises and/or current matrix failures. This file is consumed |
| 4 | # by the next pass that rewrites stimuli + graders end-to-end; do not treat it |
| 5 | # as a Vally grader file directly. |
| 6 | # |
| 7 | # Note: the agent file places the BRD itself at `docs/brds/<name>-brd.md` and |
| 8 | # the session state at `.copilot-tracking/brd-sessions/<name>.state.json`. The |
| 9 | # current `brd-builder-class-recipe` stimulus' `tracking-file-write` grader |
| 10 | # expects the BRD path to start with `.copilot-tracking/(brd-sessions|research)/`, |
| 11 | # which does not match the agent's own contract. Expectations below ground in |
| 12 | # the agent file; the next pass should either align the grader to `docs/brds/` |
| 13 | # or expect both paths. |
| 14 | slug: brd-builder |
| 15 | class: research-writer |
| 16 | agent_file: .github/agents/project-planning/brd-builder.agent.md |
| 17 | stimulus_file: evals/agent-behavior/stimuli/brd-builder.yml |
| 18 | latest_result: evals/results/agent-matrix/2026-05-28/brd-builder.json |
| 19 | source_review_date: 2026-05-28 |
| 20 | |
| 21 | expectations: |
| 22 | - expectation_id: brd-file-location |
| 23 | summary: BRD file is written to `docs/brds/<kebab-case-name>-brd.md`. |
| 24 | signal: Output names a workspace path matching `docs/brds/<slug>-brd.md`. |
| 25 | pass_criteria: | |
| 26 | The reported BRD path is workspace-relative, starts with `docs/brds/`, |
| 27 | uses a kebab-case slug derived from the initiative name, and ends with |
| 28 | `-brd.md`. |
| 29 | failure_modes: |
| 30 | - BRD written to a temp directory or absolute path |
| 31 | (current 2026-05-28 output reports `C:\Users\…\AppData\Local\Temp\vally-eval-…\…-brd.md`). |
| 32 | - BRD written to repo root with no `docs/brds/` prefix. |
| 33 | - Filename ends with `.md` only (missing `-brd` suffix). |
| 34 | priority: high |
| 35 | contract_ref: "agent §File Management › File Locations (BRD file: `docs/brds/<kebab-case-name>-brd.md`)" |
| 36 | |
| 37 | - expectation_id: state-file-location |
| 38 | summary: State file is written under the BRD-sessions tracking subtree. |
| 39 | signal: Output names a workspace path matching `.copilot-tracking/brd-sessions/<slug>.state.json`. |
| 40 | pass_criteria: | |
| 41 | When the agent creates a BRD it also reports a state-file path beginning |
| 42 | with `.copilot-tracking/brd-sessions/` whose filename ends with |
| 43 | `.state.json` and whose slug matches the BRD slug. |
| 44 | failure_modes: |
| 45 | - State file omitted entirely. |
| 46 | - State file written next to BRD in `docs/brds/`. |
| 47 | - State file written outside `.copilot-tracking/brd-sessions/`. |
| 48 | priority: high |
| 49 | contract_ref: "agent §File Management › File Locations + §State Tracking" |
| 50 | |
| 51 | - expectation_id: required-brd-sections |
| 52 | summary: Drafted BRD contains all required sections from the agent contract. |
| 53 | signal: Output references each of the six required section names. |
| 54 | pass_criteria: | |
| 55 | The response (or the BRD body it summarizes) names all six required |
| 56 | sections (case-insensitive): Business Context and Background, Problem |
| 57 | Statement and Business Drivers, Business Objectives and Success Metrics, |
| 58 | Stakeholders and Roles, Scope, Business Requirements. |
| 59 | failure_modes: |
| 60 | - Response shows only goals/scope/metrics and omits Stakeholders or |
| 61 | Problem Statement. |
| 62 | - Conditional sections (Processes, Data, Benefits) substituted for required ones. |
| 63 | priority: high |
| 64 | contract_ref: "agent §BRD Structure (Required sections list)" |
| 65 | |
| 66 | - expectation_id: requirement-id-format |
| 67 | summary: Each requirement has the documented BR-NNN identifier. |
| 68 | signal: Requirements use `BR-001`, `BR-002`, etc. |
| 69 | pass_criteria: | |
| 70 | Listed business requirements use the `BR-<NNN>` ID format documented in |
| 71 | §BRD Structure › Requirement Quality, with zero-padded three-digit |
| 72 | numbers. Each requirement also includes a testable description and a |
| 73 | priority. |
| 74 | failure_modes: |
| 75 | - Requirements numbered as `R1`, `Req-01`, `1.`, or unnumbered bullets. |
| 76 | - IDs present but with no testable description or priority. |
| 77 | priority: medium |
| 78 | contract_ref: "agent §BRD Structure › Requirement Quality (unique ID `BR-001`, testable description, priority)" |
| 79 | |
| 80 | - expectation_id: frontmatter-fields |
| 81 | summary: BRD frontmatter includes the documented metadata fields. |
| 82 | signal: Output shows YAML frontmatter with `title`, `description`, `author`, `ms.date`, `ms.topic`. |
| 83 | pass_criteria: | |
| 84 | The BRD body the agent shows or summarizes includes YAML frontmatter |
| 85 | with at least `title`, `description`, `author`, `ms.date`, and |
| 86 | `ms.topic` keys, and does NOT include `<!-- markdownlint-disable-file -->`. |
| 87 | failure_modes: |
| 88 | - No frontmatter shown. |
| 89 | - Frontmatter missing one or more required keys. |
| 90 | - File begins with `<!-- markdownlint-disable-file -->`. |
| 91 | priority: medium |
| 92 | contract_ref: "agent §BRD Creation (Include YAML frontmatter with `title`, `description`, `author`, `ms.date`, `ms.topic`; exclude markdownlint disable comment)" |
| 93 | |
| 94 | - expectation_id: ambiguous-request-clarification |
| 95 | summary: Vague requests trigger 2–3 initial scope questions before file creation. |
| 96 | signal: Output asks initial Business Initiative / Scope Boundaries questions. |
| 97 | pass_criteria: | |
| 98 | When the stimulus is vague (no initiative name, problem-only statement, |
| 99 | or multiple unrelated ideas), the response asks 2–3 essential questions |
| 100 | from the documented Initial Questions list (initiative name, business |
| 101 | problem, driver, initiative type, primary stakeholders) before creating |
| 102 | the BRD file. |
| 103 | failure_modes: |
| 104 | - Agent immediately creates a BRD file for a vague request with no questioning. |
| 105 | - Agent asks ten or more questions, ignoring the 2–3 budget. |
| 106 | priority: medium |
| 107 | applies_when: "stimulus is vague (no clear initiative name or scope)" |
| 108 | contract_ref: "agent §Handling Ambiguous Requests + §Questioning Strategy › Initial Questions" |
| 109 | |
| 110 | - expectation_id: explicit-request-immediate-creation |
| 111 | summary: Clear initiative requests trigger immediate file creation, not extended questioning. |
| 112 | signal: Output reports a BRD file path on the first turn when the initiative is named. |
| 113 | pass_criteria: | |
| 114 | When the user provides an explicit initiative name, clear business change, |
| 115 | or specific project reference (e.g. "Create a BRD for Claims Automation |
| 116 | Program" or "Draft a BRD for self-service password reset"), the response |
| 117 | creates the BRD file on the first turn and reports its path, then asks |
| 118 | refinement questions afterward. |
| 119 | failure_modes: |
| 120 | - Agent asks ten clarifying questions before creating the file for an |
| 121 | explicit initiative. |
| 122 | - Agent never reports a file path for a clear initiative request. |
| 123 | priority: medium |
| 124 | applies_when: "stimulus names a clear initiative or feature (e.g. `brd-builder-class-recipe`)" |
| 125 | contract_ref: "agent §Handling Ambiguous Requests (Create files immediately when the user provides an explicit initiative name)" |
| 126 | |
| 127 | - expectation_id: emoji-checklist-questions |
| 128 | summary: Refinement questions use the documented emoji-checklist format. |
| 129 | signal: Output uses `[ ] ❓` / `[x] ✅` / `[ ] ❌` markers on numbered question items. |
| 130 | pass_criteria: | |
| 131 | When the agent presents refinement questions, items appear as |
| 132 | `* <id>. [ ] ❓ <question>` (unanswered), `[x] ✅` (answered), or |
| 133 | `[ ] ❌` (N/A), grouped under emoji-prefixed area headings (`🎯`, `👥`, |
| 134 | `🔄`, `📊`, `⚡`), per §Questioning Strategy › Refinement Questions Checklist. |
| 135 | failure_modes: |
| 136 | - Questions presented as plain numbered list with no checklist or emoji. |
| 137 | - Question IDs renumbered between turns instead of staying stable. |
| 138 | priority: low |
| 139 | applies_when: "refinement / elicitation turn" |
| 140 | contract_ref: "agent §Questioning Strategy › Refinement Questions Checklist + §Initial Questions" |
| 141 | |
| 142 | - expectation_id: no-source-edit |
| 143 | summary: BRD authoring does not modify source code or build manifests. |
| 144 | signal: Output does not name modifications to source-tree files. |
| 145 | pass_criteria: | |
| 146 | No occurrences of edit/create verbs paired with `.cs`/`.py`/`.ts`/`.js`/ |
| 147 | `.go`/`.rs`/`.java`/`package.json`/`pyproject.toml`/`Cargo.toml` paths. |
| 148 | failure_modes: |
| 149 | - Drafting a password-reset BRD leads to modifying auth service source files. |
| 150 | - Agent claims to update `package.json` as part of BRD work. |
| 151 | priority: medium |
| 152 | contract_ref: "agent scope (writes are confined to `docs/brds/` and `.copilot-tracking/brd-sessions/`)" |
| 153 | |
| 154 | - expectation_id: stimulus-topic-fidelity |
| 155 | summary: Response substantively addresses the BRD topic from the stimulus. |
| 156 | signal: Stimulus-derived keywords appear in the response body. |
| 157 | pass_criteria: | |
| 158 | For the `brd-builder-class-recipe` stimulus, the response contains terms |
| 159 | from {password, reset, business, requirement, scope, success} and the |
| 160 | BRD body addresses password-reset specifics (e.g. self-service flow, |
| 161 | MFA, ticket-volume reduction) rather than generic template prose. |
| 162 | failure_modes: |
| 163 | - Off-topic BRD with no password-reset references. |
| 164 | - Requirements written as placeholders ("BR-001: Describe the feature"). |
| 165 | priority: medium |
| 166 | stimulus_scoped: true |
| 167 | contract_ref: "stimulus design (per-stimulus, not agent-intrinsic)" |
| 168 | |