microsoft/hve-core
Publicmirrored from https://github.com/microsoft/hve-coreAvailable
evals/agent-behavior/expectations/agentic-workflows.expectations.yml
121lines · modecode
| 1 | # Bucket-A expectations for agentic-workflows |
| 2 | # Format: per-agent YAML, 5–10 grader-worthy expectations grounded in the agent |
| 3 | # file's explicit promises and/or current matrix failures. This file is consumed |
| 4 | # by the next pass that rewrites stimuli + graders end-to-end; do not treat it |
| 5 | # as a Vally grader file directly. |
| 6 | # |
| 7 | # Note: the 2026-05-28 matrix run for `agentic-workflows` FAILS both |
| 8 | # `phase-marker-present` and `tracking-file-write` because the output |
| 9 | # wrote the plan to an arbitrary path |
| 10 | # (`C:\Users\…\.copilot\session-state\…\plan.md`) instead of the |
| 11 | # `.copilot-tracking/plans/` subtree, and structured phases inside a table |
| 12 | # rather than as headed sections matching `^(##|###|Step \d+|Phase \d+|\d+\.)`. |
| 13 | # Expectations below promote contract-grounded checks targeting both |
| 14 | # failures plus broader Engineering-Excellence planning conformance. |
| 15 | slug: agentic-workflows |
| 16 | class: planner-coach |
| 17 | agent_file: .github/agents/hve-core/agentic-workflows.agent.md |
| 18 | stimulus_file: evals/agent-behavior/stimuli/agentic-workflows.yml |
| 19 | latest_result: evals/results/agent-matrix/2026-05-28/agentic-workflows.json |
| 20 | source_review_date: 2026-05-28 |
| 21 | |
| 22 | expectations: |
| 23 | - expectation_id: phase-headings-present |
| 24 | summary: Plan is organized into headed phases or numbered steps. |
| 25 | signal: Output contains lines matching `^(##|###|Step \d+|Phase \d+|\d+\.)`. |
| 26 | pass_criteria: | |
| 27 | Plan body contains multiple top-level markers from |
| 28 | `## <heading>`, `### <heading>`, `Step <n>`, `Phase <n>`, or numbered |
| 29 | list items (`1.` …). A table row alone does not satisfy this. |
| 30 | failure_modes: |
| 31 | - Phases captured only as rows in a single table. |
| 32 | - Single prose block with no headings or step markers |
| 33 | (matches current 2026-05-28 matrix failure). |
| 34 | priority: high |
| 35 | contract_ref: "current `phase-marker-present` grader (regex /(?m)^(##|###|Step \\d+|Phase \\d+|\\d+\\.)/)" |
| 36 | |
| 37 | - expectation_id: tracking-plan-path |
| 38 | summary: Plan is written under the `.copilot-tracking/plans/` subtree. |
| 39 | signal: Output names a path matching `.copilot-tracking/plans/<YYYY-MM-DD>/<slug>-plan.instructions.md`. |
| 40 | pass_criteria: | |
| 41 | Reported plan path is workspace-relative, starts with |
| 42 | `.copilot-tracking/plans/`, includes a dated subdirectory, and ends in |
| 43 | `-plan.instructions.md`. Absolute paths to `.copilot` session state, |
| 44 | temp directories, or repo root are not acceptable. |
| 45 | failure_modes: |
| 46 | - Plan written to an absolute path under `.copilot/session-state/` |
| 47 | (current 2026-05-28 matrix failure). |
| 48 | - Plan written to repo root with no tracking prefix. |
| 49 | - Path matches `.copilot-tracking/plans/` but is missing the dated |
| 50 | subdirectory or `-plan.instructions.md` suffix. |
| 51 | priority: high |
| 52 | contract_ref: "agent §Tracking Artifacts (Implementation Plan path under `.copilot-tracking/plans/{{YYYY-MM-DD}}/{{task-description}}-plan.instructions.md`)" |
| 53 | |
| 54 | - expectation_id: success-criteria-per-phase |
| 55 | summary: Each phase names success criteria. |
| 56 | signal: Phases include explicit success or exit criteria. |
| 57 | pass_criteria: | |
| 58 | Each phase carries at least one explicit success/exit criterion |
| 59 | (acceptance test, observable outcome, validation command, or |
| 60 | measurable result). Criteria are concrete rather than aspirational |
| 61 | ("works correctly"). |
| 62 | failure_modes: |
| 63 | - Phases listed with names only and no acceptance criteria. |
| 64 | - Success criteria are vague ("complete and tested"). |
| 65 | priority: high |
| 66 | contract_ref: "stimulus instruction (Break into phases with success criteria) + agent §Implementation Plan (per-phase success criteria)" |
| 67 | |
| 68 | - expectation_id: parallelization-markers |
| 69 | summary: Plan annotates phases with parallelization markers when appropriate. |
| 70 | signal: Output uses `<!-- parallelizable: true -->` or `<!-- parallelizable: false -->` per phase. |
| 71 | pass_criteria: | |
| 72 | Each phase carries a `<!-- parallelizable: true -->` or |
| 73 | `<!-- parallelizable: false -->` annotation. When all phases must run |
| 74 | serially, the markers are still present and set to `false`. |
| 75 | failure_modes: |
| 76 | - Markers omitted entirely. |
| 77 | - Parallelization expressed only in prose with no machine-readable |
| 78 | annotation. |
| 79 | priority: medium |
| 80 | contract_ref: "agent §Implementation Plan (mark phases with `<!-- parallelizable: true/false -->`)" |
| 81 | |
| 82 | - expectation_id: tracking-markdown-disable-comment |
| 83 | summary: Plan file begins with the markdownlint-disable directive. |
| 84 | signal: Output references `<!-- markdownlint-disable-file -->` at the top of the tracking file. |
| 85 | pass_criteria: | |
| 86 | Plan file (or summary thereof) begins with the literal directive |
| 87 | `<!-- markdownlint-disable-file -->` on the first line. The same |
| 88 | directive is NOT placed in published markdown under `docs/`. |
| 89 | failure_modes: |
| 90 | - Plan file shown without the directive. |
| 91 | - Directive placed in non-tracking surfaces. |
| 92 | priority: low |
| 93 | applies_when: "agent reports plan-file creation" |
| 94 | contract_ref: "repo convention (`.copilot-tracking/` files begin with `<!-- markdownlint-disable-file -->`)" |
| 95 | |
| 96 | - expectation_id: research-context-reference |
| 97 | summary: Plan references prior research or context inputs. |
| 98 | signal: Output references a `.copilot-tracking/research/` path or names the inputs that informed the plan. |
| 99 | pass_criteria: | |
| 100 | Plan body or summary names the research inputs it builds on, either |
| 101 | by referencing a research file under |
| 102 | `.copilot-tracking/research/<YYYY-MM-DD>/` or by listing the |
| 103 | requirements/sources analyzed before phase design. |
| 104 | failure_modes: |
| 105 | - Plan presented with no rationale for its phase structure. |
| 106 | - Plan invented with no reference to user requests or research. |
| 107 | priority: medium |
| 108 | contract_ref: "agent §Plan Creation (reference research and user requests)" |
| 109 | |
| 110 | - expectation_id: no-source-edit |
| 111 | summary: Planning-only — no edits to source code or build manifests. |
| 112 | signal: Output does not reference modifications to source-tree files. |
| 113 | pass_criteria: | |
| 114 | No occurrences of edit/create verbs paired with `.cs`/`.py`/`.ts`/`.js`/ |
| 115 | `.go`/`.rs`/`.java`/`package.json`/`pyproject.toml`/`Cargo.toml` paths. |
| 116 | Implementation steps appear in the plan, not as claimed edits. |
| 117 | failure_modes: |
| 118 | - Agent claims to have started implementing the plan. |
| 119 | - Modifies build manifests while planning. |
| 120 | priority: high |
| 121 | contract_ref: "agent scope (planning-only); current `no-source-edit` grader" |
| 122 | |