microsoft/hve-core
Publicmirrored from https://github.com/microsoft/hve-coreAvailable
evals/agent-behavior/eval.yaml
67lines · modecode
| 1 | name: agent-behavior |
| 2 | description: > |
| 3 | Evaluate hve-core skill+agent behavior via copilot-sdk. Tests that the |
| 4 | combination of skills loaded in an agent context produces correct structure, |
| 5 | applies specialized perspectives, and stays within defined boundaries. |
| 6 | Note: Tests skill behavior under agent-style prompts rather than invoking |
| 7 | a specific .agent.md file directly (Vally does not yet support agent routing). |
| 8 | type: capability |
| 9 | config: |
| 10 | runs: 3 |
| 11 | timeout: 120s |
| 12 | executor: copilot-sdk |
| 13 | |
| 14 | environment: |
| 15 | skills: |
| 16 | - ../../.github/skills/security/owasp-top-10 |
| 17 | - ../../.github/skills/coding-standards/python-foundational |
| 18 | |
| 19 | scoring: |
| 20 | threshold: 0.7 |
| 21 | |
| 22 | stimuli: |
| 23 | - name: pr-review-identifies-security |
| 24 | prompt: | |
| 25 | Review this code change for a Python web application: |
| 26 | ```python |
| 27 | @app.route('/user/<id>') |
| 28 | def get_user(id): |
| 29 | query = f"SELECT * FROM users WHERE id = {id}" |
| 30 | return db.execute(query).fetchone() |
| 31 | ``` |
| 32 | Focus on security and code quality. |
| 33 | tags: |
| 34 | category: agent-behavior |
| 35 | agent: pr-review |
| 36 | graders: |
| 37 | - type: output-matches |
| 38 | name: identifies-sql-injection |
| 39 | config: |
| 40 | pattern: "(?i)\\bsql\\s*injection\\b|\\binjection\\b" |
| 41 | - type: output-matches |
| 42 | name: provides-remediation |
| 43 | config: |
| 44 | pattern: "(?i)parameterized|prepared|placeholder|bind" |
| 45 | |
| 46 | - name: pr-review-identifies-error-handling |
| 47 | prompt: | |
| 48 | Review this code change: |
| 49 | ```python |
| 50 | def process_payment(amount): |
| 51 | response = requests.post(PAYMENT_API, json={"amount": amount}) |
| 52 | return response.json()["transaction_id"] |
| 53 | ``` |
| 54 | What issues do you see? |
| 55 | tags: |
| 56 | category: agent-behavior |
| 57 | agent: pr-review |
| 58 | graders: |
| 59 | - type: output-matches |
| 60 | name: identifies-missing-error-handling |
| 61 | config: |
| 62 | pattern: "(?i)error.handling|exception|try|status.code|timeout" |
| 63 | - type: output-matches |
| 64 | name: identifies-missing-validation |
| 65 | config: |
| 66 | # cspell:disable-next-line |
| 67 | pattern: "(?i)validat|check|verify|amount|negative" |
| 68 | |