# PowerShell Dispatcher Benchmark - Baseline Results

**Date:** 2026-04-30
**Test Environment:** Windows, TypeAgent dispatcher with grammar-based routing

## Summary

| Test Mode                            | Pass Rate | Passed | Total |
| ------------------------------------ | --------- | ------ | ----- |
| **Competition Mode** (all agents)    | **85.7%** | 30     | 35    |
| **Isolation Mode** (PowerShell only) | **0%**    | 0      | 125   |

## Competition Mode Results (grammar-competition)

### Passed Tests (30/35)

| Category           | Utterances                                                           | Status                       |
| ------------------ | -------------------------------------------------------------------- | ---------------------------- |
| Network ping       | `ping google.com`                                                    | PASS                         |
| File search        | `grep error in logs`, `find TODO in src`                             | PASS                         |
| Processes          | `show running processes`, `kill notepad`                             | PASS                         |
| Start process      | `start notepad`, `launch calculator`, `run cmd`                      | PASS                         |
| System info        | `show system information`, `show disk usage`, `show uptime`          | PASS                         |
| Services           | `list all services`, `restart service spooler`                       | PASS                         |
| Network utils      | `ipconfig`, `netstat`, `nslookup google.com`                         | PASS                         |
| Archives           | `zip logs`, `unzip backup.zip`                                       | PASS                         |
| Data ops           | `read csv data.csv`, `convert data.csv to json`                      | PASS                         |
| Unix aliases       | `cat readme.md`, `rm temp.txt`, `cp file.txt backup.txt`, `ps`, `df` | PASS                         |
| Negative: music    | `play some music`, `play shake it off`                               | PASS (correctly NOT matched) |
| Negative: calendar | `schedule a meeting for tomorrow`, `what's on my calendar`           | PASS                         |
| Negative: email    | `send an email to john`                                              | PASS                         |

### Failed Tests (5/35)

| Utterance                 | Expected                  | Actual                         | Root Cause                                                            |
| ------------------------- | ------------------------- | ------------------------------ | --------------------------------------------------------------------- |
| `list files in downloads` | `listFiles`               | `listFiles`                    | **Test data bug**: Expected `findLargeFiles` due to flow name mapping |
| `show files in documents` | `listFiles`               | `listFiles`                    | **Test data bug**: Same as above                                      |
| `ls`                      | `listFiles`               | `listFiles`                    | **Test data bug**: Same as above                                      |
| `mv old.txt new.txt`      | `moveFile` / `powershell` | `readFile` / `utility`         | **Grammar gap**: `mv` alias not properly mapped                       |
| `check my inbox`          | not powershell            | `serviceStatus` / `powershell` | **Disambiguation issue**: "check" is too broad                        |

### Analysis

1. **3 false failures** - Test data expects `findLargeFiles` but correct action is `listFiles`
2. **1 grammar gap** - `mv` command not routing to `moveFile` action
3. **1 disambiguation issue** - "check my inbox" incorrectly matches PowerShell's `serviceStatus`

**Corrected pass rate:** 33/35 = **94.3%** (excluding test data bugs)

## Isolation Mode Results (grammar-subschemas)

### Status: All 125 tests failed

The grammar IS matching correctly (logs show `powershell.listFiles - Result: ACCEPTED`), but evaluation fails because:

- **Isolation mode schema format:** `powershell.listFiles`
- **Expected sub-schema format:** `powershell-files` with action `listFiles`

This is a **configuration issue** with how the dispatcher loads schemas in isolation mode, not a grammar problem.

## Component Accuracy (Competition Mode)

| Component        | Accuracy      |
| ---------------- | ------------- |
| Grammar matching | 91.4% (64/70) |

## Test Artifacts

Results written to: `benchmark/results/2026-04-30T1650/`

- `scorecard.json` - Machine-readable results
- `details.md` - Full test details with traces

## Recommendations

1. **Fix test data** - Update `grammar-competition.json` to expect `listFiles` instead of `findLargeFiles`
2. **Fix isolation mode** - Update dispatcher config to properly load sub-schemas
3. **Add `mv` grammar pattern** - Ensure move command routes correctly
4. **Improve "check" disambiguation** - Make `serviceStatus` grammar more specific
