# PowerShell Dispatcher Benchmark - Results

**Date:** 2026-05-04
**Test Environment:** Windows, TypeAgent dispatcher with grammar-based routing

## Summary

| Test Mode                                        | Pass Rate | Passed | Total |
| ------------------------------------------------ | --------- | ------ | ----- |
| **Competition Mode** (all agents)                | **91.4%** | 32     | 35    |
| **Isolation Mode** (PowerShell only, no samples) | **92.0%** | 115    | 125   |

## Fixes Applied Since Last Baseline

### 1. Schema Name Configuration (Critical)

**File:** `benchmark/run-benchmark.mts`

Schema names changed from short form to fully qualified form:

```typescript
// Before (broken)
schemas: ["powershell", "powershell-files", "powershell-processes", ...]

// After (working)
schemas: ["powershell", "powershell.powershell-files", "powershell.powershell-processes", ...]
```

### 2. Browser Agent Initialization Error

**File:** `packages/agents/browser/src/agent/browserActionHandler.mts`

Added null check for `sessionDir` in `createViewServiceHost` to prevent TypeError when session directory is unavailable.

### 3. Sample Flow Control

**Files:** `benchmark/run-benchmark.mts`, `src/actionHandler.mts`

Added `--no-samples` CLI flag and `TYPEAGENT_NO_SAMPLES` environment variable to disable sample recipe seeding during benchmark runs, allowing namespace schema grammars to be tested in isolation.

## Competition Mode Results (grammar-competition)

**Pass Rate:** 91.4% (32/35)

### Failed Tests (3/35)

| Utterance            | Expected                  | Actual          | Root Cause                   |
| -------------------- | ------------------------- | --------------- | ---------------------------- |
| `mv old.txt new.txt` | `moveFile` / `powershell` | Other agent     | Grammar pattern gap          |
| `check my inbox`     | not powershell            | `serviceStatus` | "check" is too broad         |
| Various              | Test data issues          | -               | Flow name mapping mismatches |

## Isolation Mode Results (grammar-subschemas)

**Pass Rate:** 92.0% (115/125)

### Component Accuracy

| Component            | Accuracy  | Details         |
| -------------------- | --------- | --------------- |
| Grammar matching     | **97.6%** | 122/125 correct |
| Parameter extraction | **90.8%** | 109/120 correct |

### Passed Tests by Namespace

| Namespace                   | Pass Rate    | Notes                           |
| --------------------------- | ------------ | ------------------------------- |
| Files (listFiles)           | 80% (4/5)    | 1 parameter extraction issue    |
| Files (readFile)            | 100% (5/5)   | All patterns working            |
| Files (writeFile)           | 100% grammar | Parameter quote handling issues |
| Files (copyFile)            | 100% (3/3)   | All patterns working            |
| Files (moveFile)            | 100% (3/3)   | All patterns working            |
| Files (deleteFile)          | 100% (3/3)   | All patterns working            |
| Files (testPath)            | 100% (3/3)   | All patterns working            |
| Files (findText)            | 67% (2/3)    | 1 pattern/path split issue      |
| Files (newItem)             | 100% (4/4)   | All patterns working            |
| Processes (list/stop/start) | 92%          | 2 grammar mismatches            |
| System                      | 100% (16/16) | All patterns working            |
| Services                    | 100% (15/15) | All patterns working            |
| Network                     | 92%          | 2 minor issues                  |
| Data                        | 100% (10/10) | All patterns working            |
| Archives                    | 88% (7/8)    | 1 parameter extraction issue    |

### Failed Tests (10/125)

| Scenario    | Utterance                         | Issue Type | Details                                 |
| ----------- | --------------------------------- | ---------- | --------------------------------------- |
| gs-files-01 | "show me the files in documents"  | Parameter  | Path captured extra words               |
| gs-files-03 | "write 'hello world' to test.txt" | Parameter  | Quote handling                          |
| gs-files-03 | "save 'test data' to output.txt"  | Parameter  | Quote handling                          |
| gs-files-03 | "append 'new line' to log.txt"    | Parameter  | Quote handling                          |
| gs-files-08 | "grep TODO src"                   | Parameter  | Pattern/path split                      |
| gs-proc-02  | "show top 5 memory hogs"          | Grammar    | Matched `listFiles` not `processMemory` |
| gs-proc-03  | "show top 10 cpu hogs"            | Grammar    | Matched `listFiles` not `processCpu`    |
| gs-net-04   | "show my ip address"              | Grammar    | Matched `listFiles` not `ipConfig`      |
| gs-net-05   | "resolve gmail.com MX"            | Parameter  | Type suffix not parsed                  |
| gs-arch-02  | "extract archive.zip to output"   | Parameter  | Path extraction issue                   |

### Root Cause Analysis

1. **Grammar Conflicts (3 failures):** Phrases like "show top 5" and "show my" match the broad `listFiles` pattern before more specific patterns
2. **Greedy Wildcards (4 failures):** Wildcard captures consume more tokens than intended
3. **Quote Handling (3 failures):** Content parameters include surrounding quotes that tests expect stripped

## Test Artifacts

Results written to: `benchmark/results/2026-05-04T1728/`

- `scorecard.json` - Machine-readable results
- `details.md` - Full test details with traces

## Recommendations

### High Priority

1. **Add specificity to memory/cpu patterns** - Ensure "show top N memory/cpu" routes to process namespace
2. **Fix "show my" grammar conflict** - "show my ip" should match `ipConfig` not `listFiles`

### Medium Priority

3. **Improve wildcard boundaries** - Add stop words to prevent greedy matching
4. **Handle quoted content** - Strip quotes from content parameters in grammar

### Low Priority

5. **DNS type parsing** - Support "resolve X MX" syntax with type parameter
6. **Archive path extraction** - Improve "extract X to Y" pattern

## Comparison with Previous Baseline

| Metric           | 2026-04-30 | 2026-05-04 | Change |
| ---------------- | ---------- | ---------- | ------ |
| Competition Mode | 85.7%      | 91.4%      | +5.7%  |
| Isolation Mode   | 0%         | 92.0%      | +92.0% |
| Grammar Accuracy | 91.4%      | 97.6%      | +6.2%  |

The isolation mode is now functional and achieves comparable accuracy to competition mode, validating that namespace schema grammars work correctly when properly configured.
