microsoft/TypeAgent

Commit

actionGrammar: dedup dispatch tables, expand fuzzer, fix separator-in-literal dispatch bug (#2271)

Compacts the serialized grammar JSON, expands the fuzzer, and fixes a
dispatch bug surfaced by the new fuzz coverage. Scoped entirely to the
`actionGrammar` package.

## Serialized grammar size

Player grammar `.ag.json` drops from 83298 to 67959 bytes (~18% smaller)
via two structural changes:

- **Shared `dispatches` pool.** Dispatch tables move into a
grammar-level pool indexed by `RulesPart.dispatch` / `Grammar.dispatch`,
mirroring the existing `rules` pool. The optimizer's per-input-identity
sharing of dispatch tables (e.g. multiple references to a named rule
whose body was dispatched) now survives a serialize/deserialize round
trip instead of being inlined per site.
- **Omit empty `index`.** When a `RulesPart`'s alternatives array is
empty (typical for a fully-dispatched part with no fallback), the
serializer omits the `index` field entirely. The deserializer
substitutes a frozen shared empty-array sentinel.

This is a **breaking JSON format change**: cached `.ag.json` files
written by the previous serializer will not load with the new
deserializer. All in-tree `.ag.json` artifacts are build outputs and
regenerate on a clean build. Any consumer persisting compiled grammars
to user data should clear the cache after upgrading. The deserializer
surfaces stale-file load failures as recognizable errors rather than
silent corruption.

## Dispatch bug fix

In `classifyDispatchMember`, the dispatch bucket key in `required` mode
was the full literal string, but `peekNextToken` truncates input at the
first separator. A literal like `"d?"` bucketed under `"d?"` while peek
returned `"d"` for input `"d? ..."`, causing a hash miss and dropping
the match.

Fix: derive the bucket key from `leadingNonSeparatorRun(literal)`,
mirroring what peek returns. Two consequences:
1. **Key alignment** restored: the matcher now finds the bucket.
2. **Bucket collapse**: literals like `"d?"`, `"d!"`, `"d."` all share
bucket `"d"`. This is correct - peek routes all such inputs to the same
bucket, and the member rules' StringPart regexes discriminate among them
after routing.

A literal that starts with a separator goes to the fallback subset (no
non-empty bucket key is derivable).

The leading-non-separator-run regex now lives once in
`grammarMatcher.ts` (exported as `leadingNonSeparatorRun`) and is used
by both the matcher's hot path and the optimizer, so the two cannot
drift.

## Fuzzer expansion

Five new dimensions in `FuzzFeatureFlags`:
- Rich value expressions (15 shapes covering the full operator table:
`??`, `||`, `&&`, comparisons, unary, member access, optional chaining,
spread, multi-substitution templates).
- Comment injection (line/block) at part-gaps, alt-gaps, rule-leading.
- Escape sequences in literals (identity, hex, unicode, brace forms).
- Nested rule captures `$(name:<RuleName>)`.
- Separator chars embedded in literals - the dimension that surfaced the
dispatch bug above.

Plus optimizer-stress knobs targeting each of the four optimizer passes
(inline / factor / tail-factor / dispatchify), and a per-variant
equivalence harness that runs each pass in isolation so a regression
surfaces in its own variant rather than ambiguously across the
recommended set.

`DEFAULT_FEATURES` now enables matcher-invariant or cheap
parser/writer-only knobs at low probability so the broad fuzz pass
exercises them. Shape-changing knobs stay at 0 (covered by targeted
`fuzzDescribe` blocks for clean attribution); a decision rubric for
future contributors is documented inline.

## Tests

11,007 tests pass. New regression coverage:
- `grammarOptimizerDispatch.spec.ts`: required-mode separator-in-literal
sweep across 13 separator chars; leading-separator literal lands in
fallback.
- `grammarOptimizerSharing.spec.ts`: dispatch-table dedup into the pool
and round-trip identity restoration.
- `grammarFuzz.spec.ts`: 5 new `fuzzDescribe` blocks for the new
generator dimensions.