microsoft/TypeAgent
Publicmirrored fromhttps://github.com/microsoft/TypeAgentAvailable
ts/README.md
358lines · modecode
| 1 | # TypeAgent Typescript Code |
| 2 | |
| 3 | ## Overview |
| 4 | |
| 5 | **TypeAgent** is **sample code** that explores an architecture for building a _personal agent_ with _natural language interfaces_ leveraging current advances in LLM technology. |
| 6 | |
| 7 | This directory contains Typescript implemented packages and main entry point for **TypeAgent**. For more details about the project, please review the TypeAgent [ReadMe](./../README.md). |
| 8 | |
| 9 | The main entry point to explore TypeAgent is the [TypeAgent Shell](./packages/shell) example. Currently, we only support running from the repo (i.e. no published/installable builds). Follow the instruction below to [build](#build) and [run](#running) the [TypeAgent Shell](./packages/shell) example. |
| 10 | |
| 11 | ## Build |
| 12 | |
| 13 | ### Setup |
| 14 | |
| 15 | To build: |
| 16 | |
| 17 | - Install [Node 20+](https://nodejs.org/en/download) |
| 18 | - NOTE: HPC Tools conflict with node so be sure that the node.exe you are running is the correct one! |
| 19 | - Install pnpm (`npm i -g pnpm && pnpm setup`) |
| 20 | - **(Linux/WSL Only)** Read TypeAgent Shell's [README.md](./packages/shell/README.md) for additional requirements |
| 21 | |
| 22 | ### Steps |
| 23 | |
| 24 | In this directory: |
| 25 | |
| 26 | - Run `pnpm i` |
| 27 | - Run `pnpm run build` |
| 28 | |
| 29 | ### Agent Specific Steps (Optional) |
| 30 | |
| 31 | #### VSCode Agent |
| 32 | |
| 33 | If you want to `deploy` the **VS Code** extension `CODA` locally please run: |
| 34 | |
| 35 | - From the root, `cd ./ts/packages/coda` |
| 36 | - `pnpm run deploy:local` |
| 37 | |
| 38 | You should now be able to access the extension from VS Code. |
| 39 | |
| 40 | #### Desktop Agent (Windows only) |
| 41 | |
| 42 | To use the [Desktop Agent](./packages/agents/desktop/) for windows, follow the instruction in [README.md](./packages/agents/desktop/README.md) to build the [AutoShell](../dotnet/autoShell/) C# code necessary to interact with the OS. |
| 43 | |
| 44 | ### Local Whisper Service (Optional) |
| 45 | |
| 46 | If you want to use a local whisper service for voice input in the [TypeAgent Shell](./packages/shell), please follow instruction in the [README.md](../python/stt/whisperService/README.md) in the python's [whisperService](../python/stt/whisperService/) directory. |
| 47 | |
| 48 | ## Running Prerequisites |
| 49 | |
| 50 | ### Service Keys |
| 51 | |
| 52 | Multiple services are required to run the scenarios. Configuration is stored in a YAML file at this directory (TypeAgent repo's `./ts` directory). Copy `config.sample.yaml` to `config.local.yaml` and fill in the necessary keys. Legacy `.env` files are still supported but deprecated and will stop working after September 2026. |
| 53 | |
| 54 | Here is an example of the minimal `config.local.yaml` targeting Azure: |
| 55 | |
| 56 | ```yaml |
| 57 | azureOpenAI: |
| 58 | defaultAuth: <service key or "identity" for keyless> |
| 59 | deployments: |
| 60 | default: |
| 61 | endpoint: <endpoint URL for LLM model, e.g. GPT-4o> |
| 62 | responseFormat: true |
| 63 | endpoints: |
| 64 | embedding: |
| 65 | endpoint: <endpoint URL for text-embedding-ada-002 or equivalent> |
| 66 | ``` |
| 67 | |
| 68 | Here is an example of the minimal `config.local.yaml` targeting OpenAI: |
| 69 | |
| 70 | ```yaml |
| 71 | openAI: |
| 72 | default: |
| 73 | organization: <organization id> |
| 74 | apiKey: <service key> |
| 75 | endpoint: https://api.openai.com/v1/chat/completions |
| 76 | model: gpt-4o |
| 77 | responseFormat: true |
| 78 | embedding: |
| 79 | endpoint: https://api.openai.com/v1/embeddings |
| 80 | model: text-embedding-ada-002 |
| 81 | ``` |
| 82 | |
| 83 | The follow set of functionality will need the services keys. Please read the links for details about the variables needed. It is possible to use "keyless" configuration for some APIs. See [Keyless API Access](#keyless-api-access) below. |
| 84 | |
| 85 | **Minimum requirements** to try out the experience with the [List](./packages/agents/list/README.md) TypeAgent: |
| 86 | |
| 87 | | Requirements | Functionality | Variables | Instructions | Keyless Access Supported | |
| 88 | | ------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | |
| 89 | | LLM (GPT-4 or equivalent) | Request translation | AZURE_OPENAI_API_KEY<br>AZURE_OPENAI_ENDPOINT<br>AZURE_OPENAI_RESPONSE_FORMAT<br>or<br>OPENAI_API_KEY<br>OPENAI_ORGANIZATION<br>OPENAI_ENDPOINT<br>OPENAI_MODEL<br>OPENAI_RESPONSE_FORMAT | [TypeChat instruction](https://github.com/microsoft/TypeChat/tree/main/typescript/examples#step-3-configure-environment-variables). | Yes | |
| 90 | | Embeddings | Conversation Memory<br><br>[Desktop](./packages/agents/desktop/) App name Fuzzy match | AZURE_OPENAI_API_KEY_EMBEDDING<br>AZURE_OPENAI_ENDPOINT_EMBEDDING<br>or<br> OPENAI_ENDPOINT_EMBEDDING<br>OPENAI_MODEL_EMBEDDING<br>OPENAI_API_KEY_EMBEDDING (optional if different from OPENAI_API_KEY) | | Yes | |
| 91 | |
| 92 | **Optional requirements** |
| 93 | |
| 94 | | Requirements | Functionality | Variables | Instructions | Keyless Access Supported | |
| 95 | | ----------------------------------------------------------------------------------------------------------------- | ------------------------ | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------ | |
| 96 | | [Speech to Text service](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/index-speech-to-text) | Voice input (shell only) | SPEECH_SDK_ENDPOINT<br>SPEECH_SDK_KEY<br>SPEECH_SDK_REGION | [Shell setup instruction](./packages/shell/README.md#azure-speech-to-text-service-optional) | Yes | |
| 97 | |
| 98 | **Additional keys required for individual AppAgents** (Optional if not using these AppAgents) |
| 99 | |
| 100 | | Requirements | Functionality | Variables | Instructions | Keyless Access Supported | |
| 101 | | ---------------------------------------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------ | -------------------------------------------------------------------------- | ------------------------ | |
| 102 | | Grounding with Bing | Chat Lookup | BING_WITH_GROUNDING_ENDPOINT<br>BING_WITH_GROUNDING_AGENT_ID | [Internet lookups](./packages/dispatcher/src/context/dispatcher/README.md) | No | |
| 103 | | GPT-3.5 Turbo | Fast Chat Response<br>[Email](./packages/agents/email) content generation | AZURE_OPENAI_API_KEY_GPT_35_TURBO<br>AZURE_OPENAI_ENDPOINT_GPT_35_TURBO | | Yes | |
| 104 | | [Spotify Web API](https://developer.spotify.com/documentation/web-api) | [Music player](./packages/agents/player/) | SPOTIFY_APP_CLI<br>SPOTIFY_APP_CLISEC<br>SPOTIFY_APP_PORT | [Music player setup](./packages/agents/player/README.md#application-keys) | No | |
| 105 | | [Graph Application](https://developer.microsoft.com/en-us/graph) | [Calendar](./packages/agents/calendar/)/[Email](./packages/agents/email) | MSGRAPH_APP_CLIENTID<br>MSGRAPH_APP_CLIENTSECRET<br>MSGRAPH_APP_TENANTID | | No | |
| 106 | | GPT-4o | [Browser](./packages/agents/browser/) - Crossword Page | AZURE_OPENAI_API_KEY_GPT_4_O<br>AZURE_OPENAI_ENDPOINT_GPT_4_O | | Yes | |
| 107 | |
| 108 | Other examples in the [example directory](./examples/) may have additional service keys requirements. See the README in those examples for more detail. |
| 109 | |
| 110 | Read the [Debugging](#debugging) section for additional service keys that can be used for debugging. |
| 111 | |
| 112 | #### Using Azure Key Vault to manage keys |
| 113 | |
| 114 | The [getKey](./tools/scripts/getKeys.mjs) script is created for developer convenience to manage service secret using Azure Key Vault and set up the local development environments. |
| 115 | |
| 116 | To setup: |
| 117 | |
| 118 | - Install the latest [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) |
| 119 | - Run `az login` to login using the CLI. |
| 120 | - Run `az account set --subscription <Subscription Id>` to set the subscription. |
| 121 | - [Create a Azure Key Vault](https://learn.microsoft.com/en-us/azure/key-vault/general/quick-create-cli) with name `<name>`. |
| 122 | |
| 123 | To update keys on the key vault: |
| 124 | |
| 125 | - Add or change the values in `config.local.yaml` (or `.env` for legacy setups) |
| 126 | - Add new keys name in `tools/scripts/getKeys.config.json` |
| 127 | - Run `npm run getKeys -- push [--vault <name>]`. (If the `--vault` option is omitted, the default from vault name in `tools/scripts/getKeys.config.json` is used.) |
| 128 | - Check in the changes to `tools/scripts/getKeys.config.json` |
| 129 | |
| 130 | To get the required config and keys saved to `config.local.yaml` under the `ts` folder: |
| 131 | |
| 132 | - Run `npm run getKeys [--vault <name>]` at the root to pull secret from the key vault with `<name>`. (If the `--vault` option is omitted, the default from vault name in `tools/scripts/getKeys.config.json` is used.) |
| 133 | |
| 134 | Note: Shared keys don't include Spotify integration, which can be created using the [Spotify API keys instructions](./packages/agents/player/README.md) |
| 135 | |
| 136 | ### Keyless API Access |
| 137 | |
| 138 | For additional security, it is possible to run a subset of the TypeAgent endpoints in a keyless environment and use Azure Entra user identities to authenticate against endpoint. |
| 139 | |
| 140 | In order to use keyless access you must also configure your services to use [RBAC](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview) and assign users access to the correct roles for each endpoint. Please see the tables above to determine keyless endpoint support. |
| 141 | |
| 142 | After configuring your service, modify `config.local.yaml` and specify `identity` as the `defaultAuth` value instead of using keys in the examples provided. To authenticate at runtime, make sure you have installed [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) and logged in with the account that has access to the models using `az login`. Make sure choose the subscription where your services are as the default. |
| 143 | |
| 144 | Alternatively, TypeAgent make use of the Azure SDK's [DefaultAzureCredential](https://learn.microsoft.com/en-us/javascript/api/%40azure/identity/defaultazurecredential?view=azure-node-latest) which provide other methods to authenticate at runtime. Follow those instructions to provide keyless access to the services. |
| 145 | |
| 146 | ### Just-in-time Access |
| 147 | |
| 148 | TypeAgent also supports least privileged security approach using [Azure Entra Privileged Identity Management](https://learn.microsoft.com/en-us/entra/id-governance/privileged-identity-management/pim-configure). [Elevate.js](./tools/scripts/elevate.js) is a script used to automate elevation. Default configuration |
| 149 | options for elevation (duration, justification message, etc.) are stored in `tools/scripts/elevate.config.json`. A typical developer workflow is to run `npm run elevate` once at the beginning of each workday. |
| 150 | |
| 151 | To learn more about JIT access: [start here](https://techcommunity.microsoft.com/t5/microsoft-entra-blog/just-in-time-access-to-groups-and-conditional-access-integration/ba-p/2466926). |
| 152 | |
| 153 | ### Linux/WSL |
| 154 | |
| 155 | For TypeAgents that operates on the Microsoft Graph (e.g. [Calendar](./packages/agents/calendar/) and [Email](./packages/agents/email/)), they leverage [@azure/identity](https://www.npmjs.com/package/@azure/identity) for authentication and use [@azure/identity-cache-persistence](https://www.npmjs.com/package/@azure/identity-cache-persistence) to cache on the local machine. |
| 156 | |
| 157 | Install the following packages if you are on Linux or WSL2 environment (please `restart` the shell after running the commands below): |
| 158 | |
| 159 | ```shell |
| 160 | sudo apt-get update |
| 161 | sudo apt install -y gnome-keyring |
| 162 | ``` |
| 163 | |
| 164 | After the step above, you will need to enter a password to protect the secrets in the keyring. The popup normally appears when you restart the shell and run the code that needs to persists secrets in the keyring. |
| 165 | |
| 166 | ## Running |
| 167 | |
| 168 | There are two main apps to start exploring TypeAgent: [TypeAgent Shell](#shell) and [TypeAgent CLI](#cli). Both provides _interactive agents_ with _natural language interfaces_ experience via a shared package [dispatcher](./packages/dispatcher/) that implemented core TypeAgent functionalities. Currently, we only support running from the repo (i.e. no published/installable builds). |
| 169 | |
| 170 | ### Shell |
| 171 | |
| 172 | [TypeAgent Shell](./packages/shell) provides a light weight GUI _interactive agents_ with _natural language interfaces_ experience |
| 173 | |
| 174 | - Run `pnpm run shell`. |
| 175 | |
| 176 | Also, you can go to the shell directory `./packages/shell` and start from there. Please see instruction in TypeAgent Shell's [README.md](./packages/shell/README.md). |
| 177 | |
| 178 | ### CLI |
| 179 | |
| 180 | [TypeAgent CLI](./packages/cli) provides a console based _interactive agents_ with _natural language interfaces_ experience. Additional console command is available to explore different part of TypeAgent functionalities. |
| 181 | |
| 182 | - Run `pnpm run cli` to get the available command |
| 183 | - Run `pnpm run cli -- connect` will start the interactive prompt (connecting to the agent server via WebSocket RPC) |
| 184 | |
| 185 | Also, you can go to the CLI directory `./packages/cli` and start from there. Please see instruction in TypeAgent CLI's [README.md](./packages/cli/README.md) for more options and detail. |
| 186 | |
| 187 | ## Development |
| 188 | |
| 189 | ### Main packages and directory structure |
| 190 | |
| 191 | Apps: |
| 192 | |
| 193 | - [agent-cli](./packages/cli): TypeAgent dispatcher CLI and interactive prompt |
| 194 | - [agent-shell](./packages/shell): TypeAgent shell UI |
| 195 | |
| 196 | Libraries: |
| 197 | |
| 198 | - [agent-dispatcher](./packages/dispatcher): TypeAgent dispatcher used by both the CLI and shell |
| 199 | - [agent-cache](./packages/cache): Construction explanation and cache |
| 200 | |
| 201 | [Agents with natural language interfaces](./packages/agents): |
| 202 | |
| 203 | - [Music Player](./packages/agents/player/): Spotify music player TypeAgent plugin |
| 204 | - [Chat](./packages/agents/chat/) |
| 205 | - [Browser](./packages/agents/browser/) |
| 206 | - [VS Code](./packages/agents/code/) |
| 207 | - [List Management](./packages/agents/list/) |
| 208 | - [Calendar](./packages/agents/calendar/) |
| 209 | - [Email](./packages/agents/email/) |
| 210 | - [Desktop](./packages/agents/desktop/) |
| 211 | |
| 212 | Other directories: |
| 213 | |
| 214 | - [examples](./examples/): various additional standalone explorations. |
| 215 | - [tools](./tools/): tools for CI/CD and internal development environments. |
| 216 | |
| 217 | ### Testing |
| 218 | |
| 219 | All commands run from the `ts/` directory using `pnpm`. |
| 220 | |
| 221 | ```bash |
| 222 | pnpm run test:local # Unit tests (*.spec.ts) — no API keys required |
| 223 | pnpm run test:live # Integration tests (*.test.ts) — requires API keys in ts/config.local.yaml |
| 224 | pnpm run test # Both local + live + shell tests |
| 225 | ``` |
| 226 | |
| 227 | To run tests for a single package: |
| 228 | |
| 229 | ```bash |
| 230 | pnpm --filter <package-name> test |
| 231 | # e.g., pnpm --filter action-grammar test |
| 232 | ``` |
| 233 | |
| 234 | To run a single test file or test by name (run from the package directory): |
| 235 | |
| 236 | ```bash |
| 237 | pnpm run jest-esm --testPathPattern="merge.spec.js" |
| 238 | pnpm run jest-esm --testNamePattern="your test name" |
| 239 | ``` |
| 240 | |
| 241 | > **Note:** Tests run against compiled output in `dist/test/`. Run `pnpm run build` before running tests. |
| 242 | |
| 243 | #### Schema Changes |
| 244 | |
| 245 | If a new translator or explainer is added, or if any translator/explainer schema changes, the built-in construction cache and test data need to be regenerated and evaluated for correctness. |
| 246 | |
| 247 | Test data are located in the [defaultAgentProvider](./packages/defaultAgentProvider)'s [test/data](./packages/defaultAgentProvider/test/data) directory. Each test data file corresponds to a specific translator and explainer. |
| 248 | |
| 249 | Use the `agent-cli data add` command to add new test cases. |
| 250 | |
| 251 | To regenerate, run the following from the root or in the [cli](./packages/cli) directory: |
| 252 | |
| 253 | - `pnpm run regen:builtin` — Regenerate the builtin construction store. |
| 254 | - `pnpm run regen` — Regenerate test data. |
| 255 | |
| 256 | To evaluate correctness of the test data: |
| 257 | |
| 258 | - `agent-cli data diff <file>` — Open a test data file diff in VS Code. |
| 259 | - Review the translation for correctness (can be skipped if the translator schema didn't change). |
| 260 | - Run `pnpm run test` to verify the generated test data round-trips correctly (also runs in CI). |
| 261 | - Check the stats before and after regen: |
| 262 | - `pnpm run regen -- -- --none` prints per-file and total stats. |
| 263 | - Ensure explanation failure counts stay roughly the same or improve. |
| 264 | - Ensure attempt/correction ratios stay roughly the same or improve. |
| 265 | - Check whether failures are due to LLM instability: |
| 266 | - Borderline failure: was there significant correction before, and a failure now? |
| 267 | - Run `agent-cli explain --repeat 5 <RequestAction>` before and after to compare stats. |
| 268 | |
| 269 | ### Linting |
| 270 | |
| 271 | The repo is set up with prettier to help with consistent code style. Run `npm run lint` to check and `npm run lint:fix` to fix any issues. |
| 272 | |
| 273 | ### Debugging |
| 274 | |
| 275 | #### Starting Development version of TypeAgent CLI |
| 276 | |
| 277 | Go to `./packages/cli`, you don't have to build and just run `./bin/dev.js`. It will use `ts-node` and build the typescript as it goes. |
| 278 | |
| 279 | #### Launching from VSCode |
| 280 | |
| 281 | If you open this directory as a workspace in VSCode, multiple launch task is defined to quickly start debug. |
| 282 | |
| 283 | Common Debug Launch Task: |
| 284 | |
| 285 | - CLI connect - `./package/cli/bin/run.js connect` |
| 286 | - CLI (dev) connect - `./package/cli/bin/dev.js connect` with a new command prompt |
| 287 | - CLI (dev) connect [Integrated Terminal] - `./bin/dev.js connect` using VSCode terminal (needed for WSL) |
| 288 | |
| 289 | #### Attaching to running sessions |
| 290 | |
| 291 | To attaching to an existing session with TypeAgent CLI's connect mode or TypeAgent Shell, you can start inspector by issuing the command `@debug` and use the VSCode `Attach` debugger launch task to attach. |
| 292 | |
| 293 | #### TypeAgent Shell Browser Process |
| 294 | |
| 295 | With the TypeAgent Shell, press F12 will bring up the devtool to debug the browser process. |
| 296 | |
| 297 | ### Tracing |
| 298 | |
| 299 | The project uses [debug](https://www.npmjs.com/package/debug) package to enable tracing. There are two options to enable these traces: |
| 300 | |
| 301 | **Option 1**: Set the namespace pattern in the environment variable `DEBUG=typeagent:prompt` before starting the program. |
| 302 | |
| 303 | For example (in Linux), to trace the GPT prompt that get sent when running the interactive CLI. |
| 304 | |
| 305 | ```bash |
| 306 | DEBUG=typeagent:prompt packages/cli/bin/run.js connect |
| 307 | ``` |
| 308 | |
| 309 | **Option 2**: In the shell or CLI's interactive mode, you can issue the command `@trace <pattern>` to add to the list of namespace. Use "-" or "-\*" to disable all the trace. |
| 310 | |
| 311 | For example inside the CLI's interactive mode, enter |
| 312 | |
| 313 | ```txt |
| 314 | @trace *:prompt:* |
| 315 | ``` |
| 316 | |
| 317 | Search the code base with '"typeagent:' will give all the traces available. |
| 318 | |
| 319 | ### Logging |
| 320 | |
| 321 | TypeAgent does not collect telemetry by default. Developer can enable logging to a mongodb for internal debugging purpose by providing a mongodb connection string with the `MONGODB_CONNECTION_STRING` variable in `config.local.yaml` (under `storage.mongo.connectionString`) or the legacy `.env` file. |
| 322 | |
| 323 | ### Experiment with Local LLM via Ollama |
| 324 | |
| 325 | **NOTE**: TypeAgent is current only tuned to run on GPT4 and similar model. Reliability and quality may vary using other smaller LLM models. |
| 326 | |
| 327 | TypeAgent's shell and CLI will detect if Ollama is running locally via default port (11434) and expose those model to be used. |
| 328 | To use ollama, install ollama and pull some model locally. TypeAgent will automatically detect them. |
| 329 | |
| 330 | In the interactive mode (cli or shell), you can also change the translation (and explainer) model using the commands |
| 331 | |
| 332 | - `@config translation model <name>` |
| 333 | - `@config explainer model <name>` |
| 334 | |
| 335 | They are also offered in CLI's --model option for the follow commands: |
| 336 | |
| 337 | - `agent-cli interactive` |
| 338 | - `agent-cli run request` |
| 339 | - `agent-cli run translate` |
| 340 | - `agent-cli prompt` |
| 341 | - `agent-cli data add` |
| 342 | - `agent-cli data regen` |
| 343 | |
| 344 | The model name to specify are prefix with `ollama:` for example `ollama:phi3` or `ollama:llama3.2` or specifically tagged ones: e.g. `ollama:llama3.2:3b-instruct-q5_0` |
| 345 | |
| 346 | If ollama is running on a different URL, `OLLAMA_ENDPOINT` environment variable can be use to specify the URL and port for the Ollama endpoints. |
| 347 | |
| 348 | ### User data location |
| 349 | |
| 350 | To share user data with other developers for debugging, please look for the folder `.typeagent` under `%USERPROFILE%`on Windows and the home directory `~/` on WSL/Linux/MacOS. |
| 351 | |
| 352 | ## Trademarks |
| 353 | |
| 354 | This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft |
| 355 | trademarks or logos is subject to and must follow |
| 356 | [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). |
| 357 | Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. |
| 358 | Any use of third-party trademarks or logos are subject to those third-party's policies. |
| 359 | |