# TypeAgent Typescript Repo
## Overview
**TypeAgent** is **sample code** that explores architectures for building _interactive agents_ with _natural language interfaces_ using [TypeChat](https://github.com/microsoft/typechat).
This directory contains Typescript implemented packages and main entry point for **TypeAgent**. For more details about the project, please review the TypeAgent [ReadMe](./../README.md).
## Build
### Setup
To build:
- Install [Node 18+](https://nodejs.org/en/download)
- NOTE: HPC Tools conflict with node so be sure that the node.exe you are running is the correct one!
- Install pnpm (`npm i -g pnpm && pnpm setup`)
- **(Linux/WSL Only)** Read TypeAgent Shell's [README.md](./packages/shell/README.md) for additional requirements
### Steps
In this directory:
- Run `pnpm i`
- Run `pnpm run build`
### Agent Specific Steps (Optional)
#### VSCode Agent
If you want to `deploy` the **VS Code** extension `CODA` locally please run:
- From the root, `cd ./ts/packages/coda`
- `pnpm run deploy:local`
You should now be able to access the extension from VS Code.
#### Desktop Agent (Windows only)
To use the [Desktop Agent](./packages/agents/desktop/] for windows, follow the instruction in [README.md](./packages/agents/desktop/README.md) to build the [AutoShell](../dotnet/autoShell/) C# code necessary to interact with the OS.
### Local Whisper Service (Optional)
If you want to use a local whisper service for voice input in the [TypeAgent Shell](./packages/shell), please follow instruction in the [README.md](../python/whisperService/README.md) in the python's [whisperService](../python/whisperService/) directory.
## Running Prerequisites
### Service Keys
Multiple services are required to run the scenarios. Put the necessary keys in the `.env` file at this directory (TypeAgent repo's `./ts` directory).
The follow set of functionality will need the services keys. Please read the links for details about the variables needed.
**Minimum requirements** to try out the experience with the [List](./packages/agents/list/README.md) TypeAgent:
| Requirements | Functionality | Variables | Instructions |
| ------------------------- | ------------------- | ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| LLM (GPT-4 or equivalent) | Request translation | AZURE_OPENAI_API_KEY<br>AZURE_OPENAI_ENDPOINT<br>AZURE_OPENAI_RESPONSE_FORMAT | [TypeChat instruction](https://github.com/microsoft/TypeChat/tree/main/typescript/examples#step-3-configure-environment-variables). |
**Optional requirements**
| Requirements | Functionality | Variables | Instructions |
| ----------------------------------------------------------------------------------------------------------------- | ------------------------ | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| [Speech to Text service](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/index-speech-to-text) | Voice input (shell only) | SPEECH_SDK_ENDPOINT<br>SPEECH_SDK_KEY<br>SPEECH_SDK_REGION | [Shell setup instruction](./packages/shell/README.md#azure-speech-to-text-service-optional) |
**Additional keys required per TypeAgent** (Optional if not using these TypeAgent)
| Requirements | Functionality | Variables | Instructions |
| ---------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------- |
| Bing Search API | Chat Lookup | BING_API_KEY | |
| GPT-3.5 Turbo | Fast Chat Response<br>[Email](./packages/agents/email) content generation | AZURE_OPENAI_API_KEY_GPT_35_TURBO<br>AZURE_OPENAI_ENDPOINT_GPT_35_TURBO |
| [Spotify Web API](https://developer.spotify.com/documentation/web-api) | [Music player](./packages/agents/player/) | SPOTIFY_APP_CLI<br>SPOTIFY_APP_CLISEC<br>SPOTIFY_APP_PORT | [Music player setup](./packages/agents/player/README.md#application-keys) |
| [Graph Application](https://developer.microsoft.com/en-us/graph) | [Calendar](./packages/agents/calendar/)/[Email](./packages/agents/email) | MSGRAPH_APP_CLIENTID<br>MSGRAPH_APP_CLIENTSECRET<br>MSGRAPH_APP_TENANTID | |
| Embeddings | [Desktop](./packages/agents/desktop/) App name Fuzzy match | AZURE_OPENAI_API_KEY_EMBEDDING<br>AZURE_OPENAI_ENDPOINT_EMBEDDING | |
| GPT-4o | [Browser](./packages/agents/browser/) - Crossword Page | AZURE_OPENAI_API_KEY_GPT_4_O<br>AZURE_OPENAI_ENDPOINT_GPT_4_O | |
| [Bing Maps Location Rest API](https://learn.microsoft.com/en-us/bingmaps/rest-services/locations/find-a-location-by-address) | [Browser](./packages/agents/browser/) - PaleoBioDB set Lat/Longitude action | BING_MAPS_API_KEY | |
Other examples in the [example directory](./examples/) may have additional service keys requirements. See the README in those examples for more detail.
Read the [Debugging](#debugging) section for additional service keys that can be used for debugging.
**For AISystems Team Devs**:
AISystems Team Devs has shared instances of these services. To get the required config and key saved to the `.env` file:
- Install [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli)
- Run `az login` to login using the CLI. Run `az account set --subscription <Subscription Id>` to set the subscription.
- Run `npm run getKeys` at the root to pull secret from our team key vault.
Note: Shared keys doesn't include Spotify integration, which can be created using the the [Spotify API keys instructions](./packages/agents/player/README.md)
To update keys on the key vault
- Add or change the values in the `.env` file
- Add new keys name in `tools/scripts/getKeys.config.json`
- Run `npm run getKeys -- push`
- Check in the changes to `tools/scripts/getKeys.config.json`
### WSL
For TypeAgents that operates on the Microsoft Graph (e.g. [Calendar](./packages/agents/calendar/) and [Email](./packages/agents/email/)), they leverage [@azure/identity](https://www.npmjs.com/package/@azure/identity) for authentication and use [@azure/identity-cache-persistence](https://www.npmjs.com/package/@azure/identity-cache-persistence) to cache on the local machine.
Install the following packages if you are on `WSL2` environment (please `restart` the shell after running the commands below):
```shell
sudo apt-get update
sudo apt install -y gnome-keyring
```
After the step above, you will need to enter a password to protect the secrets in the keyring. The popup normally appears when you restart the shell and run the code that needs to persists secrets in the keyring.
## Running
There are two main apps to start using the system: [TypeAgent Shell](#shell) or [TypeAgent CLI](#cli). Currently, we only support running builds from the repo (i.e. no published/installable builds). Both shared a common core package [dispatcher](./packages/dispatcher/)
### Shell
- Run `pnpm run shell`.
Also, you can go to the shell directory `./packages/shell` and start from there. Please see instruction in TypeAgent Shell's [README.md](./packages/shell/README.md).
### CLI
- Run `pnpm run cli` to get the available command
- Run `pnpm run cli -- interactive` will start the interactive prompt
Also, you can go to the CLI directory `./packages/cli` and start from there. Please see instruction in TypeAgent CLI's [README.md](./packages/cli/README.md) for more options and detail.
## Development
### Main packages and directory structure
Apps:
- [agent-cli](./packages/cli): TypeAgent dispatcher CLI and interactive prompt
- [agent-shell](./packages/shell): TypeAgent shell UI
Libraries:
- [agent-dispatcher](./packages/dispatcher): TypeAgent dispatcher used by both the CLI and shell
- [agent-cache](./packages/cache): Construction explanation and cache
[Agents with natural language interfaces](./packages/agents):
- [Music Player](./packages/agents/player/): Spotify music player TypeAgent plugin
- [Chat](./packages/agents/chat/)
- [Browser](./packages/agents/browser/)
- [VS Code](./packages/agents/code/)
- [List Management](./packages/agents/list/)
- [Calendar](./packages/agents/calendar/)
- [Email](./packages/agents/email/)
- [Desktop](./packages/agents/desktop/)
Other directories:
- [examples](./examples/): various additional standalone explorations.
- [tools](./tools/): tools for CI/CD and internal development environments.
### Testing
Run `npm run test` at the root.
#### Schema Changes
If new translator or explainer, or any of the translator schema or explanation schema changes, the built-in construction cache and the test data needs to be regenerated and be evaluated for correctness.
Test data are located in the [dispatcher](./packages/dispatcher)'s [test/data](./packages/dispatcher/test/data) directory. Each test data files are for specify translator and explainer.
Use the `agent-cli data add` command to add new test cases.
To regenerated you can run the following at the root or in the [cli](./packages/cli) directory:
- `npm run regen:builtin` - Regenerate builtin construction store.
- `npm run regen` - Regenerate test data
To evaluate correctness for the test data:
- `agent-cli data diff <file>` can be used to open test data file diff in the vscode.
- Look at the translation to check if its correct. (can be skipped if translator schema didn't change).
- Run `npm run test` to make sure the generated test data can be round tripped (Run in the CI as well).
- Check the stats in the regen before and after:
- `npm run regen -- -- --none` at the root will print out per file stats and total stats.
- Make sure that the number explanation failure per file and total stay roughly same (or improved).
- Make sure that the attempts (corrections) ratios stay roughly the same (or improve).
- Examine if the failures are because of LLM instability:
- Borderline failure: was there a lot of correction before and failed now.
- Run the explanation before and after `agent-cli explain --repeat 5 <RequestAction>` to repeat it 5 times and compare the stats.
### Linting
The repo is set up with prettier to help with consistent code style. Run `npm run lint` to check and `npm run lint:fix` to fix any issues.
### Debugging
#### Starting Development version of TypeAgent CLI
Go to `./packages/cli`, you don't have to build and just run `./bin/dev.js`. It will use `ts-node` and build the typescript as it goes.
#### Launching from VSCode
If you open this directory as a workspace in VSCode, multiple launch task is defined to quickly start debug.
Common Debug Launch Task:
- CLI interactive - `./package/cli/bin/run.js interactive`
- CLI (dev) interactive - `./package/cli/bin/dev.js interactive` with a new command prompt
- CLI (dev) interactive [Integrated Terminal] - `./bin/dev.js interactive` using VSCode terminal (needed for WSL)
#### Attaching to running sessions
To attaching to an existing session with TypeAgent CLI's interactive mode or TypeAgent Shell, you can start inspector by issuing the command `@debug` and use the VSCode `Attach` debugger launch task to attach.
#### TypeAgent Shell Browser Process
With the TypeAgent Shell, press F12 will bring up the devtool to debug the browser process.
### Tracing
The project uses [debug](https://www.npmjs.com/package/debug) package to enable tracing. There are two options to enable these traces:
**Option 1**: Set the namespace pattern in the environment variable `DEBUG=typeagent:prompt` before starting the program.
For example (in Linux), to trace the GPT prompt that get sent when running the interactive CLI.
```bash
DEBUG=typeagent:prompt packages/cli/bin/run.js interactive
```
**Option 2**: In the shell or CLI's interactive mode, you can issue the command `@trace <pattern>` to add to the list of namespace. Use "-" or "-\*" to disable all the trace.
For example inside the CLI's interactive mode, enter
```txt
@trace *:prompt:*
```
Search the code base with '"typeagent:' will give all the traces available.
### Logging
TypeAgent does not collect telemetry by default. Developer can enable logging to a mongodb for internal debugging purpose by providing a mongodb connection string with the `MONGODB_CONNECTION_STRING` variable in the `.env` file.
### Alternate LLM
Other LLM can be substituted for GPT-4 as long as they are REST API compatible.
To use a local model the follow environment variable can be used:
```
OPENAI_API_KEY_LOCAL=None
OPENAI_ENDPOINT_LOCAL=
OPENAI_MODEL_LOCAL=
OPENAI_ORGANIZATION_LOCAL=
OPENAI_RESPONSE_FORMAT_LOCAL=
```
### User data location
To share user data with other developers for debugging, please look for the folder `.typeagent` under `%USERPROFILE%`on Windows and the home directory `~/` on WSL/Linux/MacOS.
## Trademarks
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.microsoft/TypeAgent
Publicmirrored from https://github.com/microsoft/TypeAgentAvailable
ts/README.md
265lines · modepreview