microsoft/TypeAgent

Commit

Elastic Memory Provider & Podcast Commands (#516)

## Elastic Memory Provider

This PR adds an implementation of an
[Elastisearch](https://www.elastic.co/) memory provider to the
`/examples/memoryProvider` directory. This memory provider uses the
existing implementations based on the filesystem for the object table
and the temporal table. In the future we hope to add support for these
to the elastic instance as well.

This implementation is built off of the `StorageProvider` interface
defined in the Knowledge Processor package.

In addition to the memory provider, two commands were added for
interacting with podcasts that have been indexed.
This pull request introduces significant updates to the memory
management system in the chat and podcast examples, primarily by
integrating Elasticsearch for enhanced storage capabilities.
Additionally, it includes various dependency updates and minor code
improvements.

Automatically Generated by GitHub:

### Integration of Elasticsearch:

*
[`ts/examples/chat/src/memory/podcastMemory.ts`](diffhunk://#diff-f4f5a547c138d8545a6fb589ec58c79caff2d4644af63a8fd30558e3421d0122R11):
Added support for Elasticsearch as a storage option in the
`createPodcastMemory` function and introduced a new command
`podcastEntities` to list podcast entities.
[[1]](diffhunk://#diff-f4f5a547c138d8545a6fb589ec58c79caff2d4644af63a8fd30558e3421d0122R11)
[[2]](diffhunk://#diff-f4f5a547c138d8545a6fb589ec58c79caff2d4644af63a8fd30558e3421d0122R45-R66)
[[3]](diffhunk://#diff-f4f5a547c138d8545a6fb589ec58c79caff2d4644af63a8fd30558e3421d0122R90)
[[4]](diffhunk://#diff-f4f5a547c138d8545a6fb589ec58c79caff2d4644af63a8fd30558e3421d0122R128-R144)
*
[`ts/examples/memoryProviders/src/elastic/common.ts`](diffhunk://#diff-8dd2f1760821030134e3cd9f70862761097a81470bd3c3f67dcaeb5f3c26c49cR1-R64):
Introduced utility functions to create an Elasticsearch client, delete
indices, generate text IDs, and convert strings to valid index names.
*
[`ts/examples/memoryProviders/src/elastic/storageClient.ts`](diffhunk://#diff-d7262c48f087da6dc961ba32161545f0eacac034c344bb3960a8763fae9c039fR1-R93):
Implemented a storage client for Elasticsearch, providing methods to
create object folders, temporal logs, text indices, and key-value
indices.
*
[`ts/examples/memoryProviders/src/elastic/keyValueIndex.ts`](diffhunk://#diff-51058c687cf161548bad8b4a3b135c37cc8af25cbe54dec3d136b0772eb8ba62R1-R97):
Added a key-value index implementation for Elasticsearch.
*
[`ts/examples/memoryProviders/src/elastic/index.ts`](diffhunk://#diff-cdf45b56ed4946421f57d2bc9354cbfa2788234f0643301c6f130cdc73b22a15R1-R7):
Exported Elasticsearch-related modules.

### Dependency Updates:

*
[`ts/examples/memoryProviders/package.json`](diffhunk://#diff-b9f3546e0454719e0f6a2207a0163e121d4b35b8638c004b4fd068537a36f659R30):
Added `@elastic/elasticsearch` as a dependency.
*
[`ts/pnpm-lock.yaml`](diffhunk://#diff-01ef0de5ac168b38408809193ae625671caf7201be45a505927b28c7123c33e8R100-R102):
Updated to include new dependencies and versions, such as
`@elastic/elasticsearch`, `@types/command-line-args`, and
`apache-arrow`.
[[1]](diffhunk://#diff-01ef0de5ac168b38408809193ae625671caf7201be45a505927b28c7123c33e8R100-R102)
[[2]](diffhunk://#diff-01ef0de5ac168b38408809193ae625671caf7201be45a505927b28c7123c33e8L435-R438)
[[3]](diffhunk://#diff-01ef0de5ac168b38408809193ae625671caf7201be45a505927b28c7123c33e8R2368-R2375)
[[4]](diffhunk://#diff-01ef0de5ac168b38408809193ae625671caf7201be45a505927b28c7123c33e8R3148-R3151)
[[5]](diffhunk://#diff-01ef0de5ac168b38408809193ae625671caf7201be45a505927b28c7123c33e8R3287-R3289)
[[6]](diffhunk://#diff-01ef0de5ac168b38408809193ae625671caf7201be45a505927b28c7123c33e8R3356-R3361)
[[7]](diffhunk://#diff-01ef0de5ac168b38408809193ae625671caf7201be45a505927b28c7123c33e8R3488-R3490)
[[8]](diffhunk://#diff-01ef0de5ac168b38408809193ae625671caf7201be45a505927b28c7123c33e8R3768-R3771)
[[9]](diffhunk://#diff-01ef0de5ac168b38408809193ae625671caf7201be45a505927b28c7123c33e8R3807-R3814)
[[10]](diffhunk://#diff-01ef0de5ac168b38408809193ae625671caf7201be45a505927b28c7123c33e8R4121-R4124)

### Code Improvements:

* `ts/examples/chat/src/memory/chatMemory.ts` and
`ts/examples/chat/src/memory/podcastMemory.ts`: Updated copyright
comments.
[[1]](diffhunk://#diff-69239d9a93997212cc8c49605d935548638216d2d2e20b660c76d4846f11f699L1-R1)
[[2]](diffhunk://#diff-f4f5a547c138d8545a6fb589ec58c79caff2d4644af63a8fd30558e3421d0122L1-R1)
*
[`ts/packages/knowledgeProcessor/src/conversation/entities.ts`](diffhunk://#diff-122cb1a8cdecc016bb32d5a672fe1df01a71d69acc61014e57d7f7ce19bb8be5R573-R579):
Added a safeguard in the `toCompositeEntity` function to handle
undefined entities.

These changes collectively enhance the memory management capabilities by
leveraging Elasticsearch, improve code robustness, and ensure
dependencies are up to date.

---------

Co-authored-by: Steve Lucco <steveluc@users.noreply.github.com>
Co-authored-by: robgruen <robgruen@microsoft.com>