openai/tiktoken
Publicmirrored fromhttps://github.com/openai/tiktokenAvailable
CHANGELOG.md
59lines · modecode
| 1 | # Changelog |
| 2 | |
| 3 | This is the changelog for the open source version of tiktoken. |
| 4 | |
| 5 | ## [v0.5.2] |
| 6 | - Build wheels for Python 3.12 |
| 7 | - Update version of PyO3 to allow multiple imports |
| 8 | - Avoid permission errors when using default cache logic |
| 9 | |
| 10 | ## [v0.5.1] |
| 11 | - Add `encoding_name_for_model`, undo some renames to variables that are implementation details |
| 12 | |
| 13 | ## [v0.5.0] |
| 14 | - Add `tiktoken._educational` submodule to better document how byte pair encoding works |
| 15 | - Ensure `encoding_for_model` knows about several new models |
| 16 | - Add `decode_with_offets` |
| 17 | - Better error for failures with the plugin mechanism |
| 18 | - Make more tests public |
| 19 | - Update versions of dependencies |
| 20 | |
| 21 | ## [v0.4.0] |
| 22 | - Add `decode_batch` and `decode_bytes_batch` |
| 23 | - Improve error messages and handling |
| 24 | |
| 25 | ## [v0.3.3] |
| 26 | - `tiktoken` will now make a best effort attempt to replace surrogate pairs with the corresponding |
| 27 | Unicode character and will replace lone surrogates with the Unicode replacement character. |
| 28 | |
| 29 | ## [v0.3.2] |
| 30 | - Add encoding for GPT-4 |
| 31 | |
| 32 | ## [v0.3.1] |
| 33 | - Build aarch64 wheels |
| 34 | - Make `blobfile` an optional dependency |
| 35 | |
| 36 | Thank you to @messense for the environment variable that makes cargo not OOM under emulation! |
| 37 | |
| 38 | ## [v0.3.0] |
| 39 | - Improve performance by 5-20%; thank you to @nistath! |
| 40 | - Add `gpt-3.5-turbo` models to `encoding_for_model` |
| 41 | - Add prefix matching to `encoding_for_model` to better support future model versions |
| 42 | - Fix a bug in the README instructions on extending tiktoken |
| 43 | - Update the set of available encodings |
| 44 | - Add packaging metadata |
| 45 | |
| 46 | ## [v0.2.0] |
| 47 | - Add ``tiktoken.encoding_for_model`` to get the encoding for a specific model |
| 48 | - Improve portability of caching logic |
| 49 | |
| 50 | Thank you to @fritzo, @arvid220u, @khanhvu207, @henriktorget for various small corrections |
| 51 | |
| 52 | ## [v0.1.2] |
| 53 | - Avoid use of `blobfile` for public files |
| 54 | - Add support for Python 3.8 |
| 55 | - Add py.typed |
| 56 | - Improve the public tests |
| 57 | |
| 58 | ## [v0.1.1] |
| 59 | - Initial release |
| 60 | |