microsoft/onnxruntime-extensions
Publicmirrored from https://github.com/microsoft/onnxruntime-extensionsAvailable
docs/c_api.md
30lines · modeblame
8153bc1aWenbing Li2 years ago | 1 | # ONNXRuntime Extensions C ABI |
| 2 | | |
| 3 | ONNXRuntime Extensions provides a C-style ABI for pre-processing. It offers support for tokenization, image processing, speech feature extraction, and more. You can compile the ONNXRuntime Extensions as either a static library or a dynamic library to access these APIs. | |
| 4 | | |
| 5 | The C ABI header files are named `ortx_*.h` and can be found in the include folder. There are three types of data processing APIs available: | |
| 6 | | |
| 7 | - [`ortx_tokenizer.h`](../include/ortx_tokenizer.h): Provides tokenization for LLM models. | |
| 8 | - [`ortx_processor.h`](../include/ortx_processor.h): Offers image processing APIs for multimodels. | |
| 9 | - [`ortx_extraction.h`](../include/ortx_extractor.h): Provides speech feature extraction for audio data processing to assist the Whisper model. | |
| 10 | | |
| 11 | ## ABI QuickStart | |
| 12 | | |
| 13 | Most APIs accept raw data inputs such as audio, image compressed binary formats, or UTF-8 encoded text for tokenization. | |
| 14 | | |
| 15 | **Tokenization:** You can create a tokenizer object using `OrtxCreateTokenizer` and then use the object to tokenize a text or decode the token ID into the text. A C-style code snippet is available [here](../test/pp_api_test/c_only_test.c). | |
| 16 | | |
| 17 | **Image processing:** `OrtxCreateProcessor` can create an image processor object from a pre-defined workflow in JSON format to process image files into a tensor-like data type. An example code snippet can be found [here](../test/pp_api_test/test_processor.cc#L75). | |
| 18 | | |
620050fbWenbing Li2 years ago | 19 | **Audio feature extraction:** `OrtxCreateSpeechFeatureExtractor` creates a speech feature extractor to obtain log mel spectrum data as input for the Whisper model. An example code snippet can be found [here](../test/pp_api_test/test_feature_extraction.cc#L16). |
be29e28dWenbing Li2 years ago | 20 | |
8f2c35faWenbing Li1 years ago | 21 | **NB:** If onnxruntime-extensions is to build as a shared library, which requires the OCOS_ENABLE_AUDIO OCOS_ENABLE_CV2 OCOS_ENABLE_OPENCV_CODECS OCOS_ENABLE_GPT2_TOKENIZER build flags are ON to have a full function of binary. Only onnxruntime-extensions static library can be used for a minimal build with the selected operators, so in that case, the shared library build can be switched off by `-DOCOS_BUILD_SHARED_LIB=OFF`. |
| 22 | | |
| 23 | There is a simple Python wrapper on these C API in [pp_api](../onnxruntime_extensions/pp_api.py), which can have a easy access these APIs in Python code like | |
| 24 | | |
| 25 | ```Python | |
| 26 | from onnxruntime_extensions.pp_api import Tokenizer | |
| 27 | # the name can be the same one used by Huggingface transformers.AutoTokenizer | |
| 28 | pp_tok = Tokenizer('google/gemma-2-2b') | |
| 29 | print(pp_tok.tokenize("what are you? \n 给 weiss ich, über was los ist \n")) | |
| 30 | ``` |