GitVita
New repositoryImport repositoryNew issue
Your repositories Your dashboard Sign in

microsoft/onnxruntime-extensions

Public

mirrored from https://github.com/microsoft/onnxruntime-extensionsAvailable

Watch0Fork0Star0
CodeCommitsIssuesPull requestsActionsInsightsSecurity
print_logmel

Branches

Tags

  • No tags available.
0Branches0Tags
Go to file
Add file
Code

Clone

HTTPS

Download ZIP

onnxruntime-extensions/docs

docs/custom_ops.md

1592lines · modeblame

RawDownload
Latest commit unavailable.
f12f42a0Nat Kershaw (MSFT)4 years ago1# Operators
b3a300d7Xavier Dupré5 years ago2
3
f12f42a0Nat Kershaw (MSFT)4 years ago4## Natural language operators
b3a300d7Xavier Dupré5 years ago5
f12f42a0Nat Kershaw (MSFT)4 years ago6### BertTokenizer
b3a300d7Xavier Dupré5 years ago7
f12f42a0Nat Kershaw (MSFT)4 years ago8<details>
9<summary>BertTokenizer details</summary>
b3a300d7Xavier Dupré5 years ago10
f12f42a0Nat Kershaw (MSFT)4 years ago11BertTokenizer replicates `encode_plus` function of [BertTokenizer (huggingface version )](https://huggingface.co/transformers/_modules/transformers/models/bert/tokenization_bert.html#BertTokenizer).
b3a300d7Xavier Dupré5 years ago12
f12f42a0Nat Kershaw (MSFT)4 years ago13#### Inputs
b3a300d7Xavier Dupré5 years ago14
f12f42a0Nat Kershaw (MSFT)4 years ago15***text: tensor(string)*** The string tensor for tokenization
b3a300d7Xavier Dupré5 years ago16
17#### Attributes
18
f12f42a0Nat Kershaw (MSFT)4 years ago19***vocab_file: string***
b3a300d7Xavier Dupré5 years ago20
f12f42a0Nat Kershaw (MSFT)4 years ago21The content of vocab which has same with huggingface.
b3a300d7Xavier Dupré5 years ago22
f12f42a0Nat Kershaw (MSFT)4 years ago23***do_lower_case: int64_t*** (default is 1, 1 represents True, 0 represents False)
b3a300d7Xavier Dupré5 years ago24
f12f42a0Nat Kershaw (MSFT)4 years ago25Whether or not to lowercase the input when tokenizing.
b3a300d7Xavier Dupré5 years ago26
f12f42a0Nat Kershaw (MSFT)4 years ago27***do_basic_tokenize: int64_t*** (default is 1, 1 represents True, 0 represents False)
b3a300d7Xavier Dupré5 years ago28
f12f42a0Nat Kershaw (MSFT)4 years ago29Whether or not to do basic tokenization before WordPiece.
b3a300d7Xavier Dupré5 years ago30
f12f42a0Nat Kershaw (MSFT)4 years ago31***unk_token: string***
b3a300d7Xavier Dupré5 years ago32
f12f42a0Nat Kershaw (MSFT)4 years ago33The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
34token instead.
b3a300d7Xavier Dupré5 years ago35
f12f42a0Nat Kershaw (MSFT)4 years ago36***sep_token: string***
b3a300d7Xavier Dupré5 years ago37
f12f42a0Nat Kershaw (MSFT)4 years ago38The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
39sequence classification or for a text and a question for question answering. It is also used as the last
40token of a sequence built with special tokens.
b3a300d7Xavier Dupré5 years ago41
f12f42a0Nat Kershaw (MSFT)4 years ago42***pad_token: string***
b3a300d7Xavier Dupré5 years ago43
f12f42a0Nat Kershaw (MSFT)4 years ago44The token used for padding, for example when batching sequences of different lengths.
4290400eMojimi4 years ago45
f12f42a0Nat Kershaw (MSFT)4 years ago46***cls_token: string***
4290400eMojimi4 years ago47
f12f42a0Nat Kershaw (MSFT)4 years ago48The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens.
4290400eMojimi4 years ago49
f12f42a0Nat Kershaw (MSFT)4 years ago50***mask_token: string***
4290400eMojimi4 years ago51
f12f42a0Nat Kershaw (MSFT)4 years ago52The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict.
4290400eMojimi4 years ago53
f12f42a0Nat Kershaw (MSFT)4 years ago54***tokenize_chinese_chars: int64_t*** (default is 1, 1 represents True, 0 represents False)
4290400eMojimi4 years ago55
f12f42a0Nat Kershaw (MSFT)4 years ago56Whether or not to tokenize Chinese characters.
4290400eMojimi4 years ago57
f12f42a0Nat Kershaw (MSFT)4 years ago58***strip_accents: int64_t*** (default is 1, 1 represents True, 0 represents False)
4290400eMojimi4 years ago59
f12f42a0Nat Kershaw (MSFT)4 years ago60Whether or not to strip all accents. If this option is not specified, then it will be determined by the
61value for :obj:`lowercase` (as in the original BERT).
4290400eMojimi4 years ago62
f12f42a0Nat Kershaw (MSFT)4 years ago63***tokenize_punctuation: int64_t*** (default is 0, 1 represents True, 0 represents False)
4290400eMojimi4 years ago64
f12f42a0Nat Kershaw (MSFT)4 years ago65Splits punctuation on a piece of text.
4290400eMojimi4 years ago66
f12f42a0Nat Kershaw (MSFT)4 years ago67***remove_control_chars: int64_t*** (default is 0, 1 represents True, 0 represents False)
4290400eMojimi4 years ago68
f12f42a0Nat Kershaw (MSFT)4 years ago69Remove control chars(such as NUL, BEL) in the text.
4290400eMojimi4 years ago70
f12f42a0Nat Kershaw (MSFT)4 years ago71***truncation_strategy_name: string***
4290400eMojimi4 years ago72
f12f42a0Nat Kershaw (MSFT)4 years ago73The name of truncation strategy, it could be `longest_first`, `only_first`, `only_second`, `longest_from_back`.
4290400eMojimi4 years ago74
75#### Outputs
76
f12f42a0Nat Kershaw (MSFT)4 years ago77***input_ids: tensor(int64_t)***
4290400eMojimi4 years ago78
f12f42a0Nat Kershaw (MSFT)4 years ago79List of token ids.
4290400eMojimi4 years ago80
f12f42a0Nat Kershaw (MSFT)4 years ago81***token_type_ids: tensor(64_t)***
4290400eMojimi4 years ago82
f12f42a0Nat Kershaw (MSFT)4 years ago83List of token type ids
84
85***attention_mask: tensor(64_t)***
86
87List of indices specifying which tokens should b
88e attended to by the model
89
90
91#### Examples
4290400eMojimi4 years ago92
93```python
f12f42a0Nat Kershaw (MSFT)4 years ago94import transformers
95
2c3d6f79Wenbing Li1 years ago96bert_cased_tokenizer = transformers.BertTokenizer.from_pretrained('google-bert/bert-base-cased')
4290400eMojimi4 years ago97
98node = onnx.helper.make_node(
f12f42a0Nat Kershaw (MSFT)4 years ago99'BertTokenizer',
100inputs=['text'],
101outputs=['tokens'],
4290400eMojimi4 years ago102)
103
f12f42a0Nat Kershaw (MSFT)4 years ago104text = "Hello world louder"
280ec289Edward Chen3 years ago105inputs = np.array([text], dtype=object),
4290400eMojimi4 years ago106
f12f42a0Nat Kershaw (MSFT)4 years ago107bert_tokenize_result = bert_cased_tokenizer.tokenize(text)
108
109input_ids = np.array(bert_tokenize_result[0])
110token_type_ids = np.array(bert_tokenize_result[1])
111attention_mask = np.array(bert_tokenize_result[2])
4290400eMojimi4 years ago112
f12f42a0Nat Kershaw (MSFT)4 years ago113expect(node, inputs=[inputs],
114outputs=[input_ids, token_type_ids, attention_mask], name='test_bert_tokenizer')
115```
4290400eMojimi4 years ago116</details>
117
f12f42a0Nat Kershaw (MSFT)4 years ago118### BertTokenizerDecoder
4290400eMojimi4 years ago119
f12f42a0Nat Kershaw (MSFT)4 years ago120<details>
121<summary>BertTokenizerDecoder details</summary>
4bc5c962Xavier Dupré5 years ago122
f12f42a0Nat Kershaw (MSFT)4 years ago123BertTokenizerDecoder replicates `decode` function of [BertTokenizer (huggingface version )](https://huggingface.co/transformers/_modules/transformers/models/bert/tokenization_bert.html#BertTokenizer).
4bc5c962Xavier Dupré5 years ago124
125#### Inputs
126
f12f42a0Nat Kershaw (MSFT)4 years ago127***token_ids: tensor(int64)***
4bc5c962Xavier Dupré5 years ago128
f12f42a0Nat Kershaw (MSFT)4 years ago129List of tokenized input ids.
4bc5c962Xavier Dupré5 years ago130
f12f42a0Nat Kershaw (MSFT)4 years ago131***indices: tensor(int64)***
4bc5c962Xavier Dupré5 years ago132
f12f42a0Nat Kershaw (MSFT)4 years ago133List of `[start_position, end_position]` to indicate what segments of input ids should be decoded. This input only enabled when attribute `use_indices`=1.
4bc5c962Xavier Dupré5 years ago134
f12f42a0Nat Kershaw (MSFT)4 years ago135Usually, it is used to decode the slot in the text.
4bc5c962Xavier Dupré5 years ago136
f12f42a0Nat Kershaw (MSFT)4 years ago137#### Attributes
4bc5c962Xavier Dupré5 years ago138
f12f42a0Nat Kershaw (MSFT)4 years ago139***vocab_file: string***
4bc5c962Xavier Dupré5 years ago140
f12f42a0Nat Kershaw (MSFT)4 years ago141The content of vocab which has same with huggingface.
4bc5c962Xavier Dupré5 years ago142
f12f42a0Nat Kershaw (MSFT)4 years ago143***unk_token: string***
4bc5c962Xavier Dupré5 years ago144
f12f42a0Nat Kershaw (MSFT)4 years ago145The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
146token instead.
4bc5c962Xavier Dupré5 years ago147
f12f42a0Nat Kershaw (MSFT)4 years ago148***sep_token: string***
4bc5c962Xavier Dupré5 years ago149
f12f42a0Nat Kershaw (MSFT)4 years ago150The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
151sequence classification or for a text and a question for question answering. It is also used as the last
152token of a sequence built with special tokens.
4bc5c962Xavier Dupré5 years ago153
f12f42a0Nat Kershaw (MSFT)4 years ago154***pad_token: string***
4bc5c962Xavier Dupré5 years ago155
f12f42a0Nat Kershaw (MSFT)4 years ago156The token used for padding, for example when batching sequences of different lengths.
4bc5c962Xavier Dupré5 years ago157
f12f42a0Nat Kershaw (MSFT)4 years ago158***cls_token: string***
4bc5c962Xavier Dupré5 years ago159
f12f42a0Nat Kershaw (MSFT)4 years ago160The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens.
4bc5c962Xavier Dupré5 years ago161
f12f42a0Nat Kershaw (MSFT)4 years ago162***mask_token: string***
4bc5c962Xavier Dupré5 years ago163
f12f42a0Nat Kershaw (MSFT)4 years ago164The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict.
2378ca11Mojimi5 years ago165
f12f42a0Nat Kershaw (MSFT)4 years ago166***suffix_indicator: string***
2378ca11Mojimi5 years ago167
f12f42a0Nat Kershaw (MSFT)4 years ago168The suffix indicator.
2378ca11Mojimi5 years ago169
f12f42a0Nat Kershaw (MSFT)4 years ago170***use_indices: int64_t***
2378ca11Mojimi5 years ago171
f12f42a0Nat Kershaw (MSFT)4 years ago172Whether use second input.
2378ca11Mojimi5 years ago173
f12f42a0Nat Kershaw (MSFT)4 years ago174***skip_special_tokens: int64_t***
2378ca11Mojimi5 years ago175
f12f42a0Nat Kershaw (MSFT)4 years ago176Whether or not to remove special tokens in the decoding.
2378ca11Mojimi5 years ago177
f12f42a0Nat Kershaw (MSFT)4 years ago178***clean_up_tokenization_spaces: int64_t***
2378ca11Mojimi5 years ago179
f12f42a0Nat Kershaw (MSFT)4 years ago180Whether or not to clean up the tokenization spaces.
2378ca11Mojimi5 years ago181
182#### Outputs
183
f12f42a0Nat Kershaw (MSFT)4 years ago184***sentences: tensor(int64_t)***
2378ca11Mojimi5 years ago185
f12f42a0Nat Kershaw (MSFT)4 years ago186The decoded sentences.
2378ca11Mojimi5 years ago187
188#### Examples
189
190
191```python
f12f42a0Nat Kershaw (MSFT)4 years ago192import transformers
193
194def get_file_content(path):
195with open(path, "rb") as file:
196return file.read()
197
2c3d6f79Wenbing Li1 years ago198bert_cased_tokenizer = transformers.BertTokenizer.from_pretrained('google-bert/bert-base-cased')
f12f42a0Nat Kershaw (MSFT)4 years ago199bert_cased_tokenizer.save('.', 'bert')
200
2378ca11Mojimi5 years ago201
202node = onnx.helper.make_node(
f12f42a0Nat Kershaw (MSFT)4 years ago203'BertTokenizerDecoder',
204inputs=['token_ids'],
205outputs=['sentences'],
206vocab_file=get_file_content("bert-vocab.txt")
2378ca11Mojimi5 years ago207)
208
f12f42a0Nat Kershaw (MSFT)4 years ago209text = "Hello world louder"
280ec289Edward Chen3 years ago210token_ids = np.array([bert_cased_tokenizer.tokenize(text)], dtype=object),
f12f42a0Nat Kershaw (MSFT)4 years ago211sentences = np.array(text)
2378ca11Mojimi5 years ago212
213
f12f42a0Nat Kershaw (MSFT)4 years ago214expect(node, inputs=[token_ids],
215outputs=[sentences], name='test_bert_tokenizer')
216```
2378ca11Mojimi5 years ago217</details>
218
a9a49850Mojimi5 years ago219
220
f12f42a0Nat Kershaw (MSFT)4 years ago221### GPT2Tokenizer
a9a49850Mojimi5 years ago222
f12f42a0Nat Kershaw (MSFT)4 years ago223<details>
224<summary>GPT2Tokenizer details</summary>
4a0f8929Mojimi5 years ago225
f12f42a0Nat Kershaw (MSFT)4 years ago226GPT2Tokenizer that performs byte-level bpe tokenization to the input tensor, based on the [hugging face version](https://huggingface.co/transformers/_modules/transformers/tokenization_gpt2.html).
a9a49850Mojimi5 years ago227
f12f42a0Nat Kershaw (MSFT)4 years ago228#### Attributes
4a0f8929Mojimi5 years ago229
f12f42a0Nat Kershaw (MSFT)4 years ago230***vocab***
a9a49850Mojimi5 years ago231
f12f42a0Nat Kershaw (MSFT)4 years ago232The **content** of the vocabulary file, its format is same with [hugging face](https://huggingface.co/gpt2/resolve/main/vocab.json).
4a0f8929Mojimi5 years ago233
f12f42a0Nat Kershaw (MSFT)4 years ago234***merges***
a9a49850Mojimi5 years ago235
f12f42a0Nat Kershaw (MSFT)4 years ago236The **content** of the merges file, its format is same with [hugging face](https://huggingface.co/gpt2/resolve/main/merges.txt).
4a0f8929Mojimi5 years ago237
f12f42a0Nat Kershaw (MSFT)4 years ago238***padding_length(optional)***
4a0f8929Mojimi5 years ago239
f12f42a0Nat Kershaw (MSFT)4 years ago240When the input is a set of query, the tokenized result is ragged tensor, so we need to pad the tensor to tidy tensor and the `padding_length` indicates the strategy of the padding. When the padding_length equals -1, we will pad the tensor to length of longest row. When the padding_length is more than 0, we will pad the tensor to the number of padding_length.
4a0f8929Mojimi5 years ago241
f12f42a0Nat Kershaw (MSFT)4 years ago242The default value of `padding_length` is -1.
4a0f8929Mojimi5 years ago243
f12f42a0Nat Kershaw (MSFT)4 years ago244#### Inputs
4a0f8929Mojimi5 years ago245
246***data: tensor(string)***
a9a49850Mojimi5 years ago247
f12f42a0Nat Kershaw (MSFT)4 years ago248The string tensor for tokenization
4a0f8929Mojimi5 years ago249
250#### Outputs
251
f12f42a0Nat Kershaw (MSFT)4 years ago252***input_ids: tensor(int64)***
a9a49850Mojimi5 years ago253
f12f42a0Nat Kershaw (MSFT)4 years ago254The tokenized ids of input
255
256***attention_mask: tensor(int64)***
257
258A tensor indicates which part of input_ids is padded.
4a0f8929Mojimi5 years ago259
260#### Examples
261
262
263```python
f12f42a0Nat Kershaw (MSFT)4 years ago264def get_file_content(path):
265with open(path, "rb") as file:
266return file.read()
4a0f8929Mojimi5 years ago267
268node = onnx.helper.make_node(
f12f42a0Nat Kershaw (MSFT)4 years ago269'GPT2Tokenizer',
4a0f8929Mojimi5 years ago270inputs=['x'],
f12f42a0Nat Kershaw (MSFT)4 years ago271outputs=['y'],
272vocab=get_file_content(vocabulary_file),
273merges=get_file_content(merges_file)
4a0f8929Mojimi5 years ago274)
275
f12f42a0Nat Kershaw (MSFT)4 years ago276x = ["hey cortana"]
277y = np.array([20342, 12794, 2271], dtype=np.int64)
4a0f8929Mojimi5 years ago278
279expect(node, inputs=[x], outputs=[y],
f12f42a0Nat Kershaw (MSFT)4 years ago280name='test_gpt2_tokenizer')
4a0f8929Mojimi5 years ago281```
282</details>
283
f12f42a0Nat Kershaw (MSFT)4 years ago284### WordpieceTokenizer
4a0f8929Mojimi5 years ago285
f12f42a0Nat Kershaw (MSFT)4 years ago286<details>
287<summary>WordpieceTokenizer details</summary>
4a0f8929Mojimi5 years ago288
289
f12f42a0Nat Kershaw (MSFT)4 years ago290WordpieceTokenizer that performs WordPiece tokenization to the input tensor,
291based on the [hugging face version](https://huggingface.co/transformers/model_doc/bert.html#WordpieceTokenizer).
292[WordpieceTokenizer](https://github.com/tensorflow/text/blob/master/docs/api_docs/python/text/WordpieceTokenizer.md)
293from *tensorflow_text* can be implemented by a pair of nodes
294*RegexSplitWithOffets* followed by *WordpieceTokenizer*.
295it
4a0f8929Mojimi5 years ago296
f12f42a0Nat Kershaw (MSFT)4 years ago297#### Attributes
4a0f8929Mojimi5 years ago298
f12f42a0Nat Kershaw (MSFT)4 years ago299***vocab***
4a0f8929Mojimi5 years ago300
f12f42a0Nat Kershaw (MSFT)4 years ago301The **content** of the vocabulary file, its format is same with
302[hugging face](https://huggingface.co/gpt2/resolve/main/vocab.json).
4a0f8929Mojimi5 years ago303
f12f42a0Nat Kershaw (MSFT)4 years ago304***suffix_indicator***
4a0f8929Mojimi5 years ago305
f12f42a0Nat Kershaw (MSFT)4 years ago306Suffix added to token not in the first position before looking into the vocabulary.
4a0f8929Mojimi5 years ago307
f12f42a0Nat Kershaw (MSFT)4 years ago308***unk_token***
a9a49850Mojimi5 years ago309
f12f42a0Nat Kershaw (MSFT)4 years ago310Unknown tokens. Every token not found in the vocabulary is replaced by this one.
4a0f8929Mojimi5 years ago311
f12f42a0Nat Kershaw (MSFT)4 years ago312***max_input_chars_per_word***
a9a49850Mojimi5 years ago313
f12f42a0Nat Kershaw (MSFT)4 years ago314Maximum number of characters per token (optional, defaults to 200).
4a0f8929Mojimi5 years ago315
316#### Inputs
317
318***data: tensor(string)***
a9a49850Mojimi5 years ago319
f12f42a0Nat Kershaw (MSFT)4 years ago320The string tensor for tokenization
4a0f8929Mojimi5 years ago321
f12f42a0Nat Kershaw (MSFT)4 years ago322***row_indices: tensor(int64)*** Empty or the fndices of every first token of input sentences.
323`indices[i+1] - indices[i]` is the number of tokens in input `i`.
4a0f8929Mojimi5 years ago324
f12f42a0Nat Kershaw (MSFT)4 years ago325[WordpieceTokenizer](https://github.com/tensorflow/text/blob/master/docs/api_docs/python/text/WordpieceTokenizer.md)
326includes two steps. The first one splits sentences into words and then splits
327every work into tokens. This operator only implements the second step.
328The first one can be done with operator *StringRegexSplit*.
329This parameter can either be empty or it can be the third output
330of operator *StringRegexSplit*.
a9a49850Mojimi5 years ago331
f12f42a0Nat Kershaw (MSFT)4 years ago332#### Outputs
4a0f8929Mojimi5 years ago333
f12f42a0Nat Kershaw (MSFT)4 years ago334***tokens: tensor(string)*** Every token.
4a0f8929Mojimi5 years ago335
f12f42a0Nat Kershaw (MSFT)4 years ago336***token_indices: tensor(int32)*** Indices of each token. -1 means a token outside the vocabulary.
337
338***row_indices: tensor(int64)*** Indices of every first token of input sentences.
339`indices[i+1] - indices[i]` is the number of tokens in input `i`.
340These are updates row indices given as inputs or new ones if the second input is empty.
4a0f8929Mojimi5 years ago341
342#### Examples
343
344
345```python
f12f42a0Nat Kershaw (MSFT)4 years ago346words = ["want", "##want",
347"##ed", "wa", "un", "runn", "##ing"]
348vocab = {w: i + 10 for i, w in enumerate(words)}
349st = json.dumps(vocab)
350nodes = []
351mkv = helper.make_tensor_value_info
352reg = helper.make_tensor(
353"pattern", onnx_proto.TensorProto.STRING, [1, ], ["(\\s)".encode('ascii')])
354reg_empty = helper.make_tensor(
355"keep_pattern", onnx_proto.TensorProto.STRING, [0, ], [])
4a0f8929Mojimi5 years ago356
f12f42a0Nat Kershaw (MSFT)4 years ago357nodes = [
358helper.make_node(
359'StringRegexSplitWithOffsets,
360inputs=['text', 'pattern', 'keep_pattern'],
361outputs=['words', 'begin_end', 'indices'],
362name='StringRegexPlsitOpName',
363domain='ai.onnx.contrib'),
364helper.make_node(
365'WordpieceTokenizer',
366inputs=['words', 'indices'],
367outputs=['out0', 'out1', 'out2'],
368name='WordpieceTokenizerOpName',
369domain='ai.onnx.contrib',
370vocab=st.encode('utf-8'),
371suffix_indicator="##",
372unk_token="[UNK]")
373]
374inputs = [mkv('text', onnx_proto.TensorProto.STRING, [None])]
375graph = helper.make_graph(
376nodes, 'test0', inputs, [
377mkv('out0', onnx_proto.TensorProto.STRING, [None]),
378mkv('out1', onnx_proto.TensorProto.INT32, [None]),
379mkv('out2', onnx_proto.TensorProto.INT64, [None]),
380mkv('words', onnx_proto.TensorProto.STRING, [None]),
381mkv('indices', onnx_proto.TensorProto.INT64, [None])],
382[reg, reg_empty])
383model = helper.make_model(
384graph, opset_imports=[helper.make_operatorsetid(domain, 1)])
4a0f8929Mojimi5 years ago385
280ec289Edward Chen3 years ago386text = np.array(["unwanted running", "unwantedX running"], dtype=object)
f12f42a0Nat Kershaw (MSFT)4 years ago387tokens = np.array(['un', '##want', '##ed', 'runn', '##ing', 'un', '##want', '##ed',
388'[UNK]', 'runn', '##ing'], dtype=object),
389indices = np.array([14, 11, 12, 15, 16, 14, 11, 12, -1, 15, 16], dtype=int32)
390row_indices = np.array([ 0, 5, 11], dtype=int64)
4a0f8929Mojimi5 years ago391
f12f42a0Nat Kershaw (MSFT)4 years ago392expect(model, inputs=[text], outputs=[tokens, indices, row_indices],
393name='test_bert_tokenizer')
4a0f8929Mojimi5 years ago394```
a9a49850Mojimi5 years ago395
4a0f8929Mojimi5 years ago396</details>
397
f12f42a0Nat Kershaw (MSFT)4 years ago398### SentencepieceTokenizer
4a0f8929Mojimi5 years ago399
f12f42a0Nat Kershaw (MSFT)4 years ago400<details>
401<summary>SentencepieceTokenizer details</summary>
4a0f8929Mojimi5 years ago402
f12f42a0Nat Kershaw (MSFT)4 years ago403SentencepieceTokenizer replicates [SentencepieceTokenizer](https://github.com/tensorflow/text/blob/master/docs/api_docs/python/text/SentencepieceTokenizer.md).
4a0f8929Mojimi5 years ago404
f12f42a0Nat Kershaw (MSFT)4 years ago405#### Inputs
4a0f8929Mojimi5 years ago406
f12f42a0Nat Kershaw (MSFT)4 years ago407***data: tensor(string)*** The string tensor for tokenization
4a0f8929Mojimi5 years ago408
f12f42a0Nat Kershaw (MSFT)4 years ago409***nbest_size: tensor(int64)*** A scalar for sampling. nbest_size = {0,1}: No sampling is performed.
410(default) nbest_size > 1: samples from the nbest_size results. nbest_size < 0: assuming that
411nbest_size is infinite and samples from the all hypothesis (lattice) using
412forward-filtering-and-backward-sampling algorithm.
4a0f8929Mojimi5 years ago413
f12f42a0Nat Kershaw (MSFT)4 years ago414***alpha: tensor(float)*** A scalar for a smoothing parameter. Inverse temperature for probability rescaling.
4a0f8929Mojimi5 years ago415
f12f42a0Nat Kershaw (MSFT)4 years ago416***reverse: tensor(bool)*** Reverses the tokenized sequence (Default = false)
4a0f8929Mojimi5 years ago417
f12f42a0Nat Kershaw (MSFT)4 years ago418***add_bos: tensor(bool)*** Add beginning of sentence token to the result (Default = false)
4a0f8929Mojimi5 years ago419
f12f42a0Nat Kershaw (MSFT)4 years ago420***add_eos: tensor(bool)*** Add end of sentence token to the result (Default = false).
421When reverse=True beginning/end of sentence tokens are added after reversing.
4a0f8929Mojimi5 years ago422
423#### Attributes
424
f12f42a0Nat Kershaw (MSFT)4 years ago425***model: string*** The sentencepiece model serialized proto as stored as a string.
4a0f8929Mojimi5 years ago426
427#### Outputs
428
f12f42a0Nat Kershaw (MSFT)4 years ago429***tokens: tensor(int32)*** Indices of each token.
a9a49850Mojimi5 years ago430
f12f42a0Nat Kershaw (MSFT)4 years ago431***indices: tensor(int64)*** Indices of every first token of input sentences.
432`indices[i+1] - indices[i]` is the number of tokens in input `i`.
4a0f8929Mojimi5 years ago433
f12f42a0Nat Kershaw (MSFT)4 years ago434Tokenized result of the input
4a0f8929Mojimi5 years ago435
436#### Examples
437
438
439```python
f12f42a0Nat Kershaw (MSFT)4 years ago440
441url = "https://github.com/microsoft/ort-customops/raw/main/test/data/test_sentencepiece_ops_model__6.txt"
442with urllib.request.urlopen(url) as f:
443content = f.read()
444model = np.array(list(base64.decodebytes(content.encode())), dtype=np.uint8)
4a0f8929Mojimi5 years ago445
446node = onnx.helper.make_node(
f12f42a0Nat Kershaw (MSFT)4 years ago447'SentencepieceTokenizer',
448inputs=['inputs', 'nbest_size', 'alpha', 'add_bos', 'add_eos', 'reverse'],
449outputs=['indices', 'output'],
450mapping_file_name='vocabulary.txt',
451unmapping_value="unknown_word",
85ffb941Zhipeng Han1 years ago452model=model,
453domain='ai.onnx.contrib'
4a0f8929Mojimi5 years ago454)
455
280ec289Edward Chen3 years ago456inputs = np.array(["Hello world", "Hello world louder"], dtype=object),
f12f42a0Nat Kershaw (MSFT)4 years ago457nbest_size = np.array([0], dtype=np.float32),
458alpha = np.array([0], dtype=np.float32),
459add_bos = np.array([0], dtype=np.bool_),
460add_eos = np.array([0], dtype=np.bool_),
461reverse = np.array([0], dtype=np.bool_)
4a0f8929Mojimi5 years ago462
85ffb941Zhipeng Han1 years ago463tokens = np.array([17486, 1017, 17486, 1017, 155, 21869], dtype=np.int32)
464indices = np.array([0, 2, 6], dtype=np.int64)
4a0f8929Mojimi5 years ago465
f12f42a0Nat Kershaw (MSFT)4 years ago466expect(node, inputs=[inputs, nbest_size, alpha, add_bos, add_eos, reverse],
467outputs=[tokens, indices], name='sp')
4a0f8929Mojimi5 years ago468```
469</details>
abdd5b1bMojimi4 years ago470
471
f12f42a0Nat Kershaw (MSFT)4 years ago472### BasicTokenizer
473
474<details>
475<summary>BasicTokenizer details</summary>
abdd5b1bMojimi4 years ago476
f12f42a0Nat Kershaw (MSFT)4 years ago477TODO: is this still supported?
478
479BasicTokenizer performs basic tokenization to input string tensor, based on [basic tokenizer in BertTokenizer(hugging face version)](https://huggingface.co/transformers/_modules/transformers/models/bert/tokenization_bert.html#BertTokenizer).
abdd5b1bMojimi4 years ago480
481#### Inputs
482
f12f42a0Nat Kershaw (MSFT)4 years ago483***text: tensor(string)*** The string tensor for tokenization
abdd5b1bMojimi4 years ago484
f12f42a0Nat Kershaw (MSFT)4 years ago485#### Attributes
abdd5b1bMojimi4 years ago486
f12f42a0Nat Kershaw (MSFT)4 years ago487***do_lower_case: int64_t*** (default is 1, 1 represents True, 0 represents False)
abdd5b1bMojimi4 years ago488
f12f42a0Nat Kershaw (MSFT)4 years ago489Whether or not to lowercase the input when tokenizing.
abdd5b1bMojimi4 years ago490
f12f42a0Nat Kershaw (MSFT)4 years ago491***tokenize_chinese_chars: int64_t*** (default is 1, 1 represents True, 0 represents False)
abdd5b1bMojimi4 years ago492
f12f42a0Nat Kershaw (MSFT)4 years ago493Whether or not to tokenize Chinese characters.
abdd5b1bMojimi4 years ago494
f12f42a0Nat Kershaw (MSFT)4 years ago495***strip_accents: int64_t*** (default is 1, 1 represents True, 0 represents False)
abdd5b1bMojimi4 years ago496
f12f42a0Nat Kershaw (MSFT)4 years ago497Whether or not to strip all accents. If this option is not specified, then it will be determined by the
498value for :obj:`lowercase` (as in the original BERT).
abdd5b1bMojimi4 years ago499
f12f42a0Nat Kershaw (MSFT)4 years ago500***tokenize_punctuation: int64_t*** (default is 0, 1 represents True, 0 represents False)
abdd5b1bMojimi4 years ago501
f12f42a0Nat Kershaw (MSFT)4 years ago502Splits punctuation on a piece of text.
503
504***remove_control_chars: int64_t*** (default is 0, 1 represents True, 0 represents False)
505
506Remove control chars(such as NUL, BEL) in the text.
507
508#### Outputs
509
510***tokens: tensor(string)*** Tokenized tokens.
511
512#### Examples
abdd5b1bMojimi4 years ago513
514```python
f12f42a0Nat Kershaw (MSFT)4 years ago515import transformers
516
517tokenizer = transformers.BasicTokenizer()
abdd5b1bMojimi4 years ago518
519node = onnx.helper.make_node(
f12f42a0Nat Kershaw (MSFT)4 years ago520'BasicTokenizer',
521inputs=['text'],
522outputs=['tokens'],
abdd5b1bMojimi4 years ago523)
524
280ec289Edward Chen3 years ago525inputs = np.array([ "Hello world louder"], dtype=object),
f12f42a0Nat Kershaw (MSFT)4 years ago526tokens = np.array(tokenizer(inputs), dtype=int32)
abdd5b1bMojimi4 years ago527
f12f42a0Nat Kershaw (MSFT)4 years ago528expect(node, inputs=[inputs],
529outputs=[tokens], name='test_basic_tokenizer')
abdd5b1bMojimi4 years ago530```
531</details>
4a0f8929Mojimi5 years ago532
533
f12f42a0Nat Kershaw (MSFT)4 years ago534## String operators
a9a49850Mojimi5 years ago535
f12f42a0Nat Kershaw (MSFT)4 years ago536### StringEqual
9653f523Mojimi5 years ago537
f12f42a0Nat Kershaw (MSFT)4 years ago538<details>
539<summary>StringEqual details</summary>
9653f523Mojimi5 years ago540
f12f42a0Nat Kershaw (MSFT)4 years ago541Compares two strings and returns true if they are equal and false if not.
a9a49850Mojimi5 years ago542
4a0f8929Mojimi5 years ago543#### Inputs
544
f12f42a0Nat Kershaw (MSFT)4 years ago545***x: tensor(string)***
a9a49850Mojimi5 years ago546
f12f42a0Nat Kershaw (MSFT)4 years ago547The first string input
548
549***x: tensor(string)***
550
551The second string input
4a0f8929Mojimi5 years ago552
553#### Outputs
554
f12f42a0Nat Kershaw (MSFT)4 years ago555***z: tensor(boolean)***
9653f523Mojimi5 years ago556
f12f42a0Nat Kershaw (MSFT)4 years ago557String with replacements.
9653f523Mojimi5 years ago558
f12f42a0Nat Kershaw (MSFT)4 years ago559</details>
a9a49850Mojimi5 years ago560
4a0f8929Mojimi5 years ago561
f12f42a0Nat Kershaw (MSFT)4 years ago562### StringHash
4a0f8929Mojimi5 years ago563
564<details>
f12f42a0Nat Kershaw (MSFT)4 years ago565<summary>StringHash details</summary>
4a0f8929Mojimi5 years ago566
567
f12f42a0Nat Kershaw (MSFT)4 years ago568Hashes the input string based on the number of buckets
4a0f8929Mojimi5 years ago569
f12f42a0Nat Kershaw (MSFT)4 years ago570#### Inputs
4a0f8929Mojimi5 years ago571
f12f42a0Nat Kershaw (MSFT)4 years ago572***input: tensor(string)***
4a0f8929Mojimi5 years ago573
f12f42a0Nat Kershaw (MSFT)4 years ago574The string to hash
4a0f8929Mojimi5 years ago575
f12f42a0Nat Kershaw (MSFT)4 years ago576***num_buckets: tensor(int64)***
4bc5c962Xavier Dupré5 years ago577
f12f42a0Nat Kershaw (MSFT)4 years ago578The number of buckets (must be equal to 1?)
4a0f8929Mojimi5 years ago579
f12f42a0Nat Kershaw (MSFT)4 years ago580#### Outputs
4bc5c962Xavier Dupré5 years ago581
f12f42a0Nat Kershaw (MSFT)4 years ago582***name: tensor(int64)***
4bc5c962Xavier Dupré5 years ago583
f12f42a0Nat Kershaw (MSFT)4 years ago584The hash value of the string
4bc5c962Xavier Dupré5 years ago585
f12f42a0Nat Kershaw (MSFT)4 years ago586</details>
4bc5c962Xavier Dupré5 years ago587
588
f12f42a0Nat Kershaw (MSFT)4 years ago589### StringHashFast
4bc5c962Xavier Dupré5 years ago590
f12f42a0Nat Kershaw (MSFT)4 years ago591<details>
592<summary>StringHashFast details</summary>
4bc5c962Xavier Dupré5 years ago593
594
f12f42a0Nat Kershaw (MSFT)4 years ago595A faster implementation of StringHash.
596
597</details>
598
599
600### StringJoin
601
602<details>
603<summary>StringJoin details</summary>
604
605
606Join an array of strings
4a0f8929Mojimi5 years ago607
608#### Inputs
609
f12f42a0Nat Kershaw (MSFT)4 years ago610***input_X: tensor(string)***
a9a49850Mojimi5 years ago611
f12f42a0Nat Kershaw (MSFT)4 years ago612The input array of strings
4a0f8929Mojimi5 years ago613
f12f42a0Nat Kershaw (MSFT)4 years ago614***input_sep: tensor(string)***
4bc5c962Xavier Dupré5 years ago615
f12f42a0Nat Kershaw (MSFT)4 years ago616The string separator for the resulting joing
4bc5c962Xavier Dupré5 years ago617
f12f42a0Nat Kershaw (MSFT)4 years ago618***input_axis: tensor(int64)***
4a0f8929Mojimi5 years ago619
f12f42a0Nat Kershaw (MSFT)4 years ago620The axis along which to joing
4bc5c962Xavier Dupré5 years ago621
f12f42a0Nat Kershaw (MSFT)4 years ago622#### Outputs
a9a49850Mojimi5 years ago623
f12f42a0Nat Kershaw (MSFT)4 years ago624***out: tensor(string)***
625
626The resulting joined string
4a0f8929Mojimi5 years ago627
628#### Examples
629
630
f12f42a0Nat Kershaw (MSFT)4 years ago631```bash
4bc5c962Xavier Dupré5 years ago632
f12f42a0Nat Kershaw (MSFT)4 years ago633input_X = [["a", "b", "c"], ["aa", "bb", ""]]
634input_sep=";"
635input_axis = 1
4bc5c962Xavier Dupré5 years ago636
f12f42a0Nat Kershaw (MSFT)4 years ago637out = ["a;b;c", "aa;bb;"]
638
639input_axis = 0
640
641out = ['a;aa', 'b;bb', 'c;']
4bc5c962Xavier Dupré5 years ago642
643
4a0f8929Mojimi5 years ago644</details>
645
a32f9bc2Xavier Dupré5 years ago646
f12f42a0Nat Kershaw (MSFT)4 years ago647### StringRegexReplace
648
649<details>
650<summary>StringRegexReplace details</summary>
651
652
653String replacement based on [Re2-format](https://github.com/google/re2/wiki/Syntax) regular expressions.
a32f9bc2Xavier Dupré5 years ago654
655#### Inputs
656
f12f42a0Nat Kershaw (MSFT)4 years ago657***text: tensor(string)***
a32f9bc2Xavier Dupré5 years ago658
f12f42a0Nat Kershaw (MSFT)4 years ago659String tensor to extract slices from.
a32f9bc2Xavier Dupré5 years ago660
f12f42a0Nat Kershaw (MSFT)4 years ago661***pattern: tensor(string)***
a32f9bc2Xavier Dupré5 years ago662
f12f42a0Nat Kershaw (MSFT)4 years ago663Pattern of the regular expression.
a32f9bc2Xavier Dupré5 years ago664
f12f42a0Nat Kershaw (MSFT)4 years ago665***rewrite: tensor(string)***
a32f9bc2Xavier Dupré5 years ago666
f12f42a0Nat Kershaw (MSFT)4 years ago667Replacement.
a32f9bc2Xavier Dupré5 years ago668
4c201e78Xavier Dupré5 years ago669#### Attributes
a32f9bc2Xavier Dupré5 years ago670
f12f42a0Nat Kershaw (MSFT)4 years ago671***global_replace: int64*** (default is 1)
a32f9bc2Xavier Dupré5 years ago672
f12f42a0Nat Kershaw (MSFT)4 years ago673Replace all strings matching the pattern or the first one.
a32f9bc2Xavier Dupré5 years ago674
f12f42a0Nat Kershaw (MSFT)4 years ago675#### Outputs
a32f9bc2Xavier Dupré5 years ago676
f12f42a0Nat Kershaw (MSFT)4 years ago677***output: tensor(string)***
a32f9bc2Xavier Dupré5 years ago678
f12f42a0Nat Kershaw (MSFT)4 years ago679String with replacements.
a32f9bc2Xavier Dupré5 years ago680
681#### Examples
682
f12f42a0Nat Kershaw (MSFT)4 years ago683```python
684
685node = onnx.helper.make_node(
686'StringRegexReplace',
687inputs=['text', 'pattern', 'rewrite'],
688outputs=['y'],
689)
690
691text = np.array([['def myfunc():'], ['def dummy():']])
692pattern = np.array([r'def\s+([a-zA-Z_][a-zA-Z_0-9]*)\s*\(\s*\):'])
693rewrite = np.array([r'static PyObject* py_\1(void) {'])
694y = [['static PyObject* py_myfunc(void) {'],
695['static PyObject* py_dummy(void) {']]
696
697expect(node, inputs=[text, pattern, rewrite], outputs=[y],
698name='test_string_regex_replace')
699```
700
701</details>
702
703### StringECMARegexReplace
704
a32f9bc2Xavier Dupré5 years ago705<details>
f12f42a0Nat Kershaw (MSFT)4 years ago706<summary>StringECMARegexReplace details</summary>
707
708String replacement based on [ECMA-format](https://en.cppreference.com/w/cpp/regex/ecmascript) regular expressions.
709
710#### Inputs
711
712***text: tensor(string)***
713
714String tensor to extract slices from.
715
716***pattern: tensor(string)***
717
718Pattern of the regular expression.
719
720***rewrite: tensor(string)***
721
722Replacement.
723
724#### Attributes
725
726***global_replace: int64*** (default is 1)
727
728Replace all strings matching the pattern or the first one.
729
730
731***ignore_case: int64*** (default is 0)
732
733Replace
734
735#### Outputs
736
737***output: tensor(string)***
738
739String with replacements.
740
741#### Examples
742
a32f9bc2Xavier Dupré5 years ago743
744```python
745
f12f42a0Nat Kershaw (MSFT)4 years ago746node = onnx.helper.make_node(
747'StringRegexReplace',
748inputs=['text', 'pattern', 'rewrite'],
749outputs=['y'],
750)
751
752text = np.array([['def myfunc():'], ['def dummy():']])
753pattern = np.array([r'def\s+([a-zA-Z_][a-zA-Z_0-9]*)\s*\(\s*\):'])
754rewrite = np.array([r'static PyObject* py_$1(void) {'])
755y = [['static PyObject* py_myfunc(void) {'],
756['static PyObject* py_dummy(void) {']]
757
758expect(node, inputs=[text, pattern, rewrite], outputs=[y],
759name='test_string_regex_replace')
760```
761
762</details>
763
764
765
766### StringSplit
767
768TODO
769
770### StringUpper
771
772TODO
773
774### StringLower
775
776TODO
777
778### StringLength
779
780<details>
781<summary>StringECMARegexReplace details</summary>
782
783Get the length of each string element in input tensor. Similar to the function `len("abcde"")` in python.
784
785#### Inputs
786
787***data: tensor(string)***
788
789String tensor to get length of its each string element.
790
791#### Outputs
792
793***output: tensor(int64)***
794
795Data length tensor.
796
797#### Examples
798
799
800```python
a32f9bc2Xavier Dupré5 years ago801
802node = onnx.helper.make_node(
f12f42a0Nat Kershaw (MSFT)4 years ago803'StringLength',
804inputs=['x'],
805outputs=['y']
a32f9bc2Xavier Dupré5 years ago806)
807
f12f42a0Nat Kershaw (MSFT)4 years ago808x = ["abcdef", "hijkl"]
809y = np.array([len(x[0]), len(x[1])], dtype=np.int64)
a32f9bc2Xavier Dupré5 years ago810
811
f12f42a0Nat Kershaw (MSFT)4 years ago812expect(node, inputs=[x], outputs=[y],
813name='test_string_length')
a32f9bc2Xavier Dupré5 years ago814```
815</details>
f12f42a0Nat Kershaw (MSFT)4 years ago816
817### StringConcat
818
819<details>
820<summary>StringConcat details</summary>
821
822Concat the corresponding string in the two string tensor. Two input tensors should have the same dimension.
823
824```python
825output = []
826shape = input1.shape
827input1 = input1.flatten()
828input2 = input2.flatten()
829for i in range(len(input1)):
830output.append(input1[i] + input2[i])
831output = np.array(output).reshape(shape)
832```
833
834#### Inputs
835
836***input_1: tensor(string)***
837
838The first string tensor.
839
840***input_2: tensor(string)***
841
842The second string tensor.
843
844
845#### Outputs
846
847***output: tensor(string)***
848
849The result.
850
851#### Examples
852
853
854```python
855
856node = onnx.helper.make_node(
857'StringConcat',
858inputs=['x', 'y'],
859outputs=['result'],
860)
861
862x = np.array(["abcd", "efgh"])
863y = np.array(["wxyz", "stuv"])
864result = np.array([x[0] + y[0], x[1] + y[1]])
865
866expect(node, inputs=[x, y], outputs=[result],
867name='test_string_concat')
868```
869
870</details>
871
872### StringRegexSplitWithOffsets
873
874<details>
875<summary>StringRegexSplitWithOffsets details</summary>
876
877Splits string based on regular expressions.
878
879#### Inputs
880
881***text: tensor(string)***
882
883String tensor to extract slices from.
884
885***delim_regex_pattern: tensor(string)***
886
887Splitting attern of the regular expression.
888
889***keep_delim_regex_pattern: tensor(string)***
890
891By default, delimiters are not included in the split string results. Delimiters may be included by specifying a regex pattern keep_delim_regex_pattern.
892
893#### Outputs
894
895***words: tensor(string)*** Tensor of words.
896
897***offsets: tensor(int64)*** 2D tensor with 3 columns:
898sentence index, position of the first character, position of the last one (excluded)
899
900***row_indices: tensor(int64)*** Indices of every first token of input sentences.
901`row_indices[i+1] - row_indices[i]` is the number of tokens in input `i`.
902These are updates row indices given as inputs or new ones if the second input is empty.
903
904
905#### Examples
906
907
908```python
909
910node = onnx.helper.make_node(
911'StringRegexSplit',
912inputs=['text', 'pattern', 'rewrite'],
913outputs=['y', 'begin_end', 'indices'],
914)
915
916text = np.array(["hello there"])
917pattern = np.array([r'\s'])
918rewrite = np.array([r'\s'])
919y = np.array(["hello", " ", "there"])
920z1 = np.array([[0, 0, 5],
921[0, 5, 6],
922[0, 6, 11]], dtype=np.int64)
923z2 = np.array([0, 2], dtype=np.int64)
924
925expect(node, inputs=[text, pattern, rewrite], outputs=[y, z1, z2],
926name='test_string_regex_replace')
927```
928
929</details>
930
931
932### StringECMARegexSplitWithOffsets
933
934TODO
935
936### VectorToString
937
938<details>
939<summary>VectorToString details</summary>
940
941VectorToString is the contrary operation to the `StringToVector` , they share same format of mapping table:
942
943<string>\t<scalar_1>\s<scalar_2>\s<scalar_3>...<scalar_n>
944
945Unmapped vector will output the value of the attribute `unk`.
946
947Example:
948
949*Attributes:*
950
951- `map`:
952```
953a 0 0 1 2
954b 0 1 2 3
955d 0 1 3 4
956```
957
958- `unk`: "unknown_word"
959
960*Inputs:*
961- data: [[0,0,1,2],[0,1,3,4],[0,0,0,0]]
962
963*Ouputs:*
964- output: ["a", "d", "unknown_word" ]
965
966#### Attributes
967
968***mapping_file_name***
969
970the formative mapping table
971
972***unmapping_value***
973
974the result returned when a vector aren't found in the map
975
976#### Inputs
977
978***data: tensor(T)***
979
980Input tensor
981
982#### Outputs
983
984***output: tensor(string)***
985
986The mapping result of the input
987
988#### Type Constraints
989***T:tensor(uint8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(int8), tensor(int16), tensor(int32), tensor(int64), tensor(bfloat16), tensor(float16), tensor(float), tensor(double), tensor(bool)***
990
991Constrain input and output types to numerical tensors.
992
993
994#### Examples
995
996
997```python
998mapping_table = \
999"""
1000a 0 0 1 2
1001b 0 1 2 3
1002d 0 1 3 4
1003"""
1004
1005node = onnx.helper.make_node(
1006'VectorToString',
1007inputs=['x'],
1008outputs=['y'],
1009map=mapping_table,
1010unk="unknown_word"
1011)
1012
1013
1014x = np.array([[0,0,1,2],[0,1,3,4],[0,0,0,0]], type=np.int64)
1015y = ["a", "d", "unknown_word"]
1016
1017
1018expect(node, inputs=[x], outputs=[y],
1019name='test_vector_to_string')
1020```
1021</details>
1022
1023
1024### StringToVector
1025
1026<details>
1027<summary>StringToVector details</summary>
1028
1029StringToVector will map each string element in the input to the corresponding vector according to the mapping file. The mapping file is a utf-8 encoding text file in tsv format:
1030
1031<string>\t<scalar_1>\s<scalar_2>\s<scalar_3>...<scalar_n>
1032
1033Unmapped string will output the value of the attribute `unmapping_value`.
a32f9bc2Xavier Dupré5 years ago1034
f12f42a0Nat Kershaw (MSFT)4 years ago1035Example:
4a0f8929Mojimi5 years ago1036
f12f42a0Nat Kershaw (MSFT)4 years ago1037*Attributes:*
4a0f8929Mojimi5 years ago1038
f12f42a0Nat Kershaw (MSFT)4 years ago1039- `mapping_file_name`: vocabulary.txt
1040```
1041a 0 0 1 2
1042b 0 1 2 3
1043d 0 1 3 4
1044```
1045
1046- `unmapping_value`: [0 0 0 0]
4a0f8929Mojimi5 years ago1047
f12f42a0Nat Kershaw (MSFT)4 years ago1048*Inputs:*
1049- data: ["a", "d", "e"]
1050
1051*Ouputs:*
1052- output: [[0,0,1,2],[0,1,3,4],[0,0,0,0]]
4290400eMojimi4 years ago1053
1054#### Attributes
1055
f12f42a0Nat Kershaw (MSFT)4 years ago1056***mapping_file_name:string***
4290400eMojimi4 years ago1057
f12f42a0Nat Kershaw (MSFT)4 years ago1058The name of your string to vector mapping file.
4290400eMojimi4 years ago1059
f12f42a0Nat Kershaw (MSFT)4 years ago1060***unmapping_value:list(int)***
4290400eMojimi4 years ago1061
f12f42a0Nat Kershaw (MSFT)4 years ago1062Mapping result for unmapped string
4290400eMojimi4 years ago1063
f12f42a0Nat Kershaw (MSFT)4 years ago1064#### Inputs
4290400eMojimi4 years ago1065
f12f42a0Nat Kershaw (MSFT)4 years ago1066***data: tensor(string)***
4290400eMojimi4 years ago1067
f12f42a0Nat Kershaw (MSFT)4 years ago1068Input tensor
4290400eMojimi4 years ago1069
f12f42a0Nat Kershaw (MSFT)4 years ago1070#### Outputs
4290400eMojimi4 years ago1071
f12f42a0Nat Kershaw (MSFT)4 years ago1072***output: tensor(T)***
4290400eMojimi4 years ago1073
f12f42a0Nat Kershaw (MSFT)4 years ago1074The mapping result of the input
4a0f8929Mojimi5 years ago1075
f12f42a0Nat Kershaw (MSFT)4 years ago1076#### Type Constraints
1077***T:tensor(uint8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(int8), tensor(int16), tensor(int32), tensor(int64), tensor(bfloat16), tensor(float16), tensor(float), tensor(double), tensor(bool)***
4a0f8929Mojimi5 years ago1078
f12f42a0Nat Kershaw (MSFT)4 years ago1079Constrain input and output types to numerical tensors.
4290400eMojimi4 years ago1080
1081#### Examples
1082
1083
1084```python
f12f42a0Nat Kershaw (MSFT)4 years ago1085# what's in vocabulary.txt
4290400eMojimi4 years ago1086
f12f42a0Nat Kershaw (MSFT)4 years ago1087mapping_table = \
1088"""
1089a 0 0 1 2
1090b 0 1 2 3
1091d 0 1 3 4
1092"""
4290400eMojimi4 years ago1093
1094node = onnx.helper.make_node(
f12f42a0Nat Kershaw (MSFT)4 years ago1095'StringToVector',
1096inputs=['x'],
1097outputs=['y'],
1098mapping_table=mapping_table,
1099unmapping_value=[0,0,0,0]
4290400eMojimi4 years ago1100)
1101
1102
f12f42a0Nat Kershaw (MSFT)4 years ago1103x = ["a", "d", "e"]
1104y = np.array([[0,0,1,2],[0,1,3,4],[0,0,0,0]], type=np.int64)
1105
1106
1107expect(node, inputs=[x], outputs=[y],
1108name='test_string_to_vector')
4290400eMojimi4 years ago1109```
f12f42a0Nat Kershaw (MSFT)4 years ago1110
4290400eMojimi4 years ago1111</details>
1112
1113
1114
f12f42a0Nat Kershaw (MSFT)4 years ago1115### StringSlice
4290400eMojimi4 years ago1116
f12f42a0Nat Kershaw (MSFT)4 years ago1117<details>
1118<summary>StringSlice details</summary>
4290400eMojimi4 years ago1119
f12f42a0Nat Kershaw (MSFT)4 years ago1120Do the slice operation to each string element in input tensor. Similar to string slice in python
4290400eMojimi4 years ago1121
f12f42a0Nat Kershaw (MSFT)4 years ago1122```python
1123a = "abcdef"
1124b = a[1:2]
1125c = a[3:1:-1]
1126```
4290400eMojimi4 years ago1127
f12f42a0Nat Kershaw (MSFT)4 years ago1128#### Inputs
4290400eMojimi4 years ago1129
f12f42a0Nat Kershaw (MSFT)4 years ago1130***data: tensor(string)***
4290400eMojimi4 years ago1131
f12f42a0Nat Kershaw (MSFT)4 years ago1132String tensor to extract slices from.
4290400eMojimi4 years ago1133
f12f42a0Nat Kershaw (MSFT)4 years ago1134***starts: tensor(int64/int32)***
4290400eMojimi4 years ago1135
f12f42a0Nat Kershaw (MSFT)4 years ago1136The tensor of starting indices of corresponding string in data, which has same dimension of data.
4290400eMojimi4 years ago1137
f12f42a0Nat Kershaw (MSFT)4 years ago1138***ends: tensor(int64/int32)***
4290400eMojimi4 years ago1139
f12f42a0Nat Kershaw (MSFT)4 years ago1140The tensor of ending indices of corresponding string in data, which has same dimension of data.
4290400eMojimi4 years ago1141
f12f42a0Nat Kershaw (MSFT)4 years ago1142***steps(optional): tensor(int64/int32)***
4290400eMojimi4 years ago1143
f12f42a0Nat Kershaw (MSFT)4 years ago1144The tensor of slice step of corresponding string in data, which has same dimension of data.If steps is empty tensor, we will use default value 1 for each string
4290400eMojimi4 years ago1145
f12f42a0Nat Kershaw (MSFT)4 years ago1146#### Outputs
4290400eMojimi4 years ago1147
f12f42a0Nat Kershaw (MSFT)4 years ago1148***output: tensor(string)***
4290400eMojimi4 years ago1149
f12f42a0Nat Kershaw (MSFT)4 years ago1150Sliced data tensor.
4290400eMojimi4 years ago1151
f12f42a0Nat Kershaw (MSFT)4 years ago1152#### Examples
4290400eMojimi4 years ago1153
1154
f12f42a0Nat Kershaw (MSFT)4 years ago1155```python
4290400eMojimi4 years ago1156
f12f42a0Nat Kershaw (MSFT)4 years ago1157node = onnx.helper.make_node(
1158'StringSlice',
1159inputs=['x', 'starts', 'ends', 'steps'],
1160outputs=['y'],
1161)
4290400eMojimi4 years ago1162
f12f42a0Nat Kershaw (MSFT)4 years ago1163x = np.array(["abcdef", "hijkl"])
1164y = np.array([x[0][1:3:1], x[1][3:1:-1]])
1165starts = np.array([1, 3], dtype=np.int64)
1166ends = np.array([3, 1], dtype=np.int64)
1167axes = np.array([0, 1], dtype=np.int64)
1168steps = np.array([1, 1], dtype=np.int64)
4290400eMojimi4 years ago1169
f12f42a0Nat Kershaw (MSFT)4 years ago1170expect(node, inputs=[x, starts, ends, axes, steps], outputs=[y],
1171name='test_string_slice')
1172```
4290400eMojimi4 years ago1173
f12f42a0Nat Kershaw (MSFT)4 years ago1174</details>
4290400eMojimi4 years ago1175
1176
f12f42a0Nat Kershaw (MSFT)4 years ago1177### MaskedFill
4290400eMojimi4 years ago1178
f12f42a0Nat Kershaw (MSFT)4 years ago1179<details>
1180<summary>MaskedFill details</summary>
4290400eMojimi4 years ago1181
1182
f12f42a0Nat Kershaw (MSFT)4 years ago1183Fills elements of self tensor with value where mask is True. The operator is similar with [`Tensor.masked_fill_`](https://pytorch.org/docs/stable/generated/torch.Tensor.masked_fill_.html#torch.Tensor.masked_fill_) in pytorch.
4290400eMojimi4 years ago1184
1185
f12f42a0Nat Kershaw (MSFT)4 years ago1186#### Inputs
4290400eMojimi4 years ago1187
f12f42a0Nat Kershaw (MSFT)4 years ago1188***value: tensor(string)***
4290400eMojimi4 years ago1189
f12f42a0Nat Kershaw (MSFT)4 years ago1190The value to fill in with, currently we only support string type and vector&scalar dimension.
4290400eMojimi4 years ago1191
f12f42a0Nat Kershaw (MSFT)4 years ago1192***mask: tensor(bool)***
4290400eMojimi4 years ago1193
f12f42a0Nat Kershaw (MSFT)4 years ago1194The boolean mask, the dimension of mask tensor should be same with value.
4290400eMojimi4 years ago1195
f12f42a0Nat Kershaw (MSFT)4 years ago1196#### Outputs
1197
1198***output: tensor(string)***
1199
1200The filled output of input tensor.
a9a49850Mojimi5 years ago1201
4a0f8929Mojimi5 years ago1202
1203#### Examples
1204
1205
1206```python
4290400eMojimi4 years ago1207
1208node = onnx.helper.make_node(
f12f42a0Nat Kershaw (MSFT)4 years ago1209'MaskedFill',
1210inputs=['value', 'mask'],
1211outputs=['output']
4290400eMojimi4 years ago1212)
1213
1214
f12f42a0Nat Kershaw (MSFT)4 years ago1215value = np.array(["a", "b", "c", "d"])
1216mask = np.array([True, False, True, False], dtype=bool)
1217output = np.array(["a", "c"])
4290400eMojimi4 years ago1218
1219
f12f42a0Nat Kershaw (MSFT)4 years ago1220expect(node, inputs=[value, mask], outputs=[output],
1221name='test_masked_fill')
4290400eMojimi4 years ago1222```
1223</details>
1224
d853d31fRandySheriffH2 years ago1225
f12f42a0Nat Kershaw (MSFT)4 years ago1226### StringRaggedTensorToDense
4290400eMojimi4 years ago1227
f12f42a0Nat Kershaw (MSFT)4 years ago1228TODO
4290400eMojimi4 years ago1229
f12f42a0Nat Kershaw (MSFT)4 years ago1230### StringMapping
4290400eMojimi4 years ago1231
f12f42a0Nat Kershaw (MSFT)4 years ago1232TODO
4290400eMojimi4 years ago1233
f12f42a0Nat Kershaw (MSFT)4 years ago1234## Math operators
4290400eMojimi4 years ago1235
1236
f12f42a0Nat Kershaw (MSFT)4 years ago1237### Inverse
4290400eMojimi4 years ago1238
f12f42a0Nat Kershaw (MSFT)4 years ago1239TODO
4290400eMojimi4 years ago1240
f12f42a0Nat Kershaw (MSFT)4 years ago1241### NegPos
4290400eMojimi4 years ago1242
f12f42a0Nat Kershaw (MSFT)4 years ago1243TODO
4290400eMojimi4 years ago1244
f12f42a0Nat Kershaw (MSFT)4 years ago1245### SegmentExtraction
4290400eMojimi4 years ago1246
f12f42a0Nat Kershaw (MSFT)4 years ago1247TODO
4290400eMojimi4 years ago1248
f12f42a0Nat Kershaw (MSFT)4 years ago1249### SegmentSum
4290400eMojimi4 years ago1250
f12f42a0Nat Kershaw (MSFT)4 years ago1251TODO
4290400eMojimi4 years ago1252
f12f42a0Nat Kershaw (MSFT)4 years ago1253## Tensor operators
4290400eMojimi4 years ago1254
f12f42a0Nat Kershaw (MSFT)4 years ago1255### RaggedTensorToSparse
4290400eMojimi4 years ago1256
f12f42a0Nat Kershaw (MSFT)4 years ago1257TODO
4290400eMojimi4 years ago1258
f12f42a0Nat Kershaw (MSFT)4 years ago1259### RaggedTensorToDense
4290400eMojimi4 years ago1260
f12f42a0Nat Kershaw (MSFT)4 years ago1261TODO
4290400eMojimi4 years ago1262
f12f42a0Nat Kershaw (MSFT)4 years ago1263### Template
4290400eMojimi4 years ago1264
f12f42a0Nat Kershaw (MSFT)4 years ago1265<details>
1266<summary>Template details</summary>
4290400eMojimi4 years ago1267
f12f42a0Nat Kershaw (MSFT)4 years ago1268Description
4290400eMojimi4 years ago1269
f12f42a0Nat Kershaw (MSFT)4 years ago1270#### Inputs
4290400eMojimi4 years ago1271
f12f42a0Nat Kershaw (MSFT)4 years ago1272***name: tensor(type)***
4290400eMojimi4 years ago1273
f12f42a0Nat Kershaw (MSFT)4 years ago1274Description
4290400eMojimi4 years ago1275
1276#### Outputs
1277
f12f42a0Nat Kershaw (MSFT)4 years ago1278***name: tensor(type)***
4290400eMojimi4 years ago1279
f12f42a0Nat Kershaw (MSFT)4 years ago1280Description
4290400eMojimi4 years ago1281
1282#### Examples
1283
1284
1285```python
1286
1287node = onnx.helper.make_node(
f12f42a0Nat Kershaw (MSFT)4 years ago1288'StringRegexReplace',
1289inputs=['text', 'pattern', 'rewrite'],
1290outputs=['y'],
4290400eMojimi4 years ago1291)
1292
f12f42a0Nat Kershaw (MSFT)4 years ago1293text = np.array([['def myfunc():'], ['def dummy():']])
1294pattern = np.array([r'def\s+([a-zA-Z_][a-zA-Z_0-9]*)\s*\(\s*\):'])
1295rewrite = np.array([r'static PyObject* py_\1(void) {'])
1296y = [['static PyObject* py_myfunc(void) {'],
1297['static PyObject* py_dummy(void) {']]
4a0f8929Mojimi5 years ago1298
f12f42a0Nat Kershaw (MSFT)4 years ago1299expect(node, inputs=[text, pattern, rewrite], outputs=[y],
1300name='test_string_regex_replace')
4a0f8929Mojimi5 years ago1301```
f12f42a0Nat Kershaw (MSFT)4 years ago1302
4a0f8929Mojimi5 years ago1303</details>
d853d31fRandySheriffH2 years ago1304
1305
1306## Azure operators
3c22daa4Wenbing Li2 years ago1307Starting from onnxruntime-extensions 0.12, these Azure operators will be removed from the official onnxruntime-extensions packages. However, they can still be built from source using `cmake -DOCOS_ENABLE_AZURE=ON ...`.
d853d31fRandySheriffH2 years ago1308### OpenAIAudioToText
1309
1310<details>
1311<summary>OpenAIAudioToText details</summary>
1312
1313
1314OpenAIAudioToText operator talks to [openAI audio](https://platform.openai.com/docs/api-reference/audio) endpoints.
1315
1316
1317#### Attributes
1318
1319***model_uri:string***
1320
1321Endpoint uri, like "https://api.openai.com/v1/audio/transcriptions".
1322
1323***audio_format:string***
1324
1325The format of the audio, by default "wav".
1326
1327#### Inputs
1328
1329***auth_token: tensor(string)***
1330
1331An access token comes with openAI subscription.
1332
1333***model_name: tensor(string)***
1334
1335Model name to send to the endpoint, such as "whisper-1".
1336
1337***response_format: tensor(string)***
1338
1339Expected format of the response, either be "text" or "json".
1340
1341***audio_blob: tensor(uint8)***
1342
1343A byte array containing raw data from the audio file.
1344
1345#### Outputs
1346
1347***transcriptions: tensor(string)***
1348
1349
1350#### Examples
1351
4cc985faRandySheriffH2 years ago1352Note - OpenAIAudioToText operator composes a request based on last part of the input and output names split by "/",
1353
1354Meaning for input names, they must be of format:
1355- auth_token: "whatever-name-you-want-to-use"
1356- model_name: ".../.../.../model_name"
1357- response_format: ".../.../.../response_format"
1358- audio_blob: ".../.../.../file"
1359
1360for output name, it must be of format:
1361- transcriptions: ".../.../.../transcriptions"
1362
1363Hence there could be multiple OpenAIAudioToText operators accepting different inputs inside a model, and give varied outputs.
1364
1365Pls find sample code below for a better illustration.
1366
d853d31fRandySheriffH2 years ago1367
1368```python
1369
1370import os
1371import numpy as np
1372
1373from onnx import *
1374from onnxruntime_extensions import PyOrtFunction, util, get_library_path
1375from onnxruntime import *
1376
1377
4cc985faRandySheriffH2 years ago1378openai_model_uri = os.getenv('URI', '') # read uri from env
1379openai_auth_token = os.getenv('AUTH', '') # read auto token from env
1380
1381
d853d31fRandySheriffH2 years ago1382def create_openai_audio_model():
1383auth_token = helper.make_tensor_value_info('auth_token', TensorProto.STRING, [1])
4cc985faRandySheriffH2 years ago1384model = helper.make_tensor_value_info('node_1/model_name', TensorProto.STRING, [1])
1385response_format = helper.make_tensor_value_info('node_1/response_format', TensorProto.STRING, [-1])
1386file = helper.make_tensor_value_info('node_1/file', TensorProto.UINT8, [-1])
1387transcriptions = helper.make_tensor_value_info('node_1/transcriptions', TensorProto.STRING, [-1])
d853d31fRandySheriffH2 years ago1388
1389invoker = helper.make_node('OpenAIAudioToText',
4cc985faRandySheriffH2 years ago1390['auth_token', 'node_1/model_name', 'node_1/response_format', 'node_1/file'], # names must follow the format
1391['node_1/transcriptions'], # names must follow the format
d853d31fRandySheriffH2 years ago1392domain='com.microsoft.extensions',
1393name='audio_invoker',
4cc985faRandySheriffH2 years ago1394model_uri=openai_model_uri,
d853d31fRandySheriffH2 years ago1395audio_format='wav')
1396
1397graph = helper.make_graph([invoker], 'graph', [auth_token, model, response_format, file], [transcriptions])
1398model = helper.make_model(graph,
1399opset_imports=[helper.make_operatorsetid('com.microsoft.extensions', 1)])
1400
1401onnx.save(model, 'openai_audio.onnx')
1402
1403
1404create_openai_audio_model()
1405opt = SessionOptions()
1406opt.register_custom_ops_library(get_library_path())
1407sess = InferenceSession(os.path.join(test_data_dir, "openai_audio.onnx"),
1408opt, providers=["CPUExecutionProvider", "AzureExecutionProvider"])
4cc985faRandySheriffH2 years ago1409auth_token = np.array([openai_auth_token])
d853d31fRandySheriffH2 years ago1410model = np.array(['whisper-1'])
1411response_format = np.array(['text'])
1412
1413with open(os.path.join(test_data_dir, "test16.wav"), "rb") as _f:
1414audio_blob = np.asarray(list(_f.read()), dtype=np.uint8)
1415ort_inputs = {
1416"auth_token": auth_token,
4cc985faRandySheriffH2 years ago1417"node_1/model_name": model,
1418"node_1/response_format": response_format,
1419"node_1/file": audio_blob,
d853d31fRandySheriffH2 years ago1420}
1421out = sess.run(None, ort_inputs)[0]
1422```
1423</details>
1424
1425
1426### AzureTextToText
1427
1428<details>
1429<summary>AzureTextToText details</summary>
1430
1431
1432AzureTextToText talks to a GPT model hosted by [Azure openAI service](https://learn.microsoft.com/en-us/azure/ai-services/openai/).
1433
1434
1435#### Attributes
1436
1437***model_uri:string***
1438
1439Endpoint uri, like "https://myname-aoai-test.openai.azure.com/openai/deployments/mydeploy/chat/completions?api-version=2023-05-15'".
1440
1441#### Inputs
1442
1443***auth_token: tensor(string)***
1444
1445An access token comes with Azure openAI subscription.
1446
1447***chat: tensor(string)***
1448
1449A json string in requested [format](https://learn.microsoft.com/en-us/azure/ai-services/openai/chatgpt-quickstart?tabs=command-line&pivots=rest-api).
1450
1451#### Outputs
1452
1453***response_format: tensor(string)***
1454
1455A json string as response.
1456
1457
1458#### Examples
1459
1460
1461```python
1462
1463import os
1464import numpy as np
1465
1466from onnx import *
1467from onnxruntime_extensions import PyOrtFunction, util, get_library_path
1468from onnxruntime import *
1469
1470
4cc985faRandySheriffH2 years ago1471azure_model_uri = os.getenv('URI', '') # read uri from env
1472azure_auth_token = os.getenv('AUTH', '') # read auto token from env
1473
1474
d853d31fRandySheriffH2 years ago1475def create_azure_chat_model():
1476auth_token = helper.make_tensor_value_info('auth_token', TensorProto.STRING, [-1])
1477chat = helper.make_tensor_value_info('chat', TensorProto.STRING, [-1])
1478response = helper.make_tensor_value_info('response', TensorProto.STRING, [-1])
1479
1480invoker = helper.make_node('AzureTextToText', ['auth_token', 'chat'], ['response'],
1481domain='com.microsoft.extensions',
1482name='chat_invoker',
4cc985faRandySheriffH2 years ago1483model_uri=azure_model_uri)
d853d31fRandySheriffH2 years ago1484
1485graph = helper.make_graph([invoker], 'graph', [auth_token, chat], [response])
1486model = helper.make_model(graph,
1487opset_imports=[helper.make_operatorsetid('com.microsoft.extensions', 1)])
1488
1489onnx.save(model, 'azure_chat.onnx')
1490
1491
1492create_azure_chat_model()
1493opt = SessionOptions()
1494opt.register_custom_ops_library(get_library_path())
1495sess = InferenceSession(os.path.join(test_data_dir, "azure_chat.onnx"), opt, providers=["CPUExecutionProvider", "AzureExecutionProvider"])
4cc985faRandySheriffH2 years ago1496auth_token = np.array([azure_auth_token])
d853d31fRandySheriffH2 years ago1497chat = np.array([r'{"messages":[{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Does Azure OpenAI support customer managed keys?"},{"role": "assistant", "content": "Yes, customer managed keys are supported by Azure OpenAI."},{"role": "user", "content": "Do other Azure AI services support this too?"}]}'])
1498ort_inputs = {
1499"auth_token": auth_token,
1500"chat": chat,
1501}
1502out = sess.run(None, ort_inputs)[0]
1503```
1504</details>
1505
1506
1507### AzureTritonInvoker
1508
1509<details>
1510<summary>AzureTritonInvoker details</summary>
1511
1512
1513AzureTritonInvoker talks to [Azure Machine Learning triton services](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-with-triton?view=azureml-api-2&tabs=azure-cli%2Cendpoint).
1514
1515
1516#### Attributes
1517
1518***model_uri:string***
1519
1520Endpoint uri, like "'https://endpoint-12345678.westus.inference.ml.azure.com".
1521
1522***model_name:string***
1523
1524***model_version:string***
1525
1526A version string, like "1", or "2".
1527
1528#### Inputs
1529
1530***auth_token: tensor(string)***
1531
1532An access token comes with Azure Machine Learning model deployment.
1533
1534***inputs: tensor(variadic)***
1535
1536Tensors of any supported onnx data type.
1537
1538#### Outputs
1539
1540***outputs: tensor(variadic)***
1541
1542Tensors of any supported onnx data type.
1543
1544
1545#### Examples
1546
1547
1548```python
1549
1550import os
1551import numpy as np
1552
1553from onnx import *
1554from onnxruntime_extensions import PyOrtFunction, util, get_library_path
1555from onnxruntime import *
1556
1557
4cc985faRandySheriffH2 years ago1558triton_uri = os.getenv('URI', '') # read uri from env
1559triton_auth_token = os.getenv('AUTH', '') # read auto token from env
1560
1561
d853d31fRandySheriffH2 years ago1562def createAddf():
1563auth_token = helper.make_tensor_value_info('auth_token', TensorProto.STRING, [-1])
1564X = helper.make_tensor_value_info('X', TensorProto.FLOAT, [-1])
1565Y = helper.make_tensor_value_info('Y', TensorProto.FLOAT, [-1])
1566Z = helper.make_tensor_value_info('Z', TensorProto.FLOAT, [-1])
1567invoker = helper.make_node('AzureTritonInvoker', ['auth_token', 'X', 'Y'], ['Z'],
1568domain='com.microsoft.extensions', name='triton_invoker',
4cc985faRandySheriffH2 years ago1569model_uri=triton_uri,
d853d31fRandySheriffH2 years ago1570model_name='addf', model_version='1')
1571graph = helper.make_graph([invoker], 'graph', [auth_token, X, Y], [Z])
1572model = helper.make_model(graph,
1573opset_imports=[helper.make_operatorsetid('com.microsoft.extensions', 1)])
1574save(model, 'triton_addf.onnx')
1575
1576
1577def run_add_f():
1578opt = SessionOptions()
1579opt.register_custom_ops_library(get_library_path())
1580sess = InferenceSession(os.path.join(test_data_dir, "triton_addf.onnx"),
1581opt, providers=["CPUExecutionProvider", "AzureExecutionProvider"])
4cc985faRandySheriffH2 years ago1582auth_token = np.array([triton_auth_token])
d853d31fRandySheriffH2 years ago1583x = np.array([1,2,3,4]).astype(np.float32)
1584y = np.array([4,3,2,1]).astype(np.float32)
1585ort_inputs = {
1586"auth_token": auth_token,
1587"X": x,
1588"Y": y
1589}
1590out = sess.run(None, ort_inputs)[0]
1591```
85ffb941Zhipeng Han1 years ago1592</details>