microsoft/onnxruntime-extensions
Publicmirrored fromhttps://github.com/microsoft/onnxruntime-extensionsAvailable
tutorials/pytorch_custom_ops_tutorial.ipynb
219lines · modecode
| 1 | { |
| 2 | "cells": [ |
| 3 | { |
| 4 | "cell_type": "markdown", |
| 5 | "metadata": {}, |
| 6 | "source": [ |
| 7 | "# Convert And Inference Pytorch model with CustomOps\n", |
| 8 | "\n", |
| 9 | "With onnxruntime_extensions package, the Pytorch model with the operation cannot be converted into the standard ONNX operators still be converted and the converted ONNX model still can be run with ONNXRuntime, plus onnxruntime_extensions package. This tutorial show it works" |
| 10 | ] |
| 11 | }, |
| 12 | { |
| 13 | "cell_type": "markdown", |
| 14 | "metadata": {}, |
| 15 | "source": [ |
| 16 | "## Converting\n", |
| 17 | "Suppose there is a model which cannot be converted because there is no matrix inverse operation in ONNX standard opset. And the model will be defined like the following." |
| 18 | ] |
| 19 | }, |
| 20 | { |
| 21 | "cell_type": "code", |
| 22 | "execution_count": 1, |
| 23 | "metadata": {}, |
| 24 | "outputs": [], |
| 25 | "source": [ |
| 26 | "import torch\n", |
| 27 | "import torchvision\n", |
| 28 | "\n", |
| 29 | "class CustomInverse(torch.nn.Module):\n", |
| 30 | " def forward(self, x):\n", |
| 31 | " return torch.inverse(x) + x" |
| 32 | ] |
| 33 | }, |
| 34 | { |
| 35 | "cell_type": "markdown", |
| 36 | "metadata": {}, |
| 37 | "source": [ |
| 38 | "To export this model into ONNX format, we need register a custom op handler for pytorch.onn.exporter." |
| 39 | ] |
| 40 | }, |
| 41 | { |
| 42 | "cell_type": "code", |
| 43 | "execution_count": 2, |
| 44 | "metadata": {}, |
| 45 | "outputs": [], |
| 46 | "source": [ |
| 47 | "from torch.onnx import register_custom_op_symbolic\n", |
| 48 | "\n", |
| 49 | "\n", |
| 50 | "def my_inverse(g, self):\n", |
| 51 | " return g.op(\"ai.onnx.contrib::Inverse\", self)\n", |
| 52 | "\n", |
| 53 | "register_custom_op_symbolic('::inverse', my_inverse, 1)" |
| 54 | ] |
| 55 | }, |
| 56 | { |
| 57 | "cell_type": "markdown", |
| 58 | "metadata": {}, |
| 59 | "source": [ |
| 60 | "Then, invoke the exporter" |
| 61 | ] |
| 62 | }, |
| 63 | { |
| 64 | "cell_type": "code", |
| 65 | "execution_count": 3, |
| 66 | "metadata": { |
| 67 | "scrolled": false |
| 68 | }, |
| 69 | "outputs": [], |
| 70 | "source": [ |
| 71 | "import io\n", |
| 72 | "import onnx\n", |
| 73 | "\n", |
| 74 | "x0 = torch.randn(3, 3)\n", |
| 75 | "# Export model to ONNX\n", |
| 76 | "f = io.BytesIO()\n", |
| 77 | "t_model = CustomInverse()\n", |
| 78 | "torch.onnx.export(t_model, (x0, ), f, opset_version=12)\n", |
| 79 | "onnx_model = onnx.load(io.BytesIO(f.getvalue()))" |
| 80 | ] |
| 81 | }, |
| 82 | { |
| 83 | "cell_type": "markdown", |
| 84 | "metadata": {}, |
| 85 | "source": [ |
| 86 | "Now, we got a ONNX model in the memory, and it can be save into a disk file by 'onnx.save_model(onnx_model, <file_path>)" |
| 87 | ] |
| 88 | }, |
| 89 | { |
| 90 | "cell_type": "markdown", |
| 91 | "metadata": {}, |
| 92 | "source": [ |
| 93 | "## Inference\n", |
| 94 | "This converted model cannot directly run the onnxruntime due to the custom operator. but it can run with onnxruntime_extensions easily.\n", |
| 95 | "\n", |
| 96 | "Firstly, let define a PyOp function to inteprete the custom op node in the ONNNX model." |
| 97 | ] |
| 98 | }, |
| 99 | { |
| 100 | "cell_type": "code", |
| 101 | "execution_count": 4, |
| 102 | "metadata": {}, |
| 103 | "outputs": [], |
| 104 | "source": [ |
| 105 | "import numpy\n", |
| 106 | "from onnxruntime_extensions import onnx_op, PyOp\n", |
| 107 | "@onnx_op(op_type=\"Inverse\")\n", |
| 108 | "def inverse(x):\n", |
| 109 | " # the user custom op implementation here:\n", |
| 110 | " return numpy.linalg.inv(x)\n" |
| 111 | ] |
| 112 | }, |
| 113 | { |
| 114 | "cell_type": "markdown", |
| 115 | "metadata": {}, |
| 116 | "source": [ |
| 117 | "* **ONNX Inference**" |
| 118 | ] |
| 119 | }, |
| 120 | { |
| 121 | "cell_type": "code", |
| 122 | "execution_count": 5, |
| 123 | "metadata": {}, |
| 124 | "outputs": [ |
| 125 | { |
| 126 | "name": "stdout", |
| 127 | "output_type": "stream", |
| 128 | "text": [ |
| 129 | "[[-3.081008 0.20269153 0.42009977]\n", |
| 130 | " [-3.3962293 2.5986686 2.4447646 ]\n", |
| 131 | " [ 0.7805753 -0.20394287 -2.7528977 ]]\n" |
| 132 | ] |
| 133 | } |
| 134 | ], |
| 135 | "source": [ |
| 136 | "from onnxruntime_extensions import PyOrtFunction\n", |
| 137 | "onnx_fn = PyOrtFunction.from_model(onnx_model)\n", |
| 138 | "y = onnx_fn(x0.numpy())\n", |
| 139 | "print(y)" |
| 140 | ] |
| 141 | }, |
| 142 | { |
| 143 | "cell_type": "markdown", |
| 144 | "metadata": {}, |
| 145 | "source": [ |
| 146 | "* **Compare the result with Pytorch**" |
| 147 | ] |
| 148 | }, |
| 149 | { |
| 150 | "cell_type": "code", |
| 151 | "execution_count": 7, |
| 152 | "metadata": {}, |
| 153 | "outputs": [], |
| 154 | "source": [ |
| 155 | "t_y = t_model(x0)\n", |
| 156 | "numpy.testing.assert_almost_equal(t_y, y, decimal=5)" |
| 157 | ] |
| 158 | }, |
| 159 | { |
| 160 | "cell_type": "markdown", |
| 161 | "metadata": {}, |
| 162 | "source": [ |
| 163 | "## Implement the customop in C++ (optional)\n", |
| 164 | "To make the ONNX model with the CustomOp runn on all other language supported by ONNX Runtime and be independdent of Python, a C++ implmentation is needed, check here for the [inverse.hpp](../operators/math/inverse.hpp) for an example on how to do that." |
| 165 | ] |
| 166 | }, |
| 167 | { |
| 168 | "cell_type": "code", |
| 169 | "execution_count": 8, |
| 170 | "metadata": {}, |
| 171 | "outputs": [ |
| 172 | { |
| 173 | "name": "stdout", |
| 174 | "output_type": "stream", |
| 175 | "text": [ |
| 176 | "[[-3.081008 0.20269153 0.42009977]\n", |
| 177 | " [-3.3962293 2.5986686 2.4447646 ]\n", |
| 178 | " [ 0.7805753 -0.20394287 -2.7528977 ]]\n" |
| 179 | ] |
| 180 | } |
| 181 | ], |
| 182 | "source": [ |
| 183 | "from onnxruntime_extensions import enable_custom_op\n", |
| 184 | "# disable the PyOp function and run with the C++ function\n", |
| 185 | "enable_custom_op(False)\n", |
| 186 | "y = onnx_fn(x0.numpy())\n", |
| 187 | "print(y)" |
| 188 | ] |
| 189 | }, |
| 190 | { |
| 191 | "cell_type": "code", |
| 192 | "execution_count": null, |
| 193 | "metadata": {}, |
| 194 | "outputs": [], |
| 195 | "source": [] |
| 196 | } |
| 197 | ], |
| 198 | "metadata": { |
| 199 | "kernelspec": { |
| 200 | "display_name": "Python 3", |
| 201 | "language": "python", |
| 202 | "name": "python3" |
| 203 | }, |
| 204 | "language_info": { |
| 205 | "codemirror_mode": { |
| 206 | "name": "ipython", |
| 207 | "version": 3 |
| 208 | }, |
| 209 | "file_extension": ".py", |
| 210 | "mimetype": "text/x-python", |
| 211 | "name": "python", |
| 212 | "nbconvert_exporter": "python", |
| 213 | "pygments_lexer": "ipython3", |
| 214 | "version": "3.8.5" |
| 215 | } |
| 216 | }, |
| 217 | "nbformat": 4, |
| 218 | "nbformat_minor": 2 |
| 219 | } |