openai/chatkit-python
Publicmirrored fromhttps://github.com/openai/chatkit-pythonAvailable
docs/guides/let-users-pick-tools-and-models.md
138lines · modecode
| 1 | # Let users pick tools and models |
| 2 | |
| 3 | This guide shows how to expose a tool menu and model picker in the composer UI, read the user’s choices as inference options on the server, and fork your inference pipeline based on those choices. |
| 4 | |
| 5 | At a high level: |
| 6 | |
| 7 | - `composer.tools` controls which tools appear in the composer tool menu (the plus button). |
| 8 | - `composer.models` controls which models appear in the model picker in the composer below the text input. |
| 9 | - The selected tool and model arrive as `inference_options` on the `UserMessageItem` in your `respond` method. |
| 10 | |
| 11 | ## Configure tools in the composer |
| 12 | |
| 13 | Configure the tools that should appear in the composer tool menu when you initialize ChatKit on the client: |
| 14 | |
| 15 | ```ts |
| 16 | const chatkit = useChatKit({ |
| 17 | // ... |
| 18 | composer: { |
| 19 | tools: [ |
| 20 | { |
| 21 | id: "summarize", |
| 22 | icon: "book-open", |
| 23 | label: "Summarize", |
| 24 | placeholderOverride: "Summarize the current page or document.", |
| 25 | }, |
| 26 | { |
| 27 | id: "search_tickets", |
| 28 | icon: "search", |
| 29 | label: "Search tickets", |
| 30 | shortLabel: "Search", |
| 31 | placeholderOverride: "Search support tickets for similar issues.", |
| 32 | }, |
| 33 | ], |
| 34 | }, |
| 35 | }); |
| 36 | ``` |
| 37 | |
| 38 | Each entry defines a user-facing label/shortLabel and a stable `id` you’ll use on the server to decide how to handle the turn. |
| 39 | |
| 40 | ## Configure the model picker in the composer |
| 41 | |
| 42 | Expose a small set of model choices so users can trade off speed vs quality. |
| 43 | |
| 44 | ```ts |
| 45 | const chatkit = useChatKit({ |
| 46 | // ... |
| 47 | composer: { |
| 48 | models: [ |
| 49 | { |
| 50 | id: "gpt-4.1-mini", |
| 51 | label: "Fast", |
| 52 | description: "Answers right away", |
| 53 | }, |
| 54 | { |
| 55 | id: "gpt-4.1", |
| 56 | label: "Quality", |
| 57 | description: "All rounder" |
| 58 | default: true, |
| 59 | }, |
| 60 | ], |
| 61 | }, |
| 62 | }); |
| 63 | ``` |
| 64 | |
| 65 | The selected model id flows through to your server so you can route requests to the right underlying model or configuration. |
| 66 | |
| 67 | ## Read tool and model choices on the server |
| 68 | |
| 69 | On the server, `UserMessageItem.inference_options` carries the tool choice and model id for that turn. |
| 70 | |
| 71 | ```python |
| 72 | from chatkit.types import InferenceOptions |
| 73 | |
| 74 | |
| 75 | class MyChatKitServer(ChatKitServer[RequestContext]): |
| 76 | async def respond( |
| 77 | self, |
| 78 | thread: ThreadMetadata, |
| 79 | input_user_message: UserMessageItem | None, |
| 80 | context: RequestContext, |
| 81 | ) -> AsyncIterator[ThreadStreamEvent]: |
| 82 | options = input_user_message and input_user_message.inference_options |
| 83 | |
| 84 | model = options.model if options and options.model else "gpt-4.1-mini" |
| 85 | tool_choice = options.tool_choice.id if options and options.tool_choice else None |
| 86 | |
| 87 | # Use `model` and `tool_choice` when building your model request... |
| 88 | ``` |
| 89 | |
| 90 | If the user doesn’t pick anything explicit, `inference_options` may be `None` or have `model` / `tool_choice` unset; fall back to your defaults. |
| 91 | |
| 92 | ## Fork your inference pipeline based on user choices |
| 93 | |
| 94 | Use the tool and model choices to branch into different agents, prompts, or tools. |
| 95 | |
| 96 | ### Route to different tools or agents |
| 97 | |
| 98 | For example, use the composer’s tool id to decide which agent (or tool set) to run: |
| 99 | |
| 100 | ```python |
| 101 | if tool_choice == "summarize": |
| 102 | agent = summarization_agent |
| 103 | elif tool_choice == "search_tickets": |
| 104 | agent = ticket_search_agent |
| 105 | else: |
| 106 | agent = default_agent |
| 107 | |
| 108 | result = Runner.run_streamed(agent, input_items, context=agent_context) |
| 109 | ``` |
| 110 | |
| 111 | You control which tools each agent exposes; the composer’s tool menu just lets the user express intent up front instead of relying purely on model heuristics. |
| 112 | |
| 113 | ### Choose models per turn |
| 114 | |
| 115 | Use the selected model id to pick an underlying model or configuration when you call the OpenAI Responses API (or another provider): |
| 116 | |
| 117 | ```python |
| 118 | model = inference.model if inference and inference.model else "gpt-4.1-mini" |
| 119 | |
| 120 | response = await client.responses.create( |
| 121 | model=model, |
| 122 | input=..., |
| 123 | # other options |
| 124 | ) |
| 125 | ``` |
| 126 | |
| 127 | You can also use the model choice as a coarse “mode” flag—for example, always enabling safer or more verbose prompting on certain models. |
| 128 | |
| 129 | ### Combine tools and models |
| 130 | |
| 131 | Nothing stops you from combining both choices. A common pattern is: |
| 132 | |
| 133 | - Use the composer tool menu to decide **what kind of work** to do (summarization, search, drafting, etc.). |
| 134 | - Use the model picker to decide **how heavy** the model pass should be (fast vs quality, cheap vs expensive). |
| 135 | |
| 136 | This keeps the chat UI simple while still giving advanced users control over how their requests are handled end to end. |
| 137 | |
| 138 | |
| 139 | |