openai/chatkit-python
Publicmirrored fromhttps://github.com/openai/chatkit-pythonAvailable
docs/guides/build-interactive-responses-with-widgets.md
328lines · modecode
| 1 | # Build interactive responses with widgets |
| 2 | |
| 3 | Use widgets to turn assistant responses into rich, interactive UIs. Design widgets visually, hydrate them with data on the server, stream them into the conversation, and wire actions and forms so users can click, edit, and submit without writing long free-text prompts. |
| 4 | |
| 5 | This guide covers: |
| 6 | |
| 7 | - Designing and loading widget templates |
| 8 | - Streaming widgets from `respond` and from tools |
| 9 | - Handling widget actions on the server and client |
| 10 | - Building editable forms with widgets |
| 11 | |
| 12 | ## Design widgets in ChatKit Studio |
| 13 | |
| 14 | Use <https://widgets.chatkit.studio> to visually design cards, lists, forms, charts, and other widget components. Populate the **Data** panel with sample values to preview how the widget renders with real inputs. |
| 15 | |
| 16 | When the layout and bindings look correct, click **Export** to download the generated `.widget` file. Commit this file alongside the server code that builds and renders the widget. |
| 17 | |
| 18 | ## Build widgets with `WidgetTemplate` |
| 19 | |
| 20 | Load the `.widget` file with `WidgetTemplate.from_file` and hydrate it with runtime data. Placeholders inside the `.widget` template (Jinja-style `{{ }}` expressions) are rendered before the widget is streamed. |
| 21 | |
| 22 | ```python |
| 23 | from chatkit.widgets import WidgetTemplate |
| 24 | |
| 25 | message_template = WidgetTemplate.from_file("widgets/channel_message.widget") |
| 26 | |
| 27 | |
| 28 | def build_message_widget(user_name: str, message: str): |
| 29 | # Replace this helper with whatever your integration uses to build widgets. |
| 30 | return message_template.build( |
| 31 | { |
| 32 | "user_name": user_name, |
| 33 | "message": message, |
| 34 | } |
| 35 | ) |
| 36 | ``` |
| 37 | |
| 38 | `WidgetTemplate.build` accepts plain dicts or Pydantic models. Use `.build_basic` if you're working with a `BasicRoot` widget outside of streaming. |
| 39 | |
| 40 | ## Stream widgets from `respond` |
| 41 | |
| 42 | Use `stream_widget` to emit a one-off widget or stream updates from an async generator. |
| 43 | |
| 44 | ```python |
| 45 | from chatkit.server import stream_widget |
| 46 | |
| 47 | |
| 48 | async def respond(...): |
| 49 | user_name = "Harry Potter" |
| 50 | message = "Yer a wizard, Harry" |
| 51 | message_widget = build_message_widget(user_name=user_name, message=message) |
| 52 | async for event in stream_widget( |
| 53 | thread, |
| 54 | message_widget, |
| 55 | copy_text=f"Message to {user_name}: {message}", |
| 56 | generate_id=lambda item_type: self.store.generate_item_id( |
| 57 | item_type, thread, context |
| 58 | ), |
| 59 | ): |
| 60 | yield event |
| 61 | ``` |
| 62 | |
| 63 | To stream gradual updates, yield successive widget states from an async generator; `stream_widget` diffs and emits `ThreadItemUpdatedEvent`s for you. |
| 64 | |
| 65 | ## Stream widgets from tools |
| 66 | |
| 67 | Tools can enqueue widgets via `AgentContext.stream_widget`; `stream_agent_response` forwards them to the client. |
| 68 | |
| 69 | ```python |
| 70 | from agents import RunContextWrapper, function_tool |
| 71 | from chatkit.agents import AgentContext |
| 72 | |
| 73 | |
| 74 | @function_tool(description_override="Display a sample widget to the user.") |
| 75 | async def sample_widget(ctx: RunContextWrapper[AgentContext]): |
| 76 | message_widget = build_message_widget(...) |
| 77 | await ctx.context.stream_widget(message_widget) |
| 78 | ``` |
| 79 | |
| 80 | ## Stream widget updates while text streams |
| 81 | |
| 82 | The examples above return a fully completed static widget. You can also stream an updating widget by yielding new versions of the widget from a generator function. The ChatKit framework will send updates for the parts of the widget that have changed. |
| 83 | |
| 84 | !!! note "Text streaming support" |
| 85 | Currently, only `<Text>` and `<Markdown>` components marked with an `id` have their text updates streamed. Other diffs will forgo the streaming UI and replace and rerender parts of the widget client-side. |
| 86 | |
| 87 | ```python |
| 88 | from typing import AsyncGenerator |
| 89 | |
| 90 | from agents import RunContextWrapper, function_tool |
| 91 | from chatkit.agents import AgentContext, Runner |
| 92 | from chatkit.widgets import WidgetRoot |
| 93 | |
| 94 | |
| 95 | @function_tool |
| 96 | async def draft_message_to_harry(ctx: RunContextWrapper[AgentContext]): |
| 97 | # message_generator is your model/tool function that streams text |
| 98 | message_result = Runner.run_streamed( |
| 99 | message_generator, "Draft a message to Harry." |
| 100 | ) |
| 101 | |
| 102 | async def widget_generator() -> AsyncGenerator[WidgetRoot, None]: |
| 103 | message = "" |
| 104 | async for event in message_result.stream_events(): |
| 105 | if ( |
| 106 | event.type == "raw_response_event" |
| 107 | and event.data.type == "response.output_text.delta" |
| 108 | ): |
| 109 | message += event.data.delta |
| 110 | yield build_message_widget( |
| 111 | user_name="Harry Potter", |
| 112 | message=message, |
| 113 | ) |
| 114 | |
| 115 | # Final render after streaming completes. |
| 116 | yield build_message_widget( |
| 117 | user_name="Harry Potter", |
| 118 | message=message, |
| 119 | ) |
| 120 | |
| 121 | await ctx.context.stream_widget(widget_generator()) |
| 122 | ``` |
| 123 | |
| 124 | The inner generator collects the streamed text events and rebuilds the widget with the latest message so the UI updates incrementally. |
| 125 | |
| 126 | ## Handle widget actions |
| 127 | |
| 128 | Actions let widget interactions trigger server or client logic without posting a chat message. |
| 129 | |
| 130 | ### Define actions in your widget |
| 131 | |
| 132 | Configure actions as part of the widget definition while you design it in <https://widgets.chatkit.studio>. Add an action to any action-capable component such as `Button.onClickAction`; explore supported components on the components page. |
| 133 | |
| 134 | ```jsx |
| 135 | <Button |
| 136 | label="Send message" |
| 137 | onClickAction={{ |
| 138 | type: "send_message", |
| 139 | payload: { text: "Ping support" }, |
| 140 | }} |
| 141 | /> |
| 142 | ``` |
| 143 | |
| 144 | ### Choose client vs server handling |
| 145 | |
| 146 | Actions are handled on the server by default and flow into `ChatKitServer.action`. Set `handler: "client"` in the action to route it to your frontend’s `widgets.onAction` instead. Use the server when you need to update thread state or stream widgets; use the client for immediate UI work or to chain into a follow-up `sendCustomAction` after local logic completes. |
| 147 | |
| 148 | Example widget definition with a client action handler: |
| 149 | |
| 150 | ```jsx |
| 151 | <Button |
| 152 | label="Send message" |
| 153 | onClickAction={{ |
| 154 | type: "send_message", |
| 155 | handler: "client", |
| 156 | payload: { text: "Ping support" }, |
| 157 | }} |
| 158 | /> |
| 159 | ``` |
| 160 | |
| 161 | ### Handle actions on the server |
| 162 | |
| 163 | Implement `ChatKitServer.action` to process incoming actions. The `sender` argument is the widget item that triggered the action (if available). |
| 164 | |
| 165 | ```python |
| 166 | from datetime import datetime |
| 167 | |
| 168 | from chatkit.server import ChatKitServer, stream_widget |
| 169 | from chatkit.types import HiddenContextItem, WidgetItem |
| 170 | |
| 171 | |
| 172 | class MyChatKitServer(ChatKitServer[RequestContext]): |
| 173 | async def action(self, thread, action, sender, context): |
| 174 | if action.type == "send_message": |
| 175 | await send_to_chat(action.payload["text"]) |
| 176 | |
| 177 | # Record the user action so the model can see it on the next turn. |
| 178 | hidden = HiddenContextItem( |
| 179 | id="generated-item-id", |
| 180 | thread_id=thread.id, |
| 181 | created_at=datetime.now(), |
| 182 | content=f"User sent message: {action.payload['text']}", |
| 183 | ) |
| 184 | # HiddenContextItems need to be manually saved because ChatKitServer |
| 185 | # only auto-saves streamed items, and HiddenContextItem should never be streamed to the client. |
| 186 | await self.store.add_thread_item(thread.id, hidden, context) |
| 187 | |
| 188 | # Stream an updated widget back to the client. |
| 189 | updated_widget = build_message_widget(text=action.payload["text"]) |
| 190 | async for event in stream_widget( |
| 191 | thread, |
| 192 | updated_widget, |
| 193 | generate_id=lambda item_type: self.store.generate_item_id( |
| 194 | item_type, thread, context |
| 195 | ), |
| 196 | ): |
| 197 | yield event |
| 198 | ``` |
| 199 | |
| 200 | Treat action payloads as untrusted input from the client. |
| 201 | |
| 202 | ### Handle actions on the client |
| 203 | |
| 204 | Provide [`widgets.onAction`](https://openai.github.io/chatkit-js/api/openai/chatkit/type-aliases/widgetsoption) when creating ChatKit on the client; you can still forward follow-up actions to the server from your `onAction` callback with the `sendCustomAction()` command if needed. |
| 205 | |
| 206 | ```ts |
| 207 | const chatkit = useChatKit({ |
| 208 | // ... |
| 209 | widgets: { |
| 210 | onAction: async (action, widgetItem) => { |
| 211 | if (action.type === "save_profile") { |
| 212 | const result = await saveProfile(action.payload); |
| 213 | |
| 214 | // Optionally invoke a server action after client-side work completes. |
| 215 | await chatkit.sendCustomAction( |
| 216 | { |
| 217 | type: "save_profile_complete", |
| 218 | payload: {...result, user_id: action.payload.user_id}, |
| 219 | }, |
| 220 | widgetItem.id, |
| 221 | ); |
| 222 | } |
| 223 | }, |
| 224 | }, |
| 225 | }); |
| 226 | ``` |
| 227 | |
| 228 | On the server, handle the follow-up action (`save_profile_complete`) in the `action` method to stream refreshed widgets or messages. |
| 229 | |
| 230 | ### Control loading behavior |
| 231 | |
| 232 | Use `loadingBehavior` to control how actions trigger different loading states in a widget. |
| 233 | |
| 234 | ```jsx |
| 235 | <Button |
| 236 | label="Send message" |
| 237 | onClickAction={{ |
| 238 | type: "send_message", |
| 239 | loadingBehavior: "container", |
| 240 | }} |
| 241 | /> |
| 242 | ``` |
| 243 | |
| 244 | | Value | Behavior | |
| 245 | | ----------- | ------------------------------------------------------------------------------------------------------------------------------- | |
| 246 | | `auto` | The action will adapt to how it’s being used. (_default_) | |
| 247 | | `self` | The action triggers loading state on the widget node that the action was bound to. | |
| 248 | | `container` | The action triggers loading state on the entire widget container. This causes the widget to fade out slightly and become inert. | |
| 249 | | `none` | No loading state | |
| 250 | |
| 251 | Generally, we recommend using `auto`, which is the default. `auto` triggers loading states based on where the action is bound, for example: |
| 252 | |
| 253 | - `Button.onClickAction` → `self` |
| 254 | - `Select.onChangeAction` → `none` |
| 255 | - `Card.confirm.action` → `container` |
| 256 | |
| 257 | ## Create custom forms with widgets |
| 258 | |
| 259 | Wrap widgets that collect user input in a `Form` to have their values automatically injected into every action triggered inside that form. The form values arrive in the action payload, keyed by each field’s `name`. |
| 260 | |
| 261 | - `<Select name="title" />` → `action.payload["title"]` |
| 262 | - `<Select name="todo.title" />` → `action.payload["todo"]["title"]` |
| 263 | |
| 264 | ```jsx |
| 265 | <Form |
| 266 | direction="col" |
| 267 | onSubmitAction={{ |
| 268 | type: "update_todo", |
| 269 | payload: { id: todo.id }, |
| 270 | }} |
| 271 | > |
| 272 | <Title value="Edit Todo" /> |
| 273 | <Text value="Title" color="secondary" size="sm" /> |
| 274 | <Text |
| 275 | value={todo.title} |
| 276 | editable={{ name: "title", required: true }} |
| 277 | /> |
| 278 | <Text value="Description" color="secondary" size="sm" /> |
| 279 | <Text |
| 280 | value={todo.description} |
| 281 | editable={{ name: "description" }} |
| 282 | /> |
| 283 | <Button label="Save" submit /> |
| 284 | </Form> |
| 285 | ``` |
| 286 | |
| 287 | On the server, read the form values from the action payload. Any action originating from inside the form will include the latest field values. |
| 288 | |
| 289 | ```python |
| 290 | from collections.abc import AsyncIterator |
| 291 | |
| 292 | from chatkit.server import ChatKitServer |
| 293 | from chatkit.types import Action, ThreadMetadata, ThreadStreamEvent, WidgetItem |
| 294 | |
| 295 | |
| 296 | class MyChatKitServer(ChatKitServer[RequestContext]): |
| 297 | async def action( |
| 298 | self, |
| 299 | thread: ThreadMetadata, |
| 300 | action: Action[str, Any], |
| 301 | sender: WidgetItem | None, |
| 302 | context: RequestContext, |
| 303 | ) -> AsyncIterator[ThreadStreamEvent]: |
| 304 | if action.type == "update_todo": |
| 305 | todo_id = action.payload["id"] |
| 306 | # Any action that originates from within the Form will |
| 307 | # include title and description |
| 308 | title = action.payload["title"] |
| 309 | description = action.payload["description"] |
| 310 | |
| 311 | # ... |
| 312 | ``` |
| 313 | |
| 314 | ### Validation |
| 315 | |
| 316 | `Form` uses basic native form validation; it enforces `required` and `pattern` on configured fields and blocks submission when any field is invalid. |
| 317 | |
| 318 | We may add new validation modes with better UX, more expressive validation, and custom error display. Until then, widgets are not a great medium for complex forms with tricky validation. If you need this, a better pattern is to use client-side action handling to trigger a modal, show a custom form there, then pass the result back into ChatKit with `sendCustomAction`. |
| 319 | |
| 320 | ### Treating `Card` as a `Form` |
| 321 | |
| 322 | You can pass `asForm=True` to `Card` and it will behave as a `Form`, running validation and passing collected fields to the Card’s `confirm` action. |
| 323 | |
| 324 | ### Payload key collisions |
| 325 | |
| 326 | If there is a naming collision with some other existing pre-defined key on your payload, the form value will be ignored. This is probably a bug, so we’ll emit an `error` event when we see this. |
| 327 | |
| 328 | |
| 329 | |