microsoft/openvmm
Publicmirrored fromhttps://github.com/microsoft/openvmmAvailable
Guide/src/dev_guide/dev_tools/flowey/artifacts.md
70lines · modecode
| 1 | # Artifacts |
| 2 | |
| 3 | Artifacts enable typed data transfer between jobs with automatic dependency management, abstracting away CI system complexities like name collisions and manual job ordering. |
| 4 | |
| 5 | ## Typed vs Untyped Artifacts |
| 6 | |
| 7 | **Typed artifacts (recommended)** provide type-safe artifact handling by defining |
| 8 | a custom type that implements the `Artifact` trait: |
| 9 | |
| 10 | ```rust,ignore |
| 11 | #[derive(Serialize, Deserialize)] |
| 12 | struct MyArtifact { |
| 13 | #[serde(rename = "output.bin")] |
| 14 | binary: PathBuf, |
| 15 | #[serde(rename = "metadata.json")] |
| 16 | metadata: PathBuf, |
| 17 | } |
| 18 | |
| 19 | impl Artifact for MyArtifact {} |
| 20 | |
| 21 | let (pub_artifact, use_artifact) = pipeline.new_typed_artifact("my-files"); |
| 22 | ``` |
| 23 | |
| 24 | **Untyped artifacts** provide simple directory-based artifacts for simpler cases: |
| 25 | |
| 26 | ```rust,ignore |
| 27 | let (pub_artifact, use_artifact) = pipeline.new_artifact("my-files"); |
| 28 | ``` |
| 29 | |
| 30 | For detailed examples of defining and using artifacts, see the [Artifact trait documentation](https://openvmm.dev/rustdoc/linux/flowey_core/pipeline/trait.Artifact.html). |
| 31 | |
| 32 | Both `pipeline.new_typed_artifact("name")` and `pipeline.new_artifact("name")` return a tuple of handles: `(pub_artifact, use_artifact)`. When defining a job you convert them with the job context: |
| 33 | |
| 34 | ```rust,ignore |
| 35 | // In a producing job: |
| 36 | let artifact_out = ctx.publish_artifact(pub_artifact); |
| 37 | // artifact_out : WriteVar<MyArtifact> (typed) |
| 38 | // or WriteVar<PathBuf> for untyped |
| 39 | |
| 40 | // In a consuming job: |
| 41 | let artifact_in = ctx.use_artifact(use_artifact); |
| 42 | // artifact_in : ReadVar<MyArtifact> (typed) |
| 43 | // or ReadVar<PathBuf> for untyped |
| 44 | ``` |
| 45 | |
| 46 | After conversion, you treat the returned `WriteVar` / `ReadVar` like any other flowey variable (claim them in steps, write/read values). |
| 47 | Key concepts: |
| 48 | |
| 49 | - The `Artifact` trait works by serializing your type to JSON in a format that reflects a directory structure |
| 50 | - Use `#[serde(rename = "file.exe")]` to specify exact file names |
| 51 | - Typed artifacts ensure compile-time type safety when passing data between jobs |
| 52 | - Untyped artifacts are simpler but don't provide type guarantees |
| 53 | - Tuple handles must be lifted with `ctx.publish_artifact(...)` / `ctx.use_artifact(...)` to become flowey variables |
| 54 | |
| 55 | ## How Flowey Manages Artifacts Under the Hood |
| 56 | |
| 57 | During the **pipeline resolution phase** (build-time), flowey: |
| 58 | |
| 59 | 1. **Identifies artifact producers and consumers** by analyzing which jobs write to vs read from each artifact's `WriteVar`/`ReadVar` |
| 60 | 2. **Constructs the job dependency graph** ensuring producers run before consumers |
| 61 | 3. **Generates backend-specific upload/download steps** in the appropriate places: |
| 62 | - For ADO: Uses `PublishPipelineArtifact` and `DownloadPipelineArtifact` tasks |
| 63 | - For GitHub Actions: Uses `actions/upload-artifact` and `actions/download-artifact` |
| 64 | - For local execution: Uses filesystem copying |
| 65 | |
| 66 | At **runtime**, the artifact `ReadVar<PathBuf>` and `WriteVar<PathBuf>` work just like any other flowey variable: |
| 67 | |
| 68 | - Producing jobs write artifact files to the path from `WriteVar<PathBuf>` |
| 69 | - Flowey automatically uploads those files as an artifact |
| 70 | - Consuming jobs read the path from `ReadVar<PathBuf>` where flowey has downloaded the artifact |
| 71 | |