microsoft/openvmm
Publicmirrored from https://github.com/microsoft/openvmmAvailable
Guide/src/dev_guide/dev_tools/flowey/artifacts.md
70lines · modeblame
1b651d52Daman Mulye8 months ago | 1 | # Artifacts |
| 2 | | |
| 3 | Artifacts enable typed data transfer between jobs with automatic dependency management, abstracting away CI system complexities like name collisions and manual job ordering. | |
| 4 | | |
| 5 | ## Typed vs Untyped Artifacts | |
| 6 | | |
| 7 | **Typed artifacts (recommended)** provide type-safe artifact handling by defining | |
| 8 | a custom type that implements the `Artifact` trait: | |
| 9 | | |
| 10 | ```rust | |
| 11 | #[derive(Serialize, Deserialize)] | |
| 12 | struct MyArtifact { | |
| 13 | #[serde(rename = "output.bin")] | |
| 14 | binary: PathBuf, | |
| 15 | #[serde(rename = "metadata.json")] | |
| 16 | metadata: PathBuf, | |
| 17 | } | |
| 18 | | |
| 19 | impl Artifact for MyArtifact {} | |
| 20 | | |
| 21 | let (pub_artifact, use_artifact) = pipeline.new_typed_artifact("my-files"); | |
| 22 | ``` | |
| 23 | | |
| 24 | **Untyped artifacts** provide simple directory-based artifacts for simpler cases: | |
| 25 | | |
| 26 | ```rust | |
| 27 | let (pub_artifact, use_artifact) = pipeline.new_artifact("my-files"); | |
| 28 | ``` | |
| 29 | | |
| 30 | For detailed examples of defining and using artifacts, see the [Artifact trait documentation](https://openvmm.dev/rustdoc/linux/flowey_core/pipeline/trait.Artifact.html). | |
| 31 | | |
| 32 | Both `pipeline.new_typed_artifact("name")` and `pipeline.new_artifact("name")` return a tuple of handles: `(pub_artifact, use_artifact)`. When defining a job you convert them with the job context: | |
| 33 | | |
| 34 | ```rust | |
| 35 | // In a producing job: | |
| 36 | let artifact_out = ctx.publish_artifact(pub_artifact); | |
| 37 | // artifact_out : WriteVar<MyArtifact> (typed) | |
| 38 | // or WriteVar<PathBuf> for untyped | |
| 39 | | |
| 40 | // In a consuming job: | |
| 41 | let artifact_in = ctx.use_artifact(use_artifact); | |
| 42 | // artifact_in : ReadVar<MyArtifact> (typed) | |
| 43 | // or ReadVar<PathBuf> for untyped | |
| 44 | ``` | |
| 45 | | |
| 46 | After conversion, you treat the returned `WriteVar` / `ReadVar` like any other flowey variable (claim them in steps, write/read values). | |
| 47 | Key concepts: | |
| 48 | | |
| 49 | - The `Artifact` trait works by serializing your type to JSON in a format that reflects a directory structure | |
| 50 | - Use `#[serde(rename = "file.exe")]` to specify exact file names | |
| 51 | - Typed artifacts ensure compile-time type safety when passing data between jobs | |
| 52 | - Untyped artifacts are simpler but don't provide type guarantees | |
| 53 | - Tuple handles must be lifted with `ctx.publish_artifact(...)` / `ctx.use_artifact(...)` to become flowey variables | |
| 54 | | |
| 55 | ## How Flowey Manages Artifacts Under the Hood | |
| 56 | | |
| 57 | During the **pipeline resolution phase** (build-time), flowey: | |
| 58 | | |
| 59 | 1. **Identifies artifact producers and consumers** by analyzing which jobs write to vs read from each artifact's `WriteVar`/`ReadVar` | |
| 60 | 2. **Constructs the job dependency graph** ensuring producers run before consumers | |
| 61 | 3. **Generates backend-specific upload/download steps** in the appropriate places: | |
| 62 | - For ADO: Uses `PublishPipelineArtifact` and `DownloadPipelineArtifact` tasks | |
| 63 | - For GitHub Actions: Uses `actions/upload-artifact` and `actions/download-artifact` | |
| 64 | - For local execution: Uses filesystem copying | |
| 65 | | |
| 66 | At **runtime**, the artifact `ReadVar<PathBuf>` and `WriteVar<PathBuf>` work just like any other flowey variable: | |
| 67 | | |
| 68 | - Producing jobs write artifact files to the path from `WriteVar<PathBuf>` | |
| 69 | - Flowey automatically uploads those files as an artifact | |
| 70 | - Consuming jobs read the path from `ReadVar<PathBuf>` where flowey has downloaded the artifact |