cloudflare/pint
Publicmirrored fromhttps://github.com/cloudflare/pintAvailable
docs/index.md
419lines · modecode
| 1 | --- |
| 2 | layout: default |
| 3 | title: Documentation |
| 4 | nav_order: 1 |
| 5 | has_children: true |
| 6 | --- |
| 7 | |
| 8 | # pint |
| 9 | |
| 10 | pint is a Prometheus rule linter/validator. |
| 11 | |
| 12 | ## Requirements |
| 13 | |
| 14 | pint will run checks on Prometheus alerting & recording rules to detect potential problems |
| 15 | with those rules. |
| 16 | Some checks rely only on the rule itself and can be run "offline" - without talking to any |
| 17 | Prometheus server. |
| 18 | You can run pint in "offline" if you: |
| 19 | |
| 20 | - Don't pass any configuration file to pint. |
| 21 | - You pass configuration file to pint that **doesn't** contain any `prometheus` definition. |
| 22 | - You pass `--offline` flag to `pint` command. |
| 23 | |
| 24 | Most checks included in pint will require sending queries to a running Prometheus server where |
| 25 | those rules are, or would be, deployed. |
| 26 | Those checks are enabled if you pass a configuration file to pint that includes at least one |
| 27 | `prometheus` block. |
| 28 | Checks might use various Prometheus |
| 29 | [HTTP API endpoints](https://prometheus.io/docs/prometheus/latest/querying/api/) to retrieve |
| 30 | extra information, for example Prometheus configuration or metrics metadata. |
| 31 | If you run pint against a different service, like [Thanos](https://thanos.io/) some checks |
| 32 | might return problems due to API call errors, since not all Prometheus HTTP APIs are supported by it. |
| 33 | In that case, you might want to disable failing checks in the pint configuration file. |
| 34 | |
| 35 | **IMPORTANT** `pint` is a tool we wrote to work with our **Prometheus** deployment. It's not intended to be |
| 36 | used with other services that offer partial compatibility with Prometheus, there are **NO PLANS** |
| 37 | to add support for any other services. The only reason we would add support for other systems is if |
| 38 | we started to use them ourselves. |
| 39 | |
| 40 | ## Usage |
| 41 | |
| 42 | There are three modes it works in: |
| 43 | |
| 44 | - CI PR linting |
| 45 | - Ad-hoc linting of a selected files or directories |
| 46 | - A daemon that continuously checks selected files or directories and expose metrics describing |
| 47 | all discovered problems. |
| 48 | |
| 49 | ### Pull Requests |
| 50 | |
| 51 | Run it with `pint ci`. Git is currently the only supported VCS. |
| 52 | |
| 53 | When `pint ci` runs it will find all files in the current working directory and try to parse |
| 54 | them as Prometheus rules. Then it will look for all commits on the current branch that are not |
| 55 | present in the parent branch and to decide which rules were modified. |
| 56 | Checks are run only on modified rules but they require the full list of all rules to find any |
| 57 | cross-rule dependencies. |
| 58 | |
| 59 | Running `pint ci` doesn't require any configuration but it's recommended to add a pint config file |
| 60 | with `ci` section containing at least the `include` option. This will ensure that pint validates |
| 61 | only Prometheus rules and ignores other files. |
| 62 | |
| 63 | Results can optionally be reported as comments on a pull request when using one of supported platforms: |
| 64 | |
| 65 | - [BitBucket API](https://developer.atlassian.com/server/bitbucket/rest/) |
| 66 | - [GitHub API](https://docs.github.com/en/rest) |
| 67 | - [GitLab API](https://docs.gitlab.com/ee/api/rest/) |
| 68 | |
| 69 | Exit code will be one (1) if any issues were detected with severity `Bug` or higher. This permits running |
| 70 | `pint` in your CI system whilst at the same you will get detailed reports on your source control system. |
| 71 | |
| 72 | If any commit on the PR contains `[skip ci]` or `[no ci]` somewhere in the commit message then pint will |
| 73 | skip running all checks. |
| 74 | |
| 75 | #### GitHub Actions |
| 76 | |
| 77 | The easiest way of using `pint` with GitHub Actions is by using |
| 78 | [prymitive/pint-action](https://github.com/prymitive/pint-action). |
| 79 | Here's an example workflow: |
| 80 | |
| 81 | {% raw %} |
| 82 | |
| 83 | ```yaml |
| 84 | name: pint |
| 85 | |
| 86 | on: |
| 87 | push: |
| 88 | branches: |
| 89 | - main |
| 90 | pull_request: |
| 91 | branches: |
| 92 | - main |
| 93 | |
| 94 | jobs: |
| 95 | pint: |
| 96 | runs-on: ubuntu-latest |
| 97 | steps: |
| 98 | - uses: actions/checkout@v4 |
| 99 | with: |
| 100 | fetch-depth: 0 |
| 101 | |
| 102 | - name: Run pint |
| 103 | uses: prymitive/pint-action@v1 |
| 104 | with: |
| 105 | token: ${{ github.token }} |
| 106 | # directory containing Prometheus rules |
| 107 | workdir: 'rules' |
| 108 | ``` |
| 109 | |
| 110 | {% endraw %} |
| 111 | |
| 112 | To customise pint checks create a `.pint.hcl` file in the root of your repository. |
| 113 | See [Configuration](configuration.md) for a description of all options. |
| 114 | |
| 115 | When pint runs checks after a push to a branch (for example after a merge), then |
| 116 | it will pass `workdir` option to `pint lint`, which means that all files inside |
| 117 | `rules` directory will be checked. |
| 118 | |
| 119 | ### Ad-hoc |
| 120 | |
| 121 | Check specified files and report any found issue. |
| 122 | You can pass directory paths and use [glob](https://pkg.go.dev/path/filepath#Match) |
| 123 | patterns as arguments to select files for checking. |
| 124 | |
| 125 | You can lint selected files: |
| 126 | |
| 127 | ```shell |
| 128 | pint lint rules.yml |
| 129 | ``` |
| 130 | |
| 131 | or directories: |
| 132 | |
| 133 | ```shell |
| 134 | pint lint path/to/dir |
| 135 | ``` |
| 136 | |
| 137 | or both: |
| 138 | |
| 139 | ```shell |
| 140 | pint lint path/to/dir file.yml path/file.yml path/dir |
| 141 | ``` |
| 142 | |
| 143 | Using glob patterns: |
| 144 | |
| 145 | ```shell |
| 146 | pint lint path/*.yml path/*.yaml |
| 147 | ``` |
| 148 | |
| 149 | ### Watch mode |
| 150 | |
| 151 | Run pint as a daemon in watch mode where it continuously checks |
| 152 | all rules found in selected files and exposes metrics about |
| 153 | found problems. |
| 154 | |
| 155 | By default it will start a HTTP server on port `8080` and run all checks every |
| 156 | 10 minutes. This can be customised by passing extra flags to the `watch` command. |
| 157 | Run `pint watch -h` to see all available flags. |
| 158 | |
| 159 | Currently supported HTTP paths: |
| 160 | |
| 161 | - `/health` - static endpoint for liveness probes. |
| 162 | - `/metrics` - returns Prometheus metrics, see below. |
| 163 | |
| 164 | #### Manually selecting files and directories |
| 165 | |
| 166 | You can tell it to continuously test specific files or directories: |
| 167 | |
| 168 | ```shell |
| 169 | pint watch glob $GLOB_1 $GLOB_2 ... $GLOB_N |
| 170 | ``` |
| 171 | |
| 172 | Example: |
| 173 | |
| 174 | ```shell |
| 175 | pint watch glob /etc/prometheus/rules-*.yml /etc/prometheus/rules.d |
| 176 | ``` |
| 177 | |
| 178 | If provide a config file for pint with some Prometheus server definitions |
| 179 | then pint will also run "online" checks for it to, for example, ensure all |
| 180 | time series used inside your alerting rules are still present. |
| 181 | Example config: |
| 182 | |
| 183 | ```js |
| 184 | prometheus "local" { |
| 185 | uri = "http://localhost:9090" |
| 186 | } |
| 187 | ``` |
| 188 | |
| 189 | #### Getting list of files to check from Prometheus |
| 190 | |
| 191 | You can also point pint directly at a Prometheus server from the config file. |
| 192 | On every iteration, before starting any checks, pint will query Prometheus API |
| 193 | to get the current value of `rule_files` Prometheus config option and then run |
| 194 | checks on all matching files. |
| 195 | This way if you test your rules against a running Prometheus instance then you don't |
| 196 | need to manually specify any paths or directories. |
| 197 | |
| 198 | Usage: |
| 199 | |
| 200 | ```shell |
| 201 | pint watch rule_files $prometheus |
| 202 | ``` |
| 203 | |
| 204 | Where `$prometheus` is the name of `prometheus` configuration block from pint |
| 205 | config file. |
| 206 | |
| 207 | Example: |
| 208 | |
| 209 | ```shell |
| 210 | pint watch rule_files local |
| 211 | ``` |
| 212 | |
| 213 | #### Accessing watch mode metrics |
| 214 | |
| 215 | Query `/metrics` HTTP endpoint to see all expose metrics, example with default flags: |
| 216 | |
| 217 | ```shell |
| 218 | curl -s http://localhost:8080/metrics |
| 219 | ``` |
| 220 | |
| 221 | Or setup Prometheus scrape job: |
| 222 | |
| 223 | ```yaml |
| 224 | scrape_configs: |
| 225 | - job_name: pint |
| 226 | static_configs: |
| 227 | - targets: ['localhost:8080'] |
| 228 | ``` |
| 229 | |
| 230 | Available metrics: |
| 231 | |
| 232 | - `pint_problem` - exported for every problem detected by pint. |
| 233 | To avoid exposing too many metrics at once pass `--max-problems` flag to watch command. |
| 234 | When this flag is set, pint will expose only up to `--max-problems` value number of |
| 235 | `pint_problem` metrics. |
| 236 | - `pint_problems` - this metric is the total number of all problems detected by pint, |
| 237 | including those not exported due to the `--max-problems` flag. |
| 238 | |
| 239 | The `pint problem` metric can include the `owner` label for each rule. This is useful |
| 240 | to route alerts based on metrics to the right team. |
| 241 | To set a rule owner add a `# pint file/owner $owner` comment in a file, to set |
| 242 | an owner for all rules in that file. You can also set an owner per rule, by adding |
| 243 | `# pint rule/owner $owner` comment around given rule. |
| 244 | |
| 245 | Example: |
| 246 | |
| 247 | ```yaml |
| 248 | # pint file/owner bob |
| 249 | |
| 250 | - alert: ... |
| 251 | expr: ... |
| 252 | |
| 253 | # pint rule/owner alice |
| 254 | - alert: ... |
| 255 | expr: ... |
| 256 | ``` |
| 257 | |
| 258 | Here's an example alert you can use for problems detected by pint: |
| 259 | |
| 260 | {% raw %} |
| 261 | |
| 262 | ```yaml |
| 263 | - alert: Pint Problem Detected |
| 264 | # pint_problem is only present if pint detects any problems |
| 265 | # pint disable promql/series(pint_problem) |
| 266 | expr: | |
| 267 | sum without(instance, problem) (pint_problem) > 0 |
| 268 | for: 1h |
| 269 | annotations: |
| 270 | summary: | |
| 271 | {{ with printf "pint_problem{filename='%s', name='%s', reporter='%s'}" .Labels.filename .Labels.name .Labels.reporter | query }} |
| 272 | {{ . | first | label "problem" }} |
| 273 | {{ end }} |
| 274 | docs: "https://cloudflare.github.io/pint/checks/{{ $labels.reporter }}.html" |
| 275 | ``` |
| 276 | |
| 277 | {% endraw %} |
| 278 | |
| 279 | ## YAML parser |
| 280 | |
| 281 | By default pint will expect all Prometheus rule files to be following the exact |
| 282 | syntax Prometheus expects for YAML files containing [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) |
| 283 | and [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) |
| 284 | rules. |
| 285 | If you have Prometheus rules stored in YAML files with different YAML tree, but still |
| 286 | retain the same set of fields, for example: |
| 287 | |
| 288 | ```yaml |
| 289 | # Flat rule list |
| 290 | - alert: AlertName |
| 291 | expr: up == 0 |
| 292 | - record: sum:up |
| 293 | expr: count(up == 1) |
| 294 | ``` |
| 295 | |
| 296 | ```yaml |
| 297 | # Rules nested under custom tree |
| 298 | service: |
| 299 | prometheus: |
| 300 | rules: |
| 301 | - alert: AlertName |
| 302 | expr: up == 0 |
| 303 | - record: sum:up |
| 304 | expr: count(up == 1) |
| 305 | ``` |
| 306 | |
| 307 | You can still check these rules using pint, but you need to switch pint YAML |
| 308 | parser into "relaxed" mode by adding this section to pint config file: |
| 309 | |
| 310 | ```js |
| 311 | parser { |
| 312 | relaxed = [ "my/files/*.yml" ] |
| 313 | } |
| 314 | ``` |
| 315 | |
| 316 | See [parser](configuration.md#parser) documentation for more details. |
| 317 | "Relaxed" parser mode will load anything that can be parsed as Prometheus rule, |
| 318 | while "strict" parser mode will fail if it reads a file that wouldn't load |
| 319 | cleanly as Prometheus config file. |
| 320 | |
| 321 | If your repository contains other files, not only Prometheus rules, then tell pint |
| 322 | to only check selected paths when running pint: |
| 323 | |
| 324 | ```js |
| 325 | parser { |
| 326 | include = [ "rules/dev/.*.yml", "rules/prod/.*" ] |
| 327 | } |
| 328 | ``` |
| 329 | |
| 330 | ## Control comments |
| 331 | |
| 332 | There is a number of comments you can add to your rule files in order to change |
| 333 | pint behaviour, some of them allow you to exclude selected files or line, see |
| 334 | [docs here](./ignoring.md) for details. |
| 335 | |
| 336 | There are a few requirements for any comment to be recognized by pint: |
| 337 | |
| 338 | - All comments must have a `pint` prefix. |
| 339 | - All comments must have at least one space between `#` symbol and `pint` prefix. |
| 340 | |
| 341 | Good comment examples: |
| 342 | |
| 343 | ```yaml |
| 344 | # pint file/owner bob |
| 345 | # pint file/owner bob |
| 346 | ``` |
| 347 | |
| 348 | Bad comment examples: |
| 349 | |
| 350 | ```yaml |
| 351 | # file/owner bob |
| 352 | #pint file/owner bob |
| 353 | ``` |
| 354 | |
| 355 | ## Release Notes |
| 356 | |
| 357 | See [changelog](changelog.md) for history of changes. |
| 358 | |
| 359 | ## Quick start |
| 360 | |
| 361 | Requirements: |
| 362 | |
| 363 | - [Git](https://git-scm.com/) |
| 364 | - [Go](https://golang.org/) - current stable release |
| 365 | |
| 366 | Steps: |
| 367 | |
| 368 | 1. Download a binary from [Releases](https://github.com/cloudflare/pint/releases) page |
| 369 | or build from source: |
| 370 | |
| 371 | ```shell |
| 372 | git clone https://github.com/cloudflare/pint.git |
| 373 | cd pint |
| 374 | make |
| 375 | ``` |
| 376 | |
| 377 | 2. Run a simple syntax check on Prometheus |
| 378 | [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) |
| 379 | or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) |
| 380 | rules file(s). |
| 381 | |
| 382 | ```shell |
| 383 | ./pint lint /etc/prometheus/*.rules.yml |
| 384 | ``` |
| 385 | |
| 386 | 3. Configuration file is optional, but without it, pint will only run very basic |
| 387 | syntax checks. See [configuration](configuration.md) for details on |
| 388 | config syntax. |
| 389 | By default pint will try to load configuration from `.pint.hcl`, you can |
| 390 | specify a different path using `--config` flag: |
| 391 | |
| 392 | ```shell |
| 393 | ./pint --config /etc/pint.hcl lint /etc/prometheus/rules/*.yml |
| 394 | ``` |
| 395 | |
| 396 | There are docker images available on [GitHub](https://github.com/cloudflare/pint/pkgs/container/pint). |
| 397 | Example usage: |
| 398 | |
| 399 | ```shell |
| 400 | docker run --mount=type=bind,source="$(pwd)",target=/rules,readonly ghcr.io/cloudflare/pint pint lint /rules |
| 401 | ``` |
| 402 | |
| 403 | ## License |
| 404 | |
| 405 | ```text |
| 406 | Copyright (c) 2021-2023 Cloudflare, Inc. |
| 407 | |
| 408 | Licensed under the Apache License, Version 2.0 (the "License"); |
| 409 | you may not use this file except in compliance with the License. |
| 410 | You may obtain a copy of the License at |
| 411 | |
| 412 | http://www.apache.org/licenses/LICENSE-2.0 |
| 413 | |
| 414 | Unless required by applicable law or agreed to in writing, software |
| 415 | distributed under the License is distributed on an "AS IS" BASIS, |
| 416 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 417 | See the License for the specific language governing permissions and |
| 418 | limitations under the License. |
| 419 | ``` |
| 420 | |