cloudflare/pint
Publicmirrored fromhttps://github.com/cloudflare/pintAvailable
docs/configuration.md
759lines · modecode
| 1 | --- |
| 2 | layout: default |
| 3 | title: Configuration |
| 4 | parent: Documentation |
| 5 | nav_order: 2 |
| 6 | --- |
| 7 | |
| 8 | # Configuration syntax |
| 9 | |
| 10 | ## Table of contents |
| 11 | |
| 12 | {: .no_toc .text-delta } |
| 13 | |
| 14 | 1. TOC |
| 15 | {:toc} |
| 16 | |
| 17 | ## Environment variables |
| 18 | |
| 19 | Environment variables can be expanded inside the pint configuration file as `ENV_*` HCL |
| 20 | variables. To use a variable named `FOO`, reference it as `${ENV_FOO}`. |
| 21 | |
| 22 | Examples: |
| 23 | |
| 24 | If you have `AUTH_KEY` environment variable set that you want to use a header |
| 25 | for Prometheus requests then use this: |
| 26 | |
| 27 | ```js |
| 28 | prometheus "..." { |
| 29 | uri = "https://..." |
| 30 | headers = { |
| 31 | "X-Auth": "${ENV_AUTH_KEY}" |
| 32 | } |
| 33 | } |
| 34 | ``` |
| 35 | |
| 36 | ## Regexp matchers |
| 37 | |
| 38 | All regexp patterns use [Go regexp](https://pkg.go.dev/regexp) module and are fully anchored. |
| 39 | This means that when you pass `.*` regexp expression internally it will be represented as |
| 40 | `^.*$`, where `^` indicates beginning of a string and `$` is the end of string. |
| 41 | This follows [PromQL behaviour](https://prometheus.io/docs/prometheus/latest/querying/basics/) |
| 42 | for consistency with Prometheus. |
| 43 | If you have a string `alice bob john` and you want to match a sub-string `bob`, then be sure to use |
| 44 | `.*bob.*`. |
| 45 | |
| 46 | When using regexp matcher in checks configuration you can reference alerting and recording rule |
| 47 | fields in the regexp using [Go text/template](https://pkg.go.dev/text/template) syntax. |
| 48 | Rule fields are exposed as: |
| 49 | |
| 50 | - `$alert` - rule `alert` field |
| 51 | - `$record` - rule `record` field |
| 52 | - `$expr` - rule `expr` field |
| 53 | - `$for` - rule `for` field |
| 54 | - `$labels` - rule `labels` map, individual labels can be accessed as `$labels.foo` |
| 55 | - `$annotations` - rule `annotations` map, individual annotations can be accessed as `$annotations.foo` |
| 56 | |
| 57 | Accessing a field that's not present in the rule will return an empty string. |
| 58 | |
| 59 | ## Parser |
| 60 | |
| 61 | Configure how pint parses Prometheus rule files. |
| 62 | |
| 63 | Syntax: |
| 64 | |
| 65 | ```js |
| 66 | parser { |
| 67 | schema = "prometheus|thanos" |
| 68 | names = "legacy|utf-8" |
| 69 | include = [ "(.*)", ... ] |
| 70 | exclude = [ "(.*)", ... ] |
| 71 | relaxed = [ "(.*)", ... ] |
| 72 | } |
| 73 | ``` |
| 74 | |
| 75 | - `schema` - rule file schema to use when using `strict` parser mode, valid values are `prometheus` and `thanos`. |
| 76 | This option has no effect when `relaxed` mode is enabled, see below. |
| 77 | Setting it to `prometheus` means that pint will assume that all rule files have the schema |
| 78 | as defined in [alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) |
| 79 | and [recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) |
| 80 | Prometheus docs. |
| 81 | Setting it to `thanos` will tell pint to use the schema as defined |
| 82 | in [Thanos Rule](https://thanos.io/tip/components/rule.md/) docs, which currently allows for setting |
| 83 | an extra key on the rule group object - `partial_response_strategy`. |
| 84 | Default value is `prometheus`. |
| 85 | - `names` - validation scheme for label names. This controls whether Prometheus libraries are |
| 86 | allowing UTF-8 in label **names** or not. See [utf-8 guide](https://prometheus.io/docs/guides/utf8/). |
| 87 | Default value is `utf-8`. |
| 88 | - `include` - list of regexp patterns to check when running checks. Only files |
| 89 | matching those regexp rules will be checked, other modified files will be ignored. |
| 90 | - `exclude` - list of regexp patterns to ignore when running checks. |
| 91 | This option takes precedence over `include`, so if a file path matches both |
| 92 | `include` & `exclude` regexp patterns, it will be excluded. |
| 93 | - `relaxed` - by default, pint will parse all files in strict mode, where |
| 94 | all rule files must have the exact syntax Prometheus or Thanos expects: |
| 95 | |
| 96 | When using `schema: prometheus` (default): |
| 97 | |
| 98 | ```yaml |
| 99 | groups: |
| 100 | - name: example |
| 101 | rules: |
| 102 | - record: ... |
| 103 | expr: ... |
| 104 | - alert: ... |
| 105 | expr: ... |
| 106 | ``` |
| 107 | |
| 108 | When using `schema: thanos`: |
| 109 | |
| 110 | ```yaml |
| 111 | groups: |
| 112 | - name: example |
| 113 | partial_response_strategy: abort |
| 114 | rules: |
| 115 | - record: ... |
| 116 | expr: ... |
| 117 | - alert: ... |
| 118 | expr: ... |
| 119 | ``` |
| 120 | |
| 121 | If you're using pint to lint rules that are embedded inside a different structure |
| 122 | you can set this option to allow fuzzy parsing, which will try to find rule |
| 123 | definitions anywhere in the file, without requiring `groups -> rules -> rule` |
| 124 | structure to be present. |
| 125 | This option takes a list of regexp patterns, all files matching those regexp rules |
| 126 | will be parsed in relaxed mode. |
| 127 | |
| 128 | **NOTE**: remember that all options that accept regexp patterns (shown as `"(.*)"` in syntax docs) are **anchored**. |
| 129 | If you do set any patterns in the `parser` section then they must match the way you run pint. |
| 130 | For example, if you have a git repository with a `rules` directory containing Prometheus rules then |
| 131 | you might set `include = ["rules/.*"]` in the config. Internally, it will be parsed as the regexp `^rules/.*$`. |
| 132 | This means that if you then pass absolute paths to pint commands (`pint lint /my/repo/rules`) it won't match that regexp. |
| 133 | If you do want to use pint with both relative and absolute paths, make sure your regexp patterns allow for it, example: |
| 134 | |
| 135 | ```js |
| 136 | parser { |
| 137 | include = [ "(.*/)?rules/.*" ] |
| 138 | } |
| 139 | ``` |
| 140 | |
| 141 | Or explicitly set both relative and absolute path: |
| 142 | |
| 143 | ```js |
| 144 | parser { |
| 145 | include = [ |
| 146 | "rules/.*", |
| 147 | "/my/repo/rules/.*", |
| 148 | ] |
| 149 | } |
| 150 | ``` |
| 151 | |
| 152 | ## Owners |
| 153 | |
| 154 | When `pint ci` or `pint lint` is run with `--require-owner` flag it will require |
| 155 | all Prometheus rules to have an owner assigned via comment. |
| 156 | See [rule/owner](checks/rule/owner.md) for details. |
| 157 | |
| 158 | Those checks can be further customised by setting a list of allowed owner names. |
| 159 | |
| 160 | Syntax: |
| 161 | |
| 162 | ```js |
| 163 | owners { |
| 164 | allowed = [ "(.*)", ... ] |
| 165 | } |
| 166 | ``` |
| 167 | |
| 168 | - `allowed` - list of allowed owner names, this option accepts regexp rules. |
| 169 | When set, all owners set via comments must match at least one entry on this list. |
| 170 | |
| 171 | If there's no `owners:allowed` configuration block, or if it's empty, then any |
| 172 | owner name is accepted. |
| 173 | |
| 174 | ## CI |
| 175 | |
| 176 | Configure continuous integration environments. |
| 177 | |
| 178 | Syntax: |
| 179 | |
| 180 | ```js |
| 181 | ci { |
| 182 | maxCommits = 20 |
| 183 | baseBranch = "master" |
| 184 | } |
| 185 | ``` |
| 186 | |
| 187 | - `maxCommits` - by default, pint will try to find all commits on the current branch, |
| 188 | this requires full git history to be present, if we have a shallow clone this |
| 189 | might fail to find only current branch commits and give us a huge list. |
| 190 | If the number of commits returned by branch discovery is more than `maxCommits`, |
| 191 | then pint will fail to run. |
| 192 | - `baseBranch` - base branch to compare `HEAD` commit with when calculating the list |
| 193 | of commits to check. |
| 194 | |
| 195 | ## Repository |
| 196 | |
| 197 | Configure supported code hosting repository, used for reporting PR checks from CI |
| 198 | back to the repository, to be displayed in the PR UI. |
| 199 | Currently supported platforms are: |
| 200 | |
| 201 | - [BitBucket](https://bitbucket.org) |
| 202 | - [GitHub](https://github.com) |
| 203 | - [GitLab](https://gitlab.com) |
| 204 | |
| 205 | **NOTE**: BitBucket integration requires `BITBUCKET_AUTH_TOKEN` environment variable |
| 206 | to be set. It should contain a personal access token used to authenticate with the API. |
| 207 | |
| 208 | **NOTE**: GitHub integration requires `GITHUB_AUTH_TOKEN` environment variable |
| 209 | to be set to a personal access key that can access your repository. |
| 210 | |
| 211 | **NOTE**: GitLab integration requires `GITLAB_AUTH_TOKEN` environment variable |
| 212 | to be set to a personal access key that can access your repository. |
| 213 | |
| 214 | **NOTE** The pull request number must be known to pint so it can add comments if it detects any problems. |
| 215 | If pint is run as part of GitHub actions workflow, then this number will be detected from `GITHUB_REF` |
| 216 | environment variable. For other use cases, the `GITHUB_PULL_REQUEST_NUMBER` environment variable must be set |
| 217 | with the pull request number. |
| 218 | |
| 219 | Syntax: |
| 220 | |
| 221 | ```js |
| 222 | repository { |
| 223 | bitbucket { ... } |
| 224 | github { ... } |
| 225 | gitlab { ... } |
| 226 | } |
| 227 | ``` |
| 228 | |
| 229 | ### BitBucket options |
| 230 | |
| 231 | Syntax: |
| 232 | |
| 233 | ```js |
| 234 | repository { |
| 235 | bitbucket { |
| 236 | uri = "https://..." |
| 237 | timeout = "1m" |
| 238 | project = "..." |
| 239 | repository = "..." |
| 240 | maxComments = 50 |
| 241 | } |
| 242 | } |
| 243 | ``` |
| 244 | |
| 245 | - `bitbucket:uri` - base URI of this repository, will be used for HTTP |
| 246 | requests to the BitBucket API. |
| 247 | - `bitbucket:timeout` - timeout to be used for API requests, defaults to 1 minute. |
| 248 | - `bitbucket:project` - name of the BitBucket project for this repository. |
| 249 | - `bitbucket:repository` - name of the BitBucket repository. |
| 250 | - `bitbucket:maxComments` - the maximum number of comments pint can create on a single |
| 251 | pull request. Default is 50. |
| 252 | |
| 253 | ### GitHub options |
| 254 | |
| 255 | ```js |
| 256 | repository { |
| 257 | github { |
| 258 | baseuri = "https://..." |
| 259 | uploaduri = "https://..." |
| 260 | timeout = "1m" |
| 261 | owner = "..." |
| 262 | repo = "..." |
| 263 | maxComments = 50 |
| 264 | } |
| 265 | } |
| 266 | ``` |
| 267 | |
| 268 | - `github:baseuri` - base URI of GitHub or GitHub enterprise, will be used for HTTP requests to the GitHub API. |
| 269 | If not set, `pint` will try to use the `GITHUB_API_URL` environment variable instead (if set). |
| 270 | - `github:uploaduri` - upload URI of GitHub or GitHub enterprise, will be used for HTTP requests to the GitHub API. |
| 271 | If not set, `pint` will try to use the `GITHUB_API_URL` environment variable instead (if set). |
| 272 | |
| 273 | If `github:baseuri` _or_ `github:uploaduri` are not specified, then [GitHub](https://github.com) will be used. |
| 274 | |
| 275 | - `github:timeout` - timeout to be used for API requests, defaults to 1 minute. |
| 276 | - `github:owner` - name of the GitHub owner i.e. the first part that comes before the repository's name in the URI. |
| 277 | If not set, `pint` will try to use the `GITHUB_REPOSITORY` environment variable instead (if set). |
| 278 | - `github:repo` - name of the GitHub repository (e.g. `monitoring`). |
| 279 | If not set, `pint` will try to use the `GITHUB_REPOSITORY` environment variable instead (if set). |
| 280 | - `github:maxComments` - the maximum number of comments pint can create on a single pull request. Default is 50. |
| 281 | |
| 282 | Most GitHub settings can be detected from environment variables that are set inside GitHub Actions |
| 283 | environment. The only exception is `GITHUB_AUTH_TOKEN` environment variable that must be set |
| 284 | manually. |
| 285 | |
| 286 | ### GitLab options |
| 287 | |
| 288 | ```js |
| 289 | repository { |
| 290 | gitlab { |
| 291 | uri = "https://..." |
| 292 | timeout = "1m" |
| 293 | project = "..." |
| 294 | maxComments = 50 |
| 295 | } |
| 296 | } |
| 297 | ``` |
| 298 | |
| 299 | - `gitlab:uri` - optional base URI for GitLab API calls when using self hosted setup. |
| 300 | You don't need to set it if you use repositories hosted on [gitlab.com](https://gitlab.com/). |
| 301 | - `gitlab:timeout` - timeout to be used for API requests, defaults to 1 minute. |
| 302 | - `gitlab:project` - ID of the GitLab repository. |
| 303 | - `gitlab:maxComments` - the maximum number of comments pint can create on a single pull request. Default is 50. |
| 304 | |
| 305 | ## Prometheus servers |
| 306 | |
| 307 | Some checks work by querying a running Prometheus instance to verify if |
| 308 | metrics used in rules are present. If you want to use those checks, then you |
| 309 | first need to define one or more Prometheus servers. |
| 310 | |
| 311 | Syntax: |
| 312 | |
| 313 | ```js |
| 314 | prometheus "$name" { |
| 315 | uri = "https://..." |
| 316 | publicURI = "https://..." |
| 317 | failover = ["https://...", ...] |
| 318 | tags = ["...", ...] |
| 319 | headers = { "...": "..." } |
| 320 | timeout = "2m" |
| 321 | concurrency = 16 |
| 322 | rateLimit = 100 |
| 323 | required = true|false |
| 324 | include = ["...", ...] |
| 325 | exclude = ["...", ...] |
| 326 | tls { |
| 327 | serverName = "..." |
| 328 | caCert = "..." |
| 329 | clientCert = "..." |
| 330 | clientKey = "..." |
| 331 | skipVerify = true|false |
| 332 | } |
| 333 | } |
| 334 | ``` |
| 335 | |
| 336 | - `$name` - each defined server should have a unique name that can be used in check |
| 337 | definitions. |
| 338 | - `uri` - base URI of this Prometheus server, used for API requests and queries. |
| 339 | - `publicURI` - optional URI to use instead of `uri` in problems reported to users. |
| 340 | Set it if Prometheus links used by pint in comments submitted to BitBucket or GitHub |
| 341 | should use different URIs than the one used by pint when querying Prometheus. |
| 342 | If not set, `uri` will be used instead. |
| 343 | - `failover` - list of URIs to try (in order they are specified) if `uri` doesn't respond |
| 344 | to requests or returns an error. This allows one to configure fail-over Prometheus servers |
| 345 | to avoid CI failures in case the main Prometheus server is unreachable. |
| 346 | Fail over URIs are not used if Prometheus returns an error caused by the query, like |
| 347 | `many-to-many matching not allowed`. |
| 348 | It's highly recommended that all URIs point to Prometheus servers with identical |
| 349 | configuration, otherwise pint checks might return unreliable results and potential |
| 350 | false positives. |
| 351 | - `tags` - a list of strings that can be used to group Prometheus servers together. |
| 352 | Tags cannot contain spaces. |
| 353 | Tags can be later used when disabling checks via comments, see [ignoring](ignoring.md). |
| 354 | - `headers` - a list of HTTP headers that will be set on all requests for this Prometheus |
| 355 | server. |
| 356 | - `timeout` - timeout to be used for API requests. Defaults to 2 minutes. |
| 357 | - `concurrency` - how many concurrent requests pint can send to this Prometheus server. |
| 358 | Optional, defaults to 16. |
| 359 | - `rateLimit` - per second rate limit for all API requests sent to this Prometheus server. |
| 360 | Setting it to `1000` would allow for up to 1000 requests per each wall clock second. |
| 361 | Optional, default to 100 requests per second. |
| 362 | - `uptime` - metric selector used to detect gaps in Prometheus uptime. |
| 363 | Since some checks are sending queries to validate if given metric always present in Prometheus |
| 364 | they might find gaps when Prometheus itself was down. Pint tries to detect that by querying |
| 365 | metrics that are always guaranteed to be present when Prometheus is running. |
| 366 | By default metric used for this is `up`, which is generated by Prometheus itself, see |
| 367 | [Prometheus docs](https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series) |
| 368 | for details. |
| 369 | Uptime gap detection works by running a range query `count(up)` and checking for any gaps |
| 370 | in the response. |
| 371 | Since the `up` metric can have a lot of time series, `count(up)` might be slow and expensive. |
| 372 | An alternative is to use one of metrics exposed by Prometheus itself, like `prometheus_build_info`, but |
| 373 | those metrics are only present if Prometheus is configured to scrape itself, so `up` is used by default |
| 374 | since it's guaranteed to work in every setup. |
| 375 | If your Prometheus has a lot of time series and it's configured to scrape itself, then |
| 376 | it is recommended to set the `uptime` field to `prometheus_build_info`. |
| 377 | - `required` - decides how pint will report errors if it's unable to get a valid response |
| 378 | from this Prometheus server. If `required` is `true` and all API calls to this Prometheus |
| 379 | fail, pint will report those as `bug` level problems. If it's set to `false`, pint will |
| 380 | report those with the `warning` level. |
| 381 | Default value for `required` is `false`. Set it to `true` if you want to hard fail |
| 382 | in case of remote Prometheus issues. Note that setting it to `true` might block |
| 383 | PRs when running `pint ci` until pint is able to talk to Prometheus again. |
| 384 | - `include` - optional path filter, if specified only paths matching one of listed regexp |
| 385 | patterns will use this Prometheus server for checks. |
| 386 | - `exclude` - optional path filter, if specified any path matching one of listed regexp |
| 387 | patterns will never use this Prometheus server for checks. |
| 388 | `exclude` takes precedence over `include`. |
| 389 | - `tls` - optional TLS configuration for HTTP requests sent to this Prometheus server. |
| 390 | - `tls:serverName` - server name (SNI) for TLS handshakes. Optional, default is unset. |
| 391 | - `tls:caCert` - path for CA certificate to use. Optional, default is unset. |
| 392 | - `tls:clientCert` - path for client certificate to use. Optional, default is unset. |
| 393 | If set, `clientKey` must also be set. |
| 394 | - `tls:clientKey` - path for client key file to use. Optional, default is unset. |
| 395 | If set, `clientCert` must also be set. |
| 396 | - `tls:skipVerify` - if `true` all TLS certificate checks will be skipped. |
| 397 | Enabling this option can be a security risk; use only for testing. |
| 398 | Optional, default is false. |
| 399 | |
| 400 | Example: |
| 401 | |
| 402 | ```js |
| 403 | prometheus "prod" { |
| 404 | uri = "https://prometheus-prod.example.com" |
| 405 | tags = ["prod"] |
| 406 | headers = { |
| 407 | "X-Auth": "secret" |
| 408 | } |
| 409 | concurrency = 40 |
| 410 | } |
| 411 | |
| 412 | prometheus "prod-tls" { |
| 413 | uri = "https://prometheus-tls.example.com" |
| 414 | tags = ["prod"] |
| 415 | tls { |
| 416 | serverName = "prometheus.example.com" |
| 417 | caCert = "/ssl/ca.pem" |
| 418 | clientCert = "/ssl/client.pem" |
| 419 | clientKey = "/ssl/client.key" |
| 420 | } |
| 421 | } |
| 422 | |
| 423 | prometheus "staging" { |
| 424 | uri = "https://prometheus-staging.example.com" |
| 425 | uptime = "prometheus_build_info" |
| 426 | } |
| 427 | |
| 428 | prometheus "dev" { |
| 429 | uri = "https://prometheus-dev.example.com" |
| 430 | timeout = "30s" |
| 431 | include = [ "alerts/test/.*" ] |
| 432 | exclude = [ "alerts/test/docs/.*" ] |
| 433 | } |
| 434 | ``` |
| 435 | |
| 436 | ## Prometheus discovery |
| 437 | |
| 438 | Sometimes specifying a static list of Prometheus server definitions in pint |
| 439 | configuration is not possible and a dynamic discovery of running Prometheus |
| 440 | instances is needed. This can be configured using `discovery` config blocks. |
| 441 | |
| 442 | ### File path discovery |
| 443 | |
| 444 | File path discovery allows to generate Prometheus server definitions used by pint |
| 445 | based on path regexp patterns on disk. |
| 446 | Syntax: |
| 447 | |
| 448 | ```js |
| 449 | filepath { |
| 450 | directory = "..." |
| 451 | match = "(.*)" |
| 452 | ignore = [ "(.*)", ... ] |
| 453 | template { ... } |
| 454 | template { ... } |
| 455 | } |
| 456 | ``` |
| 457 | |
| 458 | - `directory` - the base directory to scan for paths. |
| 459 | - `match` - a regexp expression to match. Any named capture group defined here |
| 460 | will be accessible when rendering the Prometheus template. |
| 461 | - `ignore` - a list of regexp rules, any path matching any of these rules will |
| 462 | be ignored. |
| 463 | - `template` - a template for generating Prometheus server definitions. |
| 464 | See below. |
| 465 | |
| 466 | ### Prometheus query discovery |
| 467 | |
| 468 | ```js |
| 469 | prometheusQuery { |
| 470 | uri = "https://..." |
| 471 | headers = { "...": "..." } |
| 472 | timeout = "2m" |
| 473 | tls { |
| 474 | serverName = "..." |
| 475 | caCert = "..." |
| 476 | clientCert = "..." |
| 477 | clientKey = "..." |
| 478 | skipVerify = true|false |
| 479 | } |
| 480 | template { ... } |
| 481 | template { ... } |
| 482 | } |
| 483 | ``` |
| 484 | |
| 485 | - `uri` - Prometheus server base URI. This is where the discovery query will be sent. |
| 486 | - `headers` - optional list of headers to set on Prometheus query requests. |
| 487 | - `timeout` - Prometheus request timeout. Defaults to 2 minutes. |
| 488 | - `tls` - optional TLS configuration for Prometheus requests, see `prometheus` block |
| 489 | documentation for details. |
| 490 | - `query` - the PromQL query to use for discovery of Prometheus servers. |
| 491 | Every returned time series will generate a new Prometheus server definition using attached |
| 492 | `template` definition. You can set multiple `template` blocks for each discovery block, each |
| 493 | returned time series will generate a single Prometheus server for each `template` block. |
| 494 | You can use labels on returned time series as [Go text/template](https://pkg.go.dev/text/template) |
| 495 | variables named `$name`. Example: `instance` label will be available as the `$instance` variable. |
| 496 | |
| 497 | ### Prometheus template |
| 498 | |
| 499 | The `template` block is nearly identical to the `prometheus` configuration block, except that |
| 500 | `name` is an explicit field inside the block. |
| 501 | |
| 502 | You can use [Go text/template](https://pkg.go.dev/text/template) to render some of the |
| 503 | fields using variables from either regexp capture groups (when using `filepath` discovery) |
| 504 | or metric labels (when using `prometheusQuery` discovery). |
| 505 | |
| 506 | Fields that are allowed to be templated are: |
| 507 | |
| 508 | - `name` |
| 509 | - `uri` |
| 510 | - `failover` |
| 511 | - `headers` |
| 512 | - `tags` |
| 513 | - `include` |
| 514 | - `exclude` |
| 515 | |
| 516 | ```js |
| 517 | template { |
| 518 | name = "..." |
| 519 | uri = "https://..." |
| 520 | failover = ["https://...", ...] |
| 521 | tags = ["...", ...] |
| 522 | headers = { "...": "..." } |
| 523 | timeout = "2m" |
| 524 | concurrency = 16 |
| 525 | rateLimit = 100 |
| 526 | required = true|false |
| 527 | include = ["...", ...] |
| 528 | exclude = ["...", ...] |
| 529 | tls { |
| 530 | serverName = "..." |
| 531 | caCert = "..." |
| 532 | clientCert = "..." |
| 533 | clientKey = "..." |
| 534 | skipVerify = true|false |
| 535 | } |
| 536 | } |
| 537 | ``` |
| 538 | |
| 539 | Generated Prometheus servers will be deduplicated. If there are multiple servers with the same |
| 540 | name but different URIs then extra URIs will be added to `failover` list. |
| 541 | Servers must have identical `name`, `tags`, `include` and `exclude` fields to be deduplicated. |
| 542 | |
| 543 | ### Examples |
| 544 | |
| 545 | Each Prometheus server has a sub-directory inside `/etc/prometheus/clusters` |
| 546 | folder. This directory is named after the Prometheus cluster it's part of. |
| 547 | All rules are stored in `/etc/prometheus/clusters/<cluster>/.*.yaml` or |
| 548 | `/etc/prometheus/clusters/<cluster>/.*.yml` files. |
| 549 | Each Prometheus cluster is accessible under `https://<cluster>.prometheus.example.com` URI. |
| 550 | |
| 551 | ```js |
| 552 | filepath { |
| 553 | directory = "/etc/prometheus/clusters" |
| 554 | match = "(?P<cluster>[a-z]+[0-9]{2})" |
| 555 | ignore = [ "staging[0-9]+" ] |
| 556 | template { |
| 557 | name = "cluster-{{ $cluster }}" |
| 558 | uri = "https://{{ $cluster }}.prometheus.example.com" |
| 559 | uptime = "prometheus_ready" |
| 560 | include = [ |
| 561 | "/etc/prometheus/clusters/{{ $cluster }}/.*.ya?ml", |
| 562 | ] |
| 563 | } |
| 564 | } |
| 565 | ``` |
| 566 | |
| 567 | ## Matching rules to checks |
| 568 | |
| 569 | Most checks, except basic syntax verification, require some configuration to decide |
| 570 | which checks to run against which files and rules. |
| 571 | |
| 572 | Syntax: |
| 573 | |
| 574 | ```js |
| 575 | rule { |
| 576 | locked = true|false |
| 577 | match { |
| 578 | path = "(.+)" |
| 579 | state = [ "any|added|modified|renamed|removed|unmodified", ... ] |
| 580 | name = "(.+)" |
| 581 | kind = "alerting|recording" |
| 582 | command = "ci|lint|watch" |
| 583 | annotation "(.*)" { |
| 584 | value = "(.*)" |
| 585 | } |
| 586 | label "(.*)" { |
| 587 | value = "(.*)" |
| 588 | } |
| 589 | for = "..." |
| 590 | } |
| 591 | match { ... } |
| 592 | match { ... } |
| 593 | ignore { |
| 594 | path = "(.+)" |
| 595 | state = [ "any|added|modified|renamed|removed|unmodified", ... ] |
| 596 | name = "(.+)" |
| 597 | kind = "alerting|recording" |
| 598 | command = "ci|lint|watch" |
| 599 | annotation "(.*)" { |
| 600 | value = "(.*)" |
| 601 | } |
| 602 | label "(.*)" { |
| 603 | value = "(.*)" |
| 604 | } |
| 605 | for = "..." |
| 606 | } |
| 607 | ignore { ... } |
| 608 | ignore { ... } |
| 609 | |
| 610 | enable = [ "..." ] |
| 611 | disable = [ "..." ] |
| 612 | |
| 613 | [ check definition ] |
| 614 | ... |
| 615 | [ check definition ] |
| 616 | } |
| 617 | ``` |
| 618 | |
| 619 | - `locked` - if set to `true` this rule will be locked, which means that it cannot be disabled using |
| 620 | `# pint disable ...` or `# pint snooze ...` comments. |
| 621 | - `match:path` - only files matching this regexp pattern will be checked by this rule. |
| 622 | - `match:state` - only match rules based on their state. Default value for `state` depends on the |
| 623 | pint command that is being run, for `pint ci` the default value is `["added", "modified", "renamed"]`, |
| 624 | for any other command the default value is `["any"]`. |
| 625 | Possible values: |
| 626 | - `any` - match rule in any state |
| 627 | - `added` - a rule is being added in a git commit, a rule can only be in this state when running `pint ci` on a pull request. |
| 628 | - `modified` - a rule is being modified in a git commit, a rule can only be in this state when running `pint ci` on a pull request. |
| 629 | - `renamed` - a rule is being modified in a git commit, a rule can only be in this state when running `pint ci` on a pull request. |
| 630 | - `removed` - a rule is being removed in a git commit, a rule can only be in this state when running `pint ci` on a pull request. |
| 631 | - `unmodified` - a rule is not being modified in a git commit when running `pint ci` or other pint command was run that doesn't try to detect which rules were modified. |
| 632 | - `match:name` - only rules with names (`record` for recording rules and `alert` for alerting |
| 633 | rules) matching this regexp pattern will be checked rule. |
| 634 | - `match:kind` - optional rule type filter, only rules of this type will be checked. |
| 635 | - `match:command` - optional command type filter, this allows to include or ignore rules |
| 636 | based on the command pint is run with `pint ci`, `pint lint` or `pint watch`. |
| 637 | - `match:annotation` - optional annotation filter, only alert rules with at least one |
| 638 | annotation matching this regexp pattern will be checked by this rule. |
| 639 | - `match:label` - optional annotation filter, only rules with at least one label |
| 640 | matching this regexp pattern will be checked by this rule. For recording rules only static |
| 641 | labels set on the recording rule are considered. |
| 642 | - `match:for` - optional alerting rule `for` filter. If set, only alerting rules with the `for` |
| 643 | field present and matching the provided value will be checked by this rule. Recording rules |
| 644 | will never match it as they don't have the `for` field. |
| 645 | Syntax is `OP DURATION` where `OP` can be any of `=`, `!=`, `>`, `>=`, `<`, `<=`. |
| 646 | - `match:keep_firing_for` - optional alerting rule `keep_firing_for` filter. Works the same |
| 647 | way as `for` match filter. |
| 648 | - `ignore` - works exactly like `match` but does the opposite - any alerting or recording rule |
| 649 | matching all conditions defined on `ignore` will not be checked by this `rule` block. |
| 650 | - `enable` - list of check names to enable for any Prometheus rule matching this block. |
| 651 | Enabling checks here will overwrite `check { disable = [...] }` settings, but won't |
| 652 | enable checks disabled specifically for some Prometheus rule via `# pint disable ...` comments. |
| 653 | - `disable` - list of check names to disable for any Prometheus rule matching this block. |
| 654 | This takes precedence over `enable` option above. |
| 655 | |
| 656 | Note: both `match` and `ignore` require all defined filters to be satisfied to work. |
| 657 | If multiple `match` and/or `ignore` rules are present any of them needs to match for the rule to |
| 658 | be matched / ignored. |
| 659 | |
| 660 | Examples: |
| 661 | |
| 662 | Check applied only to severity="critical" and severity="warning" alerts in "ci" or "lint" command is run: |
| 663 | |
| 664 | ```js |
| 665 | rule { |
| 666 | match { |
| 667 | path = "rules/.*" |
| 668 | kind = "alerting" |
| 669 | label "severity" { |
| 670 | value = "(warning|critical)" |
| 671 | } |
| 672 | } |
| 673 | ignore { |
| 674 | command = "watch" |
| 675 | } |
| 676 | check { ... } |
| 677 | } |
| 678 | ``` |
| 679 | |
| 680 | Check applied unless "watch" or "lint" command is run: |
| 681 | |
| 682 | ```js |
| 683 | rule { |
| 684 | ignore { |
| 685 | command = "watch" |
| 686 | } |
| 687 | ignore { |
| 688 | command = "lint" |
| 689 | } |
| 690 | check { ... } |
| 691 | } |
| 692 | ``` |
| 693 | |
| 694 | Check applied only to alerting rules with "for" field value that is >= 5m: |
| 695 | |
| 696 | ```js |
| 697 | rule { |
| 698 | match { |
| 699 | for = ">= 5m" |
| 700 | } |
| 701 | check { ... } |
| 702 | } |
| 703 | ``` |
| 704 | |
| 705 | Check applied only to alerting rules with "keep_firing_for" field value that is > 15m: |
| 706 | |
| 707 | ```js |
| 708 | rule { |
| 709 | match { |
| 710 | keep_firing_for = "> 15m" |
| 711 | } |
| 712 | check { ... } |
| 713 | } |
| 714 | ``` |
| 715 | |
| 716 | Check that is run on all rules, including unmodified files, when running `pint ci`: |
| 717 | |
| 718 | ```js |
| 719 | rule { |
| 720 | match { |
| 721 | state = ["any"] |
| 722 | } |
| 723 | check { ... } |
| 724 | } |
| 725 | ``` |
| 726 | |
| 727 | Disable `promql/rate` check for all rules except alerting rules in the `rules/critical` folder: |
| 728 | |
| 729 | ```js |
| 730 | checks { |
| 731 | // This will disable promql/rate by default. |
| 732 | disabled = [ "promql/rate" ] |
| 733 | } |
| 734 | rule { |
| 735 | match { |
| 736 | path = "rules/critical/.*" |
| 737 | kind = "alerting" |
| 738 | } |
| 739 | // This will enable promql/rate only for Prometheus rules matching all our match conditions above. |
| 740 | enable = [ "promql/rate" ] |
| 741 | } |
| 742 | ``` |
| 743 | |
| 744 | Lock a rule so it cannot be disabled via comments: |
| 745 | |
| 746 | ```js |
| 747 | rule { |
| 748 | locked = true |
| 749 | |
| 750 | match { |
| 751 | kind = "alerting" |
| 752 | } |
| 753 | |
| 754 | // This check cannot be disabled using # pint disable ... or # pint snooze ... comments. |
| 755 | for { |
| 756 | min = "5m" |
| 757 | } |
| 758 | } |
| 759 | ``` |
| 760 | |