cloudflare/pint

Public

mirrored from https://github.com/cloudflare/pintAvailable

CodeCommitsIssuesPull requestsActionsInsightsSecurity
55fb0e1f205e56ef28d92de31d76e030d6d7a5a1

Branches

Tags

  • No tags available.
0Branches0Tags
Go to file
Add file
Code

Clone

HTTPS

Download ZIP

docs/CONFIGURATION.md

727lines · modecode

1# Configuration syntax
2
3**NOTE** all regex pattern are anchored.
4
5## CI
6
7Configure continuous integration environments.
8
9Syntax:
10
11```JS
12ci {
13 include = [ "(.*)", ... ]
14 maxCommits = 20
15 baseBranch = "master"
16}
17```
18
19- `include` - list of file patters to checks when running checks. Only files
20 matching those regex rules will be checked, other modified files will be ignored.
21- `maxCommits` - by default pint will try to find all commits on current branch,
22 this requires full git history to be present, if we have a shallow clone this
23 might fail to find only current branch commits and give us a huge list.
24 If the number of commits returned by branch discovery is more than `maxCommits`
25 then pint will fail to run.
26- `baseBranch` - base branch to compare `HEAD` commit with when calculating the list
27 of commits to check.
28
29## Repository
30
31Configure supported code hosting repository, used for reporting PR checks from CI
32back to the repository, to be displayed in the PR UI.
33Currently only supports [BitBucket](https://bitbucket.org/) and [GitHub](https://github.com/).
34
35**NOTE**: BitBucket integration requires `BITBUCKET_AUTH_TOKEN` environment variable
36to be set. It should contain a personal access token used to authenticate with the API.
37
38**NOTE**: GitHub integration requires `GITHUB_AUTH_TOKEN` environment variable
39to be set to a personal access key that can access your repository. Also, `GITHUB_PULL_REQUEST_NUMBER`
40environment variable needs to point to the pull request number which will be used whilst
41submitting comments.
42
43Syntax:
44
45```JS
46repository {
47 bitbucket {
48 uri = "https://..."
49 timeout = "30s"
50 project = "..."
51 repository = "..."
52 }
53}
54```
55
56- `bitbucket:uri` - base URI of this repository, will be used for HTTP
57 requests to the BitBucket API.
58- `bitbucket:timeout` - timeout to be used for API requests.
59- `bitbucket:project` - name of the BitBucket project for this repository.
60- `bitbucket:repository` - name of the BitBucket repository.
61
62```JS
63repository {
64 github {
65 uri = "https://..."
66 timeout = "30s"
67 owner = "..."
68 repo = "..."
69 }
70}
71```
72
73- `github:baseuri` - base URI of GitHub or GitHub enterprise, will be used for HTTP requests to the GitHub API.
74- `github:uploaduri` - upload URI of GitHub or GitHub enterprise, will be used for HTTP requests to the GitHub API.
75
76If `github:baseuri` _or_ `github:uploaduri` are not specified then [GitHub](https://github.com) will be used.
77
78- `github:timeout` - timeout to be used for API requests;
79- `github:owner` - name of the GitHub owner i.e. the first part that comes before the repository's name in the URI;
80- `github:repo` - name of the GitHub repository (e.g. `monitoring`).
81
82## Prometheus servers
83
84Some checks work by querying a running Prometheus instance to verify if
85metrics used in rules are present. If you want to use those checks then you
86first need to define one or more Prometheus servers.
87
88Syntax:
89
90```JS
91prometheus "$name" {
92 uri = "https://..."
93 timeout = "60s"
94 paths = ["...", ...]
95}
96```
97
98- `$name` - each defined server should have a unique name that can be used in check
99 definitions.
100- `uri` - base URI of this Prometheus server, used for API requests and queries.
101- `timeout` - timeout to be used for API requests.
102- `paths` - optional path filter, if specified only paths matching one of listed regex
103 patterns will use this Prometheus server for checks.
104
105Example:
106
107```JS
108prometheus "prod" {
109 uri = "https://prometheus-prod.example.com"
110 timeout = "60s"
111}
112
113prometheus "dev" {
114 uri = "https://prometheus-dev.example.com"
115 timeout = "30s"
116 paths = [ "alerts/test/.*" ]
117}
118```
119
120## Matching rules to checks
121
122Most checks, except basic syntax verification, requires some configuration to decide
123which checks to run against which files and rules.
124
125Syntax:
126
127```JS
128rule {
129 match {
130 path = "(.+)"
131 name = "(.+)"
132 kind = "alerting|recording"
133 annotation "(.*)" {
134 value = "(.*)"
135 }
136 label "(.*)" {
137 value = "(.*)"
138 }
139 }
140 ignore {
141 path = "(.+)"
142 name = "(.+)"
143 kind = "alerting|recording"
144 annotation "(.*)" {
145 value = "(.*)"
146 }
147 label "(.*)" {
148 value = "(.*)"
149 }
150 }
151
152 [ check definition ]
153 ...
154 [ check definition ]
155}
156```
157
158- `match:path` - only files matching this pattern will be checked by this rule
159- `match:name` - only rules with names (`record` for recording rules and `alert` for alerting
160 rules) matching this pattern will be checked rule
161- `match:kind` - optional rule type filter, only rule of this type will be checked
162- `match:annotation` - optional annotation filter, only alert rules with at least one
163 annotation matching this pattern will be checked by this rule.
164- `match:label` - optional annotation filter, only rules with at least one label
165 matching this pattern will be checked by this rule. For recording rules only static
166 labels set on the recording rule are considered.
167- `ignore` - works exactly like `match` but does the opposite - any alerting or recording rule
168 matching all conditions defined on `ignore` will not be checked by this `rule` block.
169
170Example:
171
172```JS
173rule {
174 match {
175 path = "rules/.*"
176 kind = "alerting"
177 label "severity" {
178 value = "(warning|critical)"
179 }
180 [ check applied only to severity="critical" and severity="warning" alerts ]
181 }
182}
183```
184
185# Check definitions
186
187## Aggregation
188
189This check is used to inspect promql expressions and ensure that specific labels
190are kept or stripped away when aggregating results. It's mostly useful in recording
191rules.
192
193Syntax:
194
195```JS
196aggregate "(.*)" {
197 severity = "bug|warning|info"
198 keep = [ "...", ... ]
199 strip = [ "...", ... ]
200}
201```
202
203- `severity` - set custom severity for reported issues, defaults to a warning
204- `keep` - list of label names that must be preserved
205- `strip` - list of label names that must be stripped
206
207Examples:
208
209Ensure that all series generated from recording rules have `job` labels preserved:
210
211```JS
212rule {
213 match {
214 kind = "recording"
215 }
216 aggregate ".+" {
217 keep = ["job"]
218 }
219}
220```
221
222In some cases you might want to ensure that specific labels are removed in aggregations.
223For example in recording rules that are producing series consumed by federation, where
224only aggregated results (not per instance) are allowed:
225
226```JS
227rule {
228 match {
229 kind = "recording"
230 }
231 aggregate "cluster:.+" {
232 strip = ["instance"]
233 }
234}
235```
236
237By default all issues found by this check will be reported as warnings. To adjust
238severity set a custom `severity` key:
239
240```JS
241aggregate ".+" {
242 ...
243 severity = "bug"
244}
245```
246
247## Annotations
248
249This check is used to ensure that all required annotations are set on alerts and that
250they have correct values.
251
252Syntax:
253
254```JS
255annotation "(.*)" {
256 severity = "bug|warning|info"
257 value = "(.*)"
258 required = true|false
259}
260```
261
262- `severity` - set custom severity for reported issues, defaults to a warning
263- `value` - optional value pattern to enforce
264- `required` - if `true` pint will require every alert to have this annotation set,
265 if `false` it will only check values where annotation is set
266
267Examples:
268
269This set of rules will:
270- require `summary` annotation to be present, if missing it will be reported as a warning
271- if a `dashboard` annotation is provided it must match `https://grafana\.example\.com/.+`
272 pattern, if it doesn't match that pattern it will be reported as a bug
273
274```JS
275rule {
276 match {
277 kind = "alerting"
278 }
279
280 annotation "summary" {
281 required = true
282 }
283
284 annotation "dashboard" {
285 severity = "bug"
286 value = "https://grafana\.example\.com/.+"
287 }
288}
289```
290
291## Labels
292
293This check works the same way as `annotation` check, but it operates on
294labels instead.
295It uses static labels set on alerting or recording rule. It doesn't use
296labels on time series used in those rules.
297
298Syntax:
299
300```JS
301label "(.*)" {
302 severity = "bug|warning|info"
303 value = "..."
304 required = true|false
305}
306```
307
308Example:
309
310Require `severity` label to be set on alert rules with two all possible values:
311
312```JS
313rule {
314 match {
315 kind = "alerting"
316 }
317
318 label "severity" {
319 value = "(warning|critical)"
320 required = true
321 }
322}
323```
324
325## Rate
326
327This check inspects `rate()` and `irate()` functions and warns if used duration
328is too low. It does so by first getting global `scrape_interval` value for selected
329Prometheus servers and comparing duration to it.
330Reported issue depends on a few factors:
331
332For `rate()` function:
333- If duration is less than 2x `scrape_interval` it will report a bug.
334- If duration is between 2x and 4x `scrape_interval` it will report a warning.
335
336For `irate()` function:
337- If duration is less than 2x `scrape_interval` it will report a bug.
338- If duration is between 2x and 3x `scrape_interval` it will report a warning.
339
340This check is enabled by default for all configured Prometheus servers.
341
342Example:
343
344```JS
345prometheus "prod" {
346 uri = "https://prometheus-prod.example.com"
347 timeout = "60s"
348 paths = [
349 "rules/prod/.*",
350 "rules/common/.*",
351 ]
352}
353
354prometheus "dev" {
355 uri = "https://prometheus-dev.example.com"
356 timeout = "30s"
357 paths = [
358 "rules/dev/.*",
359 "rules/common/.*",
360 ]
361}
362```
363
364## Alerts
365
366This check is used to estimate how many times given alert would fire.
367It will run `expr` query from every alert rule against selected Prometheus
368servers and report how many unique alerts it would generate.
369If `for` is set on alerts it will be used to adjust results.
370
371Syntax:
372
373```JS
374alerts {
375 range = "1h"
376 step = "1m"
377 resolve = "5m"
378}
379```
380
381- `range` - query range, how far to look back, `1h` would mean that pint will
382 query last 1h of metrics. If a query results in a timeout pint will retry it
383 with 50% smaller range until it succeeds.
384 Defaults to `1d`.
385- `step` - query resolution, for most accurate result use step equal
386 to `scrape_interval`, try to reduce it if that would load too many samples.
387 Defaults to `1m`.
388- `resolve` - duration after which stale alerts are resolved. Defaults to `5m`.
389
390Example:
391
392```JS
393prometheus "prod" {
394 uri = "https://prometheus-prod.example.com"
395 timeout = "60s"
396}
397
398rule {
399 match {
400 kind = "recording"
401 }
402 alerts {
403 range = "1d"
404 step = "1m"
405 resolve = "5m"
406 }
407}
408```
409
410## Comparison
411
412This check enforces use of a comparison operator in alert queries.
413Since any query result triggers an alert usual query would be something
414like `error_count > 10`, so we only get `error_count` series if the value
415is above 10. If we would remove `> 10` part query would always return `error_count`
416and so it would always trigger an alert.
417
418This check is enabled by default and doesn't require any configuration.
419
420## Cost
421
422This check is used to calculate cost of a query and optionally report an issue
423if that cost is too high. It will run `expr` query from every rule against
424selected Prometheus servers and report results.
425This check can be used for both recording and alerting rules, but is most
426useful for recording rules.
427
428Syntax:
429
430```JS
431cost {
432 severity = "bug|warning|info"
433 bytesPerSample = 1024
434 maxSeries = 5000
435}
436```
437
438- `severity` - set custom severity for reported issues, defaults to a warning.
439 This is only used when query result series exceed `maxSeries` value (if set).
440 If `maxSeries` is not set or when results count is below it pint will still
441 report it as information.
442- `bytesPerSample` - if set results will use this to calculate estimated memory
443 required to store returned series in Prometheus.
444- `maxSeries` - if set and number of results for given query exceeds this value
445 it will be reported as a bug (or custom severity if `severity` is set).
446
447Examples:
448
449All rules from files matching `rules/dev/.+` pattern will be tested against
450`dev` server. Results will be reported as information regardless of results.
451
452```JS
453prometheus "dev" {
454 uri = "https://prometheus-dev.example.com"
455 timeout = "30s"
456 paths = ["rules/dev/.+"]
457}
458
459rule {
460 cost {}
461}
462```
463
464To add memory usage estimate we first need to get average bytes per sample.
465This can be be estimated using two different queries:
466
467- for RSS usage: `process_resident_memory_bytes / prometheus_tsdb_head_series`
468- for Go allocations: `go_memstats_alloc_bytes / prometheus_tsdb_head_series`
469
470Since Go uses garbage collector RSS memory will be more than the sum of all
471memory allocations. RSS usage will be "worst case" while "Go alloc" best case,
472while real memory usage will be somewhere in between, depending on many factors
473like memory pressure, Go version, GOGC settings etc.
474
475```JS
476...
477 cost {
478 bytesPerSample = 4096
479 }
480}
481```
482
483## Series
484
485This check will also query Prometheus servers, it is used to warn about queries
486that are using metrics not currently present in Prometheus.
487It parses `expr` query from every rule, finds individual metric selectors and
488checks if they return any values.
489
490Let's say we have a rule this query: `sum(my_metric{foo="bar"}) > 10`.
491This checks would query all configured server for the existence of
492`my_metric{foo="bar"}` series and report a warning if it's missing.
493
494This check is enabled by default for all configured Prometheus servers.
495
496Example:
497
498```JS
499prometheus "dev" {
500 uri = "https://prometheus-dev.example.com"
501 timeout = "30s"
502}
503
504prometheus "prod" {
505 uri = "https://prometheus-prod.example.com"
506 timeout = "30s"
507}
508```
509
510## Reject
511
512This check allows rejecting label or annotations keys and values
513using regexp rules.
514
515Syntax:
516
517```JS
518reject "(.*)" {
519 severity = "bug|warning|info"
520 label_keys = true|false
521 label_values = true|false
522 annotation_keys = true|false
523 annotation_values = true|false
524}
525```
526
527- `severity` - set custom severity for reported issues, defaults to a bug.
528- `label_keys` - if true label keys for recording and alerting rules will
529 be checked.
530- `label_values` - if true label values for recording and alerting rules will
531 be checked.
532- `annotation_keys` - if true annotation keys for alerting rules will be checked.
533- `annotation_values` - if true label values for alerting rules will be checked.
534
535Example:
536
537Disallow using URLs as label keys or values:
538
539```JS
540rule {
541 match {
542 kind = "alerting"
543 }
544
545 reject "https?://.+" {
546 label_keys = true
547 label_values = true
548 }
549}
550```
551
552Disallow spaces in label and annotation keys:
553
554```JS
555rule {
556 reject ".* +.*" {
557 annotation_keys = true
558 label_keys = true
559 }
560}
561```
562
563## Template
564
565This check validates templating used in annotations and labels for alerting rules.
566See [Prometheus docs](https://prometheus.io/docs/prometheus/latest/configuration/template_reference/)
567for details of supported templating syntax.
568
569This check will also inspect all alert rules and warn if any of them
570uses query return values inside alert labels.
571Two alerts are identical if they have identical labels, so using
572query value will generate a new unique alert every time it changes.
573If alerting rule is using `for` it might prevent it from ever firing
574if the value keeps changing before `for` is satisfied, because
575Prometheus will consider it to be a new alert and start `for` tracking
576from zero.
577
578If you want to include query value in the alert then use annotations
579for that. Annotations are not used to compare alerts identity and so
580the value of any annotation can change between alert evaluations.
581
582See [this blog post](https://www.robustperception.io/dont-put-the-value-in-alert-labels)
583for more details.
584
585This check is enabled by default and doesn't require any configuration.
586
587## Vector Matching
588
589This check will try to find queries that try to
590[match vectors](https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching)
591but have different sets of labels on both side of the query.
592
593Consider these two time series:
594
595```
596http_errors{job="node-exporter", cluster="prod", instance="server1"}
597```
598
599and
600
601```
602cluster:http_errors{job="node-exporter", cluster="prod"}
603```
604
605One of them tracks specific instance and one aggregates series for the whole cluster.
606Because they have different set of labels if we want to calculate some value using both
607of them, for example:
608
609```
610http_errors / cluster:http_errors
611```
612
613we wouldn't get any results. To fix that we need ignore extra labels:
614
615```
616http_errors / ignoring(instance) cluster:http_errors
617```
618
619This check aims to find all queries that using vector matching where both sides
620of the query have different sets of labels causing no results to be returned.
621
622This check is enabled by default for all configured Prometheus servers.
623
624Example:
625
626```JS
627prometheus "dev" {
628 uri = "https://prometheus-dev.example.com"
629 timeout = "30s"
630}
631
632prometheus "prod" {
633 uri = "https://prometheus-prod.example.com"
634 timeout = "30s"
635}
636```
637
638# Ignoring selected lines or files
639
640While parsing files pint will look for special comment blocks and use them to
641exclude some parts all whole files from checks.
642
643## Ignoring whole files
644
645Add a `# pint ignore/file` comment on top of the file, everything below that line
646will be ignored.
647
648Example:
649
650```YAML
651# pint ignore/file
652
653groups:
654 - name: example
655 rules:
656 - record: job:http_inprogress_requests:sum
657 expr: sum by (job) (http_inprogress_requests)
658```
659
660## Ignoring individual lines
661
662To ignore just one line use `# pint ignore/line` at the end of that line or
663`# ignore/next-line` on the line before.
664This is useful if you're linting templates used to generate Prometheus
665configuration and it contains some extra lines that are not valid YAML.
666
667Example:
668
669```YAML
670{% set some_jinja_var1 = "bar" } # pint ignore/line
671groups:
672 - name: example
673 rules:
674 - record: job:http_inprogress_requests:sum
675 expr: sum by (job) (http_inprogress_requests)
676
677# pint ignore/next-line
678{% set some_jinja_var2 = "foo" }
679```
680
681## Ignoring a range of lines
682
683To ignore a part of a file wrap it with `# pint ignore/begin` and
684`# pint ignore/end` comments.
685
686Example:
687
688```YAML
689# pint ignore/begin
690{% set some_jinja_var1 = "bar" }
691{% set some_jinja_var2 = "foo" }
692# pint ignore/end
693
694groups:
695 - name: example
696 rules:
697 - record: job:http_inprogress_requests:sum
698 expr: sum by (job) (http_inprogress_requests)
699```
700
701## Disabling individual checks for specific rules
702
703To disable individual check for a specific rule use `# pint disable ...` comments.
704A single comment can only disable one check, so repeat it for every check you wish
705to disable.
706
707To disable `query/cost` check add `# pint disable query/cost` comment anywhere in
708the rule.
709
710Example:
711
712```YAML
713groups:
714 - name: example
715 rules:
716 - record: instance:http_requests_total:avg_over_time:1w
717 # pint disable query/cost
718 expr: avg_over_time(http_requests_total[1w]) by (instance)
719```
720
721```YAML
722groups:
723 - name: example
724 rules:
725 - record: instance:http_requests_total:avg_over_time:1w
726 expr: avg_over_time(http_requests_total[1w]) by (instance) # pint disable query/cost
727```