cloudflare/pint

Public

mirrored from https://github.com/cloudflare/pintAvailable

CodeCommitsIssuesPull requestsActionsInsightsSecurity
v0.1.4

Branches

Tags

  • No tags available.
0Branches0Tags
Go to file
Add file
Code

Clone

HTTPS

Download ZIP

docs/CONFIGURATION.md

785lines · modecode

1# Configuration syntax
2
3**NOTE** all regex pattern are anchored.
4
5## CI
6
7Configure continuous integration environments.
8
9Syntax:
10
11```JS
12ci {
13 include = [ "(.*)", ... ]
14 maxCommits = 20
15 baseBranch = "master"
16}
17```
18
19- `include` - list of file patters to checks when running checks. Only files
20 matching those regex rules will be checked, other modified files will be ignored.
21- `maxCommits` - by default pint will try to find all commits on current branch,
22 this requires full git history to be present, if we have a shallow clone this
23 might fail to find only current branch commits and give us a huge list.
24 If the number of commits returned by branch discovery is more than `maxCommits`
25 then pint will fail to run.
26- `baseBranch` - base branch to compare `HEAD` commit with when calculating the list
27 of commits to check.
28
29## Repository
30
31Configure supported code hosting repository, used for reporting PR checks from CI
32back to the repository, to be displayed in the PR UI.
33Currently only supports [BitBucket](https://bitbucket.org/) and [GitHub](https://github.com/).
34
35**NOTE**: BitBucket integration requires `BITBUCKET_AUTH_TOKEN` environment variable
36to be set. It should contain a personal access token used to authenticate with the API.
37
38**NOTE**: GitHub integration requires `GITHUB_AUTH_TOKEN` environment variable
39to be set to a personal access key that can access your repository. Also, `GITHUB_PULL_REQUEST_NUMBER`
40environment variable needs to point to the pull request number which will be used whilst
41submitting comments.
42
43Syntax:
44
45```JS
46repository {
47 bitbucket {
48 uri = "https://..."
49 timeout = "30s"
50 project = "..."
51 repository = "..."
52 }
53}
54```
55
56- `bitbucket:uri` - base URI of this repository, will be used for HTTP
57 requests to the BitBucket API.
58- `bitbucket:timeout` - timeout to be used for API requests.
59- `bitbucket:project` - name of the BitBucket project for this repository.
60- `bitbucket:repository` - name of the BitBucket repository.
61
62```JS
63repository {
64 github {
65 uri = "https://..."
66 timeout = "30s"
67 owner = "..."
68 repo = "..."
69 }
70}
71```
72
73- `github:baseuri` - base URI of GitHub or GitHub enterprise, will be used for HTTP requests to the GitHub API.
74- `github:uploaduri` - upload URI of GitHub or GitHub enterprise, will be used for HTTP requests to the GitHub API.
75
76If `github:baseuri` _or_ `github:uploaduri` are not specified then [GitHub](https://github.com) will be used.
77
78- `github:timeout` - timeout to be used for API requests;
79- `github:owner` - name of the GitHub owner i.e. the first part that comes before the repository's name in the URI;
80- `github:repo` - name of the GitHub repository (e.g. `monitoring`).
81
82## Prometheus servers
83
84Some checks work by querying a running Prometheus instance to verify if
85metrics used in rules are present. If you want to use those checks then you
86first need to define one or more Prometheus servers.
87
88Syntax:
89
90```JS
91prometheus "$name" {
92 uri = "https://..."
93 timeout = "60s"
94 paths = ["...", ...]
95}
96```
97
98- `$name` - each defined server should have a unique name that can be used in check
99 definitions.
100- `uri` - base URI of this Prometheus server, used for API requests and queries.
101- `timeout` - timeout to be used for API requests.
102- `paths` - optional path filter, if specified only paths matching one of listed regex
103 patterns will use this Prometheus server for checks.
104
105Example:
106
107```JS
108prometheus "prod" {
109 uri = "https://prometheus-prod.example.com"
110 timeout = "60s"
111}
112
113prometheus "dev" {
114 uri = "https://prometheus-dev.example.com"
115 timeout = "30s"
116 paths = [ "alerts/test/.*" ]
117}
118```
119
120## Matching rules to checks
121
122Most checks, except basic syntax verification, requires some configuration to decide
123which checks to run against which files and rules.
124
125Syntax:
126
127```JS
128rule {
129 match {
130 path = "..."
131 kind = "alerting|recording"
132 annotation "(.*)" {
133 value = "(.*)"
134 }
135 label "(.*)" {
136 value = "(.*)"
137 }
138 }
139
140 [ check definition ]
141 ...
142 [ check definition ]
143}
144```
145
146- `match:path` - only files matching this pattern will be checked by this rule
147- `match:kind` - optional rule type filter, only rule of this type will be checked
148- `match:annotation` - optional annotation filter, only alert rules with at least one
149 annotation matching this pattern will be checked by this rule.
150- `match:label` - optional annotation filter, only rules with at least one label
151 matching this pattern will be checked by this rule. For recording rules only static
152 labels set on the recording rule are considered.
153
154Example:
155
156```JS
157rule {
158 match {
159 path = "rules/.*"
160 kind = "alerting"
161 label "severity" {
162 value = "(warning|critical)"
163 }
164 [ check applied only to severity="critical" and severity="warning" alerts ]
165 }
166}
167```
168
169# Check definitions
170
171## Aggregation
172
173This check is used to inspect promql expressions and ensure that specific labels
174are kept or stripped away when aggregating results. It's mostly useful in recording
175rules.
176
177Syntax:
178
179```JS
180aggregate "(.*)" {
181 severity = "bug|warning|info"
182 keep = [ "...", ... ]
183 strip = [ "...", ... ]
184}
185```
186
187- `severity` - set custom severity for reported issues, defaults to a warning
188- `keep` - list of label names that must be preserved
189- `strip` - list of label names that must be stripped
190
191Examples:
192
193Ensure that all series generated from recording rules have `job` labels preserved:
194
195```JS
196rule {
197 match {
198 kind = "recording"
199 }
200 aggregate ".+" {
201 keep = ["job"]
202 }
203}
204```
205
206In some cases you might want to ensure that specific labels are removed in aggregations.
207For example in recording rules that are producing series consumed by federation, where
208only aggregated results (not per instance) are allowed:
209
210```JS
211rule {
212 match {
213 kind = "recording"
214 }
215 aggregate "cluster:.+" {
216 strip = ["instance"]
217 }
218}
219```
220
221By default all issues found by this check will be reported as warnings. To adjust
222severity set a custom `severity` key:
223
224```JS
225aggregate ".+" {
226 ...
227 severity = "bug"
228}
229```
230
231## Annotations
232
233This check is used to ensure that all required annotations are set on alerts and that
234they have correct values.
235
236Syntax:
237
238```JS
239annotation "(.*)" {
240 severity = "bug|warning|info"
241 value = "(.*)"
242 required = true|false
243}
244```
245
246- `severity` - set custom severity for reported issues, defaults to a warning
247- `value` - optional value pattern to enforce
248- `required` - if `true` pint will require every alert to have this annotation set,
249 if `false` it will only check values where annotation is set
250
251Examples:
252
253This set of rules will:
254- require `summary` annotation to be present, if missing it will be reported as a warning
255- if a `dashboard` annotation is provided it must match `https://grafana\.example\.com/.+`
256 pattern, if it doesn't match that pattern it will be reported as a bug
257
258```JS
259rule {
260 match {
261 kind = "alerting"
262 }
263
264 annotation "summary" {
265 required = true
266 }
267
268 annotation "dashboard" {
269 severity = "bug"
270 value = "https://grafana\.example\.com/.+"
271 }
272}
273```
274
275## Labels
276
277This check works the same way as `annotation` check, but it operates on
278labels instead.
279It uses static labels set on alerting or recording rule. It doesn't use
280labels on time series used in those rules.
281
282Syntax:
283
284```JS
285label "(.*)" {
286 severity = "bug|warning|info"
287 value = "..."
288 required = true|false
289}
290```
291
292Example:
293
294Require `severity` label to be set on alert rules with two all possible values:
295
296```JS
297rule {
298 match {
299 kind = "alerting"
300 }
301
302 label "severity" {
303 value = "(warning|critical)"
304 required = true
305 }
306}
307```
308
309## Rate
310
311This check inspects `rate()` and `irate()` functions and warns if used duration
312is too low. It does so by first getting global `scrape_interval` value for selected
313Prometheus servers and comparing duration to it.
314Reported issue depends on a few factors:
315
316For `rate()` function:
317- If duration is less than 2x `scrape_interval` it will report a bug.
318- If duration is between 2x and 4x `scrape_interval` it will report a warning.
319
320For `irate()` function:
321- If duration is less than 2x `scrape_interval` it will report a bug.
322- If duration is between 2x and 3x `scrape_interval` it will report a warning.
323
324Syntax:
325
326```JS
327rate {}
328```
329
330Example:
331
332```JS
333prometheus "prod" {
334 uri = "https://prometheus-prod.example.com"
335 timeout = "60s"
336}
337
338prometheus "dev" {
339 uri = "https://prometheus-dev.example.com"
340 timeout = "30s"
341}
342
343rule {
344 match {
345 kind = "recording"
346 }
347
348 rate {}
349}
350```
351
352## Alerts
353
354This check is used to estimate how many times given alert would fire.
355It will run `expr` query from every alert rule against selected Prometheus
356servers and report how many unique alerts it would generate.
357If `for` is set on alerts it will be used to adjust results.
358
359Syntax:
360
361```JS
362alerts {
363 range = "1h"
364 step = "1m"
365 resolve = "5m"
366}
367```
368
369- `range` - query range, how far to look back, `1h` would mean that pint will
370 query last 1h of metrics. If a query results in a timeout pint will retry it
371 with 50% smaller range until it succeeds.
372 Defaults to `1d`.
373- `step` - query resolution, for most accurate result use step equal
374 to `scrape_interval`, try to reduce it if that would load too many samples.
375 Defaults to `1m`.
376- `resolve` - duration after which stale alerts are resolved. Defaults to `5m`.
377
378Example:
379
380```JS
381prometheus "prod" {
382 uri = "https://prometheus-prod.example.com"
383 timeout = "60s"
384}
385
386rule {
387 match {
388 kind = "recording"
389 }
390 alerts {
391 range = "1d"
392 step = "1m"
393 resolve = "5m"
394 }
395}
396```
397
398## Comparison
399
400This check enforces use of a comparison operator in alert queries.
401Since any query result triggers an alert usual query would be something
402like `error_count > 10`, so we only get `error_count` series if the value
403is above 10. If we would remove `> 10` part query would always return `error_count`
404and so it would always trigger an alert.
405
406Syntax:
407
408```JS
409comparison {
410 severity = "bug|warning|info"
411}
412```
413
414- `severity` - set custom severity for reported issues, defaults to a bug.
415
416Example:
417
418```
419rule {
420 match {
421 kind = "alerting"
422 }
423 comparison {}
424}
425```
426
427
428## Cost
429
430This check is used to calculate cost of a query and optionally report an issue
431if that cost is too high. It will run `expr` query from every rule against
432selected Prometheus servers and report results.
433This check can be used for both recording and alerting rules, but is most
434useful for recording rules.
435
436Syntax:
437
438```JS
439cost {
440 severity = "bug|warning|info"
441 bytesPerSample = 1024
442 maxSeries = 5000
443}
444```
445
446- `severity` - set custom severity for reported issues, defaults to a warning.
447 This is only used when query result series exceed `maxSeries` value (if set).
448 If `maxSeries` is not set or when results count is below it pint will still
449 report it as information.
450- `bytesPerSample` - if set results will use this to calculate estimated memory
451 required to store returned series in Prometheus.
452- `maxSeries` - if set and number of results for given query exceeds this value
453 it will be reported as a bug (or custom severity if `severity` is set).
454
455Examples:
456
457All rules from files matching `rules/dev/.+` pattern will be tested against
458`dev` server. Results will be reported as information regardless of results.
459
460```JS
461prometheus "dev" {
462 uri = "https://prometheus-dev.example.com"
463 timeout = "30s"
464 paths = ["rules/dev/.+"]
465}
466
467rule {
468 cost {}
469}
470```
471
472To add memory usage estimate we first need to get average bytes per sample.
473This can be be estimated using two different queries:
474
475- for RSS usage: `process_resident_memory_bytes / prometheus_tsdb_head_series`
476- for Go allocations: `go_memstats_alloc_bytes / prometheus_tsdb_head_series`
477
478Since Go uses garbage collector RSS memory will be more than the sum of all
479memory allocations. RSS usage will be "worst case" while "Go alloc" best case,
480while real memory usage will be somewhere in between, depending on many factors
481like memory pressure, Go version, GOGC settings etc.
482
483```JS
484...
485 cost {
486 bytesPerSample = 4096
487 }
488}
489```
490
491## Series
492
493This check will also query Prometheus servers, it is used to warn about queries
494that are using metrics not currently present in Prometheus.
495It parses `expr` query from every rule, finds individual metric selectors and
496checks if they return any values.
497
498Let's say we have a rule this query: `sum(my_metric{foo="bar"}) > 10`.
499This checks would query all configured server for the existence of
500`my_metric{foo="bar"}` series and report a warning if it's missing.
501
502Syntax:
503
504```JS
505series {
506 severity = "bug|warning|info"
507}
508```
509
510- `severity` - set custom severity for reported issues, defaults to a warning.
511
512Example:
513
514```JS
515prometheus "dev" {
516 uri = "https://prometheus-dev.example.com"
517 timeout = "30s"
518}
519
520prometheus "prod" {
521 uri = "https://prometheus-prod.example.com"
522 timeout = "30s"
523}
524
525rule {
526 match {
527 kind = "recording"
528 }
529
530 series {}
531}
532```
533
534## Reject
535
536This check allows rejecting label or annotations keys and values
537using regexp rules.
538
539Syntax:
540
541```JS
542reject "(.*)" {
543 severity = "bug|warning|info"
544 label_keys = true|false
545 label_values = true|false
546 annotation_keys = true|false
547 annotation_values = true|false
548}
549```
550
551- `severity` - set custom severity for reported issues, defaults to a bug.
552- `label_keys` - if true label keys for recording and alerting rules will
553 be checked.
554- `label_values` - if true label values for recording and alerting rules will
555 be checked.
556- `annotation_keys` - if true annotation keys for alerting rules will be checked.
557- `annotation_values` - if true label values for alerting rules will be checked.
558
559Example:
560
561Disallow using URLs as label keys or values:
562
563```JS
564rule {
565 match {
566 kind = "alerting"
567 }
568
569 reject "https?://.+" {
570 label_keys = true
571 label_values = true
572 }
573}
574```
575
576Disallow spaces in label and annotation keys:
577
578```JS
579rule {
580 reject ".* +.*" {
581 annotation_keys = true
582 label_keys = true
583 }
584}
585```
586
587## Template
588
589This check validates templating used in annotations and labels for alerting rules.
590See [Prometheus docs](https://prometheus.io/docs/prometheus/latest/configuration/template_reference/)
591for details of supported templating syntax.
592
593This check will also inspect all alert rules and warn if any of them
594uses query return values inside alert labels.
595Two alerts are identical if they have identical labels, so using
596query value will generate a new unique alert every time it changes.
597If alerting rule is using `for` it might prevent it from ever firing
598if the value keeps changing before `for` is satisfied, because
599Prometheus will consider it to be a new alert and start `for` tracking
600from zero.
601
602If you want to include query value in the alert then use annotations
603for that. Annotations are not used to compare alerts identity and so
604the value of any annotation can change between alert evaluations.
605
606See [this blog post](https://www.robustperception.io/dont-put-the-value-in-alert-labels)
607for more details.
608
609Syntax:
610
611```JS
612template {
613 severity = "bug|warning|info"
614}
615```
616
617- `severity` - set custom severity for reported issues, defaults to a bug.
618
619Example:
620
621```JS
622rule {
623 match {
624 kind = "alerting"
625 }
626
627 template {
628 severity = "fatal"
629 }
630}
631```
632
633## Vector Matching
634
635This check will try to find queries that try to
636[match vectors](https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching)
637but have different sets of labels on both side of the query.
638
639Consider these two time series:
640
641```
642http_errors{job="node-exporter", cluster="prod", instance="server1"}
643```
644
645and
646
647```
648cluster:http_errors{job="node-exporter", cluster="prod"}
649```
650
651One of them tracks specific instance and one aggregates series for the whole cluster.
652Because they have different set of labels if we want to calculate some value using both
653of them, for example:
654
655```
656http_errors / cluster:http_errors
657```
658
659we wouldn't get any results. To fix that we need ignore extra labels:
660
661```
662http_errors / ignoring(instance) cluster:http_errors
663```
664
665This check aims to find all queries that using vector matching where both sides
666of the query have different sets of labels causing no results to be returned.
667
668Syntax:
669
670```JS
671vector_matching {
672 severity = "bug|warning|info"
673}
674```
675
676- `severity` - set custom severity for reported issues, defaults to a warning.
677
678Example:
679
680```JS
681prometheus "dev" {
682 uri = "https://prometheus-dev.example.com"
683 timeout = "30s"
684}
685
686prometheus "prod" {
687 uri = "https://prometheus-prod.example.com"
688 timeout = "30s"
689}
690
691rule {
692 vector_matching {}
693}
694```
695
696# Ignoring selected lines or files
697
698While parsing files pint will look for special comment blocks and use them to
699exclude some parts all whole files from checks.
700
701## Ignoring whole files
702
703Add a `# pint ignore/file` comment on top of the file, everything below that line
704will be ignored.
705
706Example:
707
708```YAML
709# pint ignore/file
710
711groups:
712 - name: example
713 rules:
714 - record: job:http_inprogress_requests:sum
715 expr: sum by (job) (http_inprogress_requests)
716```
717
718## Ignoring individual lines
719
720To ignore just one line use `# pint ignore/line` at the end of that line or
721`# ignore/next-line` on the line before.
722This is useful if you're linting templates used to generate Prometheus
723configuration and it contains some extra lines that are not valid YAML.
724
725Example:
726
727```YAML
728{% set some_jinja_var1 = "bar" } # pint ignore/line
729groups:
730 - name: example
731 rules:
732 - record: job:http_inprogress_requests:sum
733 expr: sum by (job) (http_inprogress_requests)
734
735# pint ignore/next-line
736{% set some_jinja_var2 = "foo" }
737```
738
739## Ignoring a range of lines
740
741To ignore a part of a file wrap it with `# pint ignore/begin` and
742`# pint ignore/end` comments.
743
744Example:
745
746```YAML
747# pint ignore/begin
748{% set some_jinja_var1 = "bar" }
749{% set some_jinja_var2 = "foo" }
750# pint ignore/end
751
752groups:
753 - name: example
754 rules:
755 - record: job:http_inprogress_requests:sum
756 expr: sum by (job) (http_inprogress_requests)
757```
758
759## Disabling individual checks for specific rules
760
761To disable individual check for a specific rule use `# pint disable ...` comments.
762A single comment can only disable one check, so repeat it for every check you wish
763to disable.
764
765To disable `query/cost` check add `# pint disable query/cost` comment anywhere in
766the rule.
767
768Example:
769
770```YAML
771groups:
772 - name: example
773 rules:
774 - record: instance:http_requests_total:avg_over_time:1w
775 # pint disable query/cost
776 expr: avg_over_time(http_requests_total[1w]) by (instance)
777```
778
779```YAML
780groups:
781 - name: example
782 rules:
783 - record: instance:http_requests_total:avg_over_time:1w
784 expr: avg_over_time(http_requests_total[1w]) by (instance) # pint disable query/cost
785```
786