GitVita
New repositoryImport repositoryNew issue
Your repositories Your dashboard Sign in

cloudflare/pint

Public

mirrored from https://github.com/cloudflare/pintAvailable

Watch0Fork0Star0
CodeCommitsIssuesPull requestsActionsInsightsSecurity
v0.74.7

Branches

Tags

  • No tags available.
0Branches0Tags
Go to file
Add file
Code

Clone

HTTPS

Download ZIP

pint/docs/checks/query

docs/checks/query/cost.md

223lines · modecode

RawDownload
Latest commit unavailable.
1---
2layout: default
3parent: Checks
4grand_parent: Documentation
5---
6
7# query/cost
8
9This check is used to calculate cost of a query and optionally report an issue
10if that cost is too high. It will run `expr` query from every rule against
11selected Prometheus servers and report results.
12This check can be used for both recording and alerting rules, but is mostly
13useful for recording rules.
14
15## Query evaluation duration
16
17The total duration of a query comes from Prometheus query stats included
18in the API response when `?stats=1` is passed.
19When enabled pint can report if `evalTotalTime` is higher than configured limit,
20which can be used either for informational purpose or to fail checks on queries
21that are too expensive (depending on configured `severity`).
22
23## Query evaluation samples
24
25Similar to evaluation duration this information comes from Prometheus query stats.
26There are two different stats that give us information about the number of samples
27used by given query:
28
29- `totalQueryableSamples` - the total number of samples read during the query execution.
30- `peakSamples` - the max samples kept in memory during the query execution and shows
31how close the query was to reach the `--query.max-samples`` limit.
32
33In general higher `totalQueryableSamples` means that a query either reads a lot of
34time series and/or queries a large time range, both translating into longer query
35execution times.
36Looking at `peakSamples` on the other hand can be useful to find queries that are
37complex and perform some operation on a large number of time series, for example
38when you run `max(...)` on a query that returns a huge number of results.
39
40## Series returned by the query
41
42For recording rules anything returned by the query will be saved into Prometheus
43as new time series. Checking how many time series does a rule return allows us
44to estimate how much extra memory will be needed.
45`pint` will try to estimate the number of bytes needed per single time series
46and use that to estimate the amount of memory needed to store all the time series
47returned by given query.
48The `bytes per time series` number is calculated using this query:
49
50```js
51avg(avg_over_time(go_memstats_alloc_bytes[2h]) / avg_over_time(prometheus_tsdb_head_series[2h]))
52```
53
54Since Go uses garbage collector total Prometheus process memory will be more than the
55sum of all memory allocations, depending on many factors like memory pressure,
56Go version, `GOGC` settings etc. The estimate `pint` gives you should be considered
57`best case` scenario.
58
59## Optimization suggestions
60
61`query/cost` check will try to find rules using queries that are being precomputed using recording rules.
62Consider these rules:
63
64```yaml
65- record: foo:rate5m
66 expr: rate(foo_total[5m])
67
68- alert: Rate Too High
69 expr: sum(rate(foo_total[5m])) without(instance) > 10
70```
71
72Here we have an alert `Rate Too High` that uses `rate(foo_total[5m])` as part of the query.
73We also have a recording rule `foo:rate5m` that calculates the same expression and stores it
74as a metric.
75Instead of calculating `rate(foo_total[5m])` in both rules we can simply query `foo:rate5m` inside
76`Rate Too High` alert to speed it up:
77
78```yaml
79- alert: Rate Too High
80 expr: sum(foo:rate5m) without(instance) > 10
81```
82
83This check will try to find cases like this and emit an information report for it.
84
85## Configuration
86
87Syntax:
88
89```js
90cost {
91 comment = "..."
92 severity = "bug|warning|info"
93 maxSeries = 5000
94 maxPeakSamples = 10000
95 maxTotalSamples = 200000
96 maxEvaluationDuration = "1m"
97}
98```
99
100- `comment` - set a custom comment that will be added to reported problems.
101- `severity` - set custom severity for reported issues, defaults to a warning.
102 This is only used when query result series exceed `maxSeries` value (if set).
103 If `maxSeries` is not set or when results count is below it pint will still
104 report it as information.
105- `maxSeries` - if set and number of results for given query exceeds this value
106 it will be reported as a bug (or custom severity if `severity` is set).
107- `maxPeakSamples` - setting this to a non-zero value will tell pint to report
108 any query that has higher `peakSamples` values than the value configured here.
109 Nothing will be reported if this option is not set.
110- `maxTotalSamples` - setting this to a non-zero value will tell pint to report
111 any query that has higher `totalQueryableSamples` values than the value
112 configured here. Nothing will be reported if this option is not set.
113- `maxEvaluationDuration` - setting this to a non-zero value will tell pint to
114 report any query that has higher `evalTotalTime` values than the value
115 configured here. Nothing will be reported if this option is not set.
116
117## How to enable it
118
119This check is not enabled by default as it requires explicit configuration
120to work.
121To enable it add one or more `prometheus {...}` blocks and a `rule {...}` block
122with this checks config.
123
124Examples:
125
126All rules from files matching `rules/dev/.+` pattern will be tested against
127`dev` server. Results will be reported as information regardless of results.
128
129```js
130prometheus "dev" {
131 uri = "https://prometheus-dev.example.com"
132 timeout = "30s"
133 include = ["rules/dev/.+"]
134}
135
136rule {
137 cost {}
138}
139```
140
141Fail checks if any recording rule is using more than 300000 peak samples
142or if it's taking more than 30 seconds to evaluate.
143
144```js
145rule {
146 match {
147 kind = "recording"
148 }
149 cost {
150 maxPeakSamples = 300000
151 maxEvaluationDuration = "30s"
152 severity = "bug"
153 comment = "This query is too expensive to run"
154 }
155}
156```
157
158## How to disable it
159
160You can disable this check globally by adding this config block:
161
162```js
163checks {
164 disabled = ["query/cost"]
165}
166```
167
168You can also disable it for all rules inside given file by adding
169a comment anywhere in that file. Example:
170
171```yaml
172# pint file/disable query/cost
173```
174
175Or you can disable it per rule by adding a comment to it. Example:
176
177```yaml
178# pint disable query/cost
179```
180
181If you want to disable only individual instances of this check
182you can add a more specific comment.
183
184### If `maxSeries` is set
185
186```yaml
187# pint disable query/cost($prometheus:$maxSeries)
188```
189
190Where `$prometheus` is the name of Prometheus server to disable.
191
192Example:
193
194```yaml
195# pint disable query/cost(dev:5000)
196```
197
198### If `maxSeries` is NOT set
199
200```yaml
201# pint disable query/cost($prometheus)
202```
203
204Where `$prometheus` is the name of Prometheus server to disable.
205
206Example:
207
208```yaml
209# pint disable query/cost(dev)
210```
211
212## How to snooze it
213
214You can disable this check until given time by adding a comment to it. Example:
215
216```yaml
217# pint snooze $TIMESTAMP query/cost
218```
219
220Where `$TIMESTAMP` is either use [RFC3339](https://www.rfc-editor.org/rfc/rfc3339)
221formatted or `YYYY-MM-DD`.
222Adding this comment will disable `query/cost` *until* `$TIMESTAMP`, after that
223check will be re-enabled.
224