cloudflare/pint
Publicmirrored from https://github.com/cloudflare/pintAvailable
docs/checks/query/cost.md
197lines · modecode
| 1 | --- |
| 2 | layout: default |
| 3 | parent: Checks |
| 4 | grand_parent: Documentation |
| 5 | --- |
| 6 | |
| 7 | # query/cost |
| 8 | |
| 9 | This check is used to calculate cost of a query and optionally report an issue |
| 10 | if that cost is too high. It will run `expr` query from every rule against |
| 11 | selected Prometheus servers and report results. |
| 12 | This check can be used for both recording and alerting rules, but is mostly |
| 13 | useful for recording rules. |
| 14 | |
| 15 | ## Query evaluation duration |
| 16 | |
| 17 | The total duration of a query comes from Prometheus query stats included |
| 18 | in the API response when `?stats=1` is passed. |
| 19 | When enabled pint can report if `evalTotalTime` is higher than configured limit, |
| 20 | which can be used either for informational purpose or to fail checks on queries |
| 21 | that are too expensive (depending on configured `severity`). |
| 22 | |
| 23 | ## Query evaluation samples |
| 24 | |
| 25 | Similar to evaluation duration this information comes from Prometheus query stats. |
| 26 | There are two different stats that give us information about the number of samples |
| 27 | used by given query: |
| 28 | |
| 29 | - `totalQueryableSamples` - the total number of samples read during the query execution. |
| 30 | - `peakSamples` - the max samples kept in memory during the query execution and shows |
| 31 | how close the query was to reach the `--query.max-samples`` limit. |
| 32 | |
| 33 | In general higher `totalQueryableSamples` means that a query either reads a lot of |
| 34 | time series and/or queries a large time range, both translating into longer query |
| 35 | execution times. |
| 36 | Looking at `peakSamples` on the other hand can be useful to find queries that are |
| 37 | complex and perform some operation on a large number of time series, for example |
| 38 | when you run `max(...)` on a query that returns a huge number of results. |
| 39 | |
| 40 | ## Series returned by the query |
| 41 | |
| 42 | For recording rules anything returned by the query will be saved into Prometheus |
| 43 | as new time series. Checking how many time series does a rule return allows us |
| 44 | to estimate how much extra memory will be needed. |
| 45 | `pint` will try to estimate the number of bytes needed per single time series |
| 46 | and use that to estimate the amount of memory needed to store all the time series |
| 47 | returned by given query. |
| 48 | The `bytes per time series` number is calculated using this query: |
| 49 | |
| 50 | ```js |
| 51 | avg(avg_over_time(go_memstats_alloc_bytes[2h]) / avg_over_time(prometheus_tsdb_head_series[2h])) |
| 52 | ``` |
| 53 | |
| 54 | Since Go uses garbage collector total Prometheus process memory will be more than the |
| 55 | sum of all memory allocations, depending on many factors like memory pressure, |
| 56 | Go version, `GOGC` settings etc. The estimate `pint` gives you should be considered |
| 57 | `best case` scenario. |
| 58 | |
| 59 | ## Configuration |
| 60 | |
| 61 | Syntax: |
| 62 | |
| 63 | ```js |
| 64 | cost { |
| 65 | comment = "..." |
| 66 | severity = "bug|warning|info" |
| 67 | maxSeries = 5000 |
| 68 | maxPeakSamples = 10000 |
| 69 | maxTotalSamples = 200000 |
| 70 | maxEvaluationDuration = "1m" |
| 71 | } |
| 72 | ``` |
| 73 | |
| 74 | - `comment` - set a custom comment that will be added to reported problems. |
| 75 | - `severity` - set custom severity for reported issues, defaults to a warning. |
| 76 | This is only used when query result series exceed `maxSeries` value (if set). |
| 77 | If `maxSeries` is not set or when results count is below it pint will still |
| 78 | report it as information. |
| 79 | - `maxSeries` - if set and number of results for given query exceeds this value |
| 80 | it will be reported as a bug (or custom severity if `severity` is set). |
| 81 | - `maxPeakSamples` - setting this to a non-zero value will tell pint to report |
| 82 | any query that has higher `peakSamples` values than the value configured here. |
| 83 | Nothing will be reported if this option is not set. |
| 84 | - `maxTotalSamples` - setting this to a non-zero value will tell pint to report |
| 85 | any query that has higher `totalQueryableSamples` values than the value |
| 86 | configured here. Nothing will be reported if this option is not set. |
| 87 | - `maxEvaluationDuration` - setting this to a non-zero value will tell pint to |
| 88 | report any query that has higher `evalTotalTime` values than the value |
| 89 | configured here. Nothing will be reported if this option is not set. |
| 90 | |
| 91 | ## How to enable it |
| 92 | |
| 93 | This check is not enabled by default as it requires explicit configuration |
| 94 | to work. |
| 95 | To enable it add one or more `prometheus {...}` blocks and a `rule {...}` block |
| 96 | with this checks config. |
| 97 | |
| 98 | Examples: |
| 99 | |
| 100 | All rules from files matching `rules/dev/.+` pattern will be tested against |
| 101 | `dev` server. Results will be reported as information regardless of results. |
| 102 | |
| 103 | ```js |
| 104 | prometheus "dev" { |
| 105 | uri = "https://prometheus-dev.example.com" |
| 106 | timeout = "30s" |
| 107 | include = ["rules/dev/.+"] |
| 108 | } |
| 109 | |
| 110 | rule { |
| 111 | cost {} |
| 112 | } |
| 113 | ``` |
| 114 | |
| 115 | Fail checks if any recording rule is using more than 300000 peak samples |
| 116 | or if it's taking more than 30 seconds to evaluate. |
| 117 | |
| 118 | ```js |
| 119 | rule { |
| 120 | match { |
| 121 | kind = "recording" |
| 122 | } |
| 123 | cost { |
| 124 | maxPeakSamples = 300000 |
| 125 | maxEvaluationDuration = "30s" |
| 126 | severity = "bug" |
| 127 | comment = "This query is too expensive to run" |
| 128 | } |
| 129 | } |
| 130 | ``` |
| 131 | |
| 132 | ## How to disable it |
| 133 | |
| 134 | You can disable this check globally by adding this config block: |
| 135 | |
| 136 | ```js |
| 137 | checks { |
| 138 | disabled = ["query/cost"] |
| 139 | } |
| 140 | ``` |
| 141 | |
| 142 | You can also disable it for all rules inside given file by adding |
| 143 | a comment anywhere in that file. Example: |
| 144 | |
| 145 | ```yaml |
| 146 | # pint file/disable query/cost |
| 147 | ``` |
| 148 | |
| 149 | Or you can disable it per rule by adding a comment to it. Example: |
| 150 | |
| 151 | ```yaml |
| 152 | # pint disable query/cost |
| 153 | ``` |
| 154 | |
| 155 | If you want to disable only individual instances of this check |
| 156 | you can add a more specific comment. |
| 157 | |
| 158 | ### If `maxSeries` is set |
| 159 | |
| 160 | ```yaml |
| 161 | # pint disable query/cost($prometheus:$maxSeries) |
| 162 | ``` |
| 163 | |
| 164 | Where `$prometheus` is the name of Prometheus server to disable. |
| 165 | |
| 166 | Example: |
| 167 | |
| 168 | ```yaml |
| 169 | # pint disable query/cost(dev:5000) |
| 170 | ``` |
| 171 | |
| 172 | ### If `maxSeries` is NOT set |
| 173 | |
| 174 | ```yaml |
| 175 | # pint disable query/cost($prometheus) |
| 176 | ``` |
| 177 | |
| 178 | Where `$prometheus` is the name of Prometheus server to disable. |
| 179 | |
| 180 | Example: |
| 181 | |
| 182 | ```yaml |
| 183 | # pint disable query/cost(dev) |
| 184 | ``` |
| 185 | |
| 186 | ## How to snooze it |
| 187 | |
| 188 | You can disable this check until given time by adding a comment to it. Example: |
| 189 | |
| 190 | ```yaml |
| 191 | # pint snooze $TIMESTAMP query/cost |
| 192 | ``` |
| 193 | |
| 194 | Where `$TIMESTAMP` is either use [RFC3339](https://www.rfc-editor.org/rfc/rfc3339) |
| 195 | formatted or `YYYY-MM-DD`. |
| 196 | Adding this comment will disable `query/cost` *until* `$TIMESTAMP`, after that |
| 197 | check will be re-enabled. |
| 198 | |