cloudflare/pint
Publicmirrored from https://github.com/cloudflare/pintAvailable
docs/checks/promql/rate.md
131lines · modecode
| 1 | --- |
| 2 | layout: default |
| 3 | parent: Checks |
| 4 | grand_parent: Documentation |
| 5 | --- |
| 6 | |
| 7 | # promql/rate |
| 8 | |
| 9 | This check inspects `rate()` and `irate()` function calls used in queries |
| 10 | to verify that: |
| 11 | |
| 12 | - [Range queries](https://prometheus.io/docs/prometheus/latest/querying/basics/#range-vector-selectors) |
| 13 | are using a valid time duration. |
| 14 | This is done by first getting global `scrape_interval` value for selected |
| 15 | Prometheus servers and comparing duration to it. |
| 16 | It will report a bug if duration is less than 2x `scrape_interval` because |
| 17 | Prometheus must have at least two samples to be able to calculate rate, so |
| 18 | the time range used in queries must be at least 2x `scrape_interval` value. |
| 19 | - Metrics passed to `rate()` and `irate()` are counters. |
| 20 | Both functions only work with counters and, although any metric type can be |
| 21 | passed to it and will return calculated value, using a non-counter will cause |
| 22 | problems. This is because counters are only allowed to increase in value and any |
| 23 | value drop is interpreted as counter overflow. |
| 24 | For gauge metrics use [`delta()`](https://prometheus.io/docs/prometheus/latest/querying/functions/#delta) |
| 25 | or [`deriv()`](https://prometheus.io/docs/prometheus/latest/querying/functions/#deriv) |
| 26 | functions instead. |
| 27 | - `rate()` is never called on result of `sum(counter)` since that will always return |
| 28 | invalid results. |
| 29 | Chaining `rate(sum(...))` is only possible when passing a metric produced via recording rules |
| 30 | to `rate()` and so pint will try to find such chains. |
| 31 | See [this blog post](https://www.robustperception.io/rate-then-sum-never-sum-then-rate/) |
| 32 | for details. |
| 33 | |
| 34 | ## Common problems |
| 35 | |
| 36 | ### Metadata mismatch |
| 37 | |
| 38 | Metric type checks are using |
| 39 | [metadata API](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata). |
| 40 | Metadata is aggregated from all scraped metrics. |
| 41 | |
| 42 | This can cause a few potential problems: |
| 43 | |
| 44 | - You might have the same metric reported with multiple different types and Prometheus or pint won't know |
| 45 | which time series is which type, because all we have to match a metric to a type is its name. |
| 46 | Best solution here is to never export same name as multiple metrics with different types. |
| 47 | - If you change the typo of some exported metric then the old type will still show up in metadata, |
| 48 | plus the new one, as long as there's at least one target still exporting old metric type. |
| 49 | If you accidentally exported some metric with wrong type, then fixed it, but pint is still complaining, |
| 50 | then it's very likely that you didn't release your fix to all targets yet. |
| 51 | |
| 52 | ## Configuration |
| 53 | |
| 54 | This check doesn't have any configuration options. |
| 55 | |
| 56 | ## How to enable it |
| 57 | |
| 58 | This check is enabled by default for all configured Prometheus servers. |
| 59 | |
| 60 | Example: |
| 61 | |
| 62 | ```js |
| 63 | prometheus "prod" { |
| 64 | uri = "https://prometheus-prod.example.com" |
| 65 | timeout = "60s" |
| 66 | include = [ |
| 67 | "rules/prod/.*", |
| 68 | "rules/common/.*", |
| 69 | ] |
| 70 | } |
| 71 | |
| 72 | prometheus "dev" { |
| 73 | uri = "https://prometheus-dev.example.com" |
| 74 | timeout = "30s" |
| 75 | include = [ |
| 76 | "rules/dev/.*", |
| 77 | "rules/common/.*", |
| 78 | ] |
| 79 | } |
| 80 | ``` |
| 81 | |
| 82 | ## How to disable it |
| 83 | |
| 84 | You can disable this check globally by adding this config block: |
| 85 | |
| 86 | ```js |
| 87 | checks { |
| 88 | disabled = ["promql/rate"] |
| 89 | } |
| 90 | ``` |
| 91 | |
| 92 | You can also disable it for all rules inside given file by adding |
| 93 | a comment anywhere in that file. Example: |
| 94 | |
| 95 | ```yaml |
| 96 | # pint file/disable promql/rate |
| 97 | ``` |
| 98 | |
| 99 | Or you can disable it per rule by adding a comment to it. Example: |
| 100 | |
| 101 | ```yaml |
| 102 | # pint disable promql/rate |
| 103 | ``` |
| 104 | |
| 105 | If you want to disable only individual instances of this check |
| 106 | you can add a more specific comment. |
| 107 | |
| 108 | ```yaml |
| 109 | # pint disable promql/rate($prometheus) |
| 110 | ``` |
| 111 | |
| 112 | Where `$prometheus` is the name of Prometheus server to disable. |
| 113 | |
| 114 | Example: |
| 115 | |
| 116 | ```yaml |
| 117 | # pint disable promql/rate(prod) |
| 118 | ``` |
| 119 | |
| 120 | ## How to snooze it |
| 121 | |
| 122 | You can disable this check until given time by adding a comment to it. Example: |
| 123 | |
| 124 | ```yaml |
| 125 | # pint snooze $TIMESTAMP promql/rate |
| 126 | ``` |
| 127 | |
| 128 | Where `$TIMESTAMP` is either use [RFC3339](https://www.rfc-editor.org/rfc/rfc3339) |
| 129 | formatted or `YYYY-MM-DD`. |
| 130 | Adding this comment will disable `promql/rate` *until* `$TIMESTAMP`, after that |
| 131 | check will be re-enabled. |
| 132 | |