cloudflare/pint
Publicmirrored from https://github.com/cloudflare/pintAvailable
docs/checks/promql/counter.md
140lines · modecode
| 1 | --- |
| 2 | layout: default |
| 3 | parent: Checks |
| 4 | grand_parent: Documentation |
| 5 | --- |
| 6 | |
| 7 | # promql/counter |
| 8 | |
| 9 | This check will find rules with invalid use of counters. |
| 10 | [Counters](https://prometheus.io/docs/concepts/metric_types/#counter) track the number of events over time and so the value of a counter can only grow and never decrease. |
| 11 | This means that the absolute value of a counter doesn't matter, it will be a random number that depends on the number of events that happened since your application was started. |
| 12 | To use the value of a counter in PromQL you most likely want to calculate the rate of events using the [rate()](https://prometheus.io/docs/prometheus/latest/querying/functions/#rate) function, or any other function that is safe to use with counters. |
| 13 | Once you calculate the rate you can use that result in other functions or aggregations that are not counter safe, like [sum()](https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators).` |
| 14 | |
| 15 | Here's an example of invalid alerting rules that uses a counter metric called `errors_total`. |
| 16 | This metric will be incremented every time there's an error. |
| 17 | |
| 18 | A bad rule could look like this: |
| 19 | |
| 20 | ```yaml |
| 21 | - alert: Too many errors |
| 22 | expr: errors_total > 10 |
| 23 | ``` |
| 24 | |
| 25 | The problem here is that a counter like `errors_total` will only go up in value until: |
| 26 | |
| 27 | - the value overflows the maximum value for a float |
| 28 | - your service restarts and resets the value of `errors_total` to zero - so it starts counting again |
| 29 | |
| 30 | Once there are 11 errors observed since your application started `Too many errors` alert will fire and will keep firing |
| 31 | until your application restarts. |
| 32 | This kind of alerts is usually unhelpful and what you really want to track is the health of your application |
| 33 | **right now**. This alert should be triggered if, for example, in the last 1 hour there were more than N errors. |
| 34 | If there's a spike of errors but then errors stop, then the alert should stop firing. |
| 35 | |
| 36 | Example of a better rule: |
| 37 | |
| 38 | ```yaml |
| 39 | - alert: Too many errors |
| 40 | expr: rate(errors_total[1h]) > 10 |
| 41 | ``` |
| 42 | |
| 43 | ## Common problems |
| 44 | |
| 45 | ### Metadata mismatch |
| 46 | |
| 47 | Metric type checks are using |
| 48 | [metadata API](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata). |
| 49 | Metadata is aggregated from all scraped metrics. |
| 50 | |
| 51 | This can cause a few potential problems: |
| 52 | |
| 53 | - You might have the same metric reported with multiple different types and Prometheus or pint won't know |
| 54 | which time series is which type, because all we have to match a metric to a type is its name. |
| 55 | Best solution here is to never export same name as multiple metrics with different types. |
| 56 | - If you change the typo of some exported metric then the old type will still show up in metadata, |
| 57 | plus the new one, as long as there's at least one target still exporting old metric type. |
| 58 | If you accidentally exported some metric with wrong type, then fixed it, but pint is still complaining, |
| 59 | then it's very likely that you didn't release your fix to all targets yet. |
| 60 | |
| 61 | ## Configuration |
| 62 | |
| 63 | This check doesn't have any configuration options. |
| 64 | |
| 65 | ## How to enable it |
| 66 | |
| 67 | This check is enabled by default for all configured Prometheus servers. |
| 68 | |
| 69 | Example: |
| 70 | |
| 71 | ```js |
| 72 | prometheus "prod" { |
| 73 | uri = "https://prometheus-prod.example.com" |
| 74 | timeout = "60s" |
| 75 | include = [ |
| 76 | "rules/prod/.*", |
| 77 | "rules/common/.*", |
| 78 | ] |
| 79 | } |
| 80 | |
| 81 | prometheus "dev" { |
| 82 | uri = "https://prometheus-dev.example.com" |
| 83 | timeout = "30s" |
| 84 | include = [ |
| 85 | "rules/dev/.*", |
| 86 | "rules/common/.*", |
| 87 | ] |
| 88 | } |
| 89 | ``` |
| 90 | |
| 91 | ## How to disable it |
| 92 | |
| 93 | You can disable this check globally by adding this config block: |
| 94 | |
| 95 | ```js |
| 96 | checks { |
| 97 | disabled = ["promql/counter"] |
| 98 | } |
| 99 | ``` |
| 100 | |
| 101 | You can also disable it for all rules inside given file by adding |
| 102 | a comment anywhere in that file. Example: |
| 103 | |
| 104 | ```yaml |
| 105 | # pint file/disable promql/counter |
| 106 | ``` |
| 107 | |
| 108 | Or you can disable it per rule by adding a comment to it. Example: |
| 109 | |
| 110 | ```yaml |
| 111 | # pint disable promql/counter |
| 112 | ``` |
| 113 | |
| 114 | If you want to disable only individual instances of this check |
| 115 | you can add a more specific comment. |
| 116 | |
| 117 | ```yaml |
| 118 | # pint disable promql/counter($prometheus) |
| 119 | ``` |
| 120 | |
| 121 | Where `$prometheus` is the name of Prometheus server to disable. |
| 122 | |
| 123 | Example: |
| 124 | |
| 125 | ```yaml |
| 126 | # pint disable promql/counter(prod) |
| 127 | ``` |
| 128 | |
| 129 | ## How to snooze it |
| 130 | |
| 131 | You can disable this check until given time by adding a comment to it. Example: |
| 132 | |
| 133 | ```yaml |
| 134 | # pint snooze $TIMESTAMP promql/counter |
| 135 | ``` |
| 136 | |
| 137 | Where `$TIMESTAMP` is either use [RFC3339](https://www.rfc-editor.org/rfc/rfc3339) |
| 138 | formatted or `YYYY-MM-DD`. |
| 139 | Adding this comment will disable `promql/counter` *until* `$TIMESTAMP`, after that |
| 140 | check will be re-enabled. |
| 141 | |