cloudflare/pint
Publicmirrored from https://github.com/cloudflare/pintAvailable
docs/checks/labels/conflict.md
140lines · modecode
| 1 | --- |
| 2 | layout: default |
| 3 | parent: Checks |
| 4 | grand_parent: Documentation |
| 5 | --- |
| 6 | |
| 7 | # labels/conflict |
| 8 | |
| 9 | This check will look for any conflicting labels used in rules. |
| 10 | Below is the list of conflicts it looks for. |
| 11 | |
| 12 | ## External labels |
| 13 | |
| 14 | ### Recording rules |
| 15 | |
| 16 | If any recording rules are manually setting some labels that are |
| 17 | already present in `external_labels` Prometheus configuration option |
| 18 | then both labels might conflict when metrics are ingested to another |
| 19 | Prometheus via federation or remote read/write. |
| 20 | |
| 21 | Example: |
| 22 | |
| 23 | Consider this recording rule: |
| 24 | |
| 25 | ```yaml |
| 26 | groups: |
| 27 | name: recording rules |
| 28 | rules: |
| 29 | - record: prometheus_http_requests_total:rate2m |
| 30 | expr: rate(prometheus_http_requests_total[2m]) |
| 31 | labels: |
| 32 | cluster: dev |
| 33 | ``` |
| 34 | |
| 35 | If this rule is deployed to Prometheus server with this configuration: |
| 36 | |
| 37 | ```yaml |
| 38 | global: |
| 39 | external_labels: |
| 40 | site: site01 |
| 41 | cluster: staging |
| 42 | ``` |
| 43 | |
| 44 | Then making a `/federate` request will return time series with `cluster="staging"` label, |
| 45 | except for `prometheus_http_requests_total:rate2m` time series which will have `cluster="dev"` |
| 46 | label from the recording rule, which might cause unexpected inconsistencies. |
| 47 | |
| 48 | If both the recording rule and `external_labels` config section uses same label value for the |
| 49 | `cluster` label then this effectively makes `cluster` label redundant, so in both cases it's |
| 50 | best to avoid setting labels used in `external_labels` on individual rules. |
| 51 | |
| 52 | ### Alerting rules |
| 53 | |
| 54 | Same problem exists for alerting rules, with the only difference being how the label is |
| 55 | being used. |
| 56 | Any label listed in `external_labels` will be added to all firing alerts. |
| 57 | Setting `cluster` label on alerting rule will override `external_labels` and |
| 58 | can cause confusion when alert sent from `cluster="staging"` Prometheus has `cluster="dev"` |
| 59 | label set. |
| 60 | |
| 61 | ## Configuration |
| 62 | |
| 63 | This check doesn't have any configuration options. |
| 64 | |
| 65 | ## How to enable it |
| 66 | |
| 67 | This check is enabled by default for all configured Prometheus servers. |
| 68 | |
| 69 | Example: |
| 70 | |
| 71 | ```js |
| 72 | prometheus "prod" { |
| 73 | uri = "https://prometheus-prod.example.com" |
| 74 | timeout = "60s" |
| 75 | include = [ |
| 76 | "rules/prod/.*", |
| 77 | "rules/common/.*", |
| 78 | ] |
| 79 | } |
| 80 | |
| 81 | prometheus "dev" { |
| 82 | uri = "https://prometheus-dev.example.com" |
| 83 | timeout = "30s" |
| 84 | include = [ |
| 85 | "rules/dev/.*", |
| 86 | "rules/common/.*", |
| 87 | ] |
| 88 | } |
| 89 | ``` |
| 90 | |
| 91 | ## How to disable it |
| 92 | |
| 93 | You can disable this check globally by adding this config block: |
| 94 | |
| 95 | ```js |
| 96 | checks { |
| 97 | disabled = ["labels/conflict"] |
| 98 | } |
| 99 | ``` |
| 100 | |
| 101 | You can also disable it for all rules inside given file by adding |
| 102 | a comment anywhere in that file. Example: |
| 103 | |
| 104 | ```yaml |
| 105 | # pint file/disable labels/conflict |
| 106 | ``` |
| 107 | |
| 108 | Or you can disable it per rule by adding a comment to it. Example: |
| 109 | |
| 110 | ```yaml |
| 111 | # pint disable labels/conflict |
| 112 | ``` |
| 113 | |
| 114 | If you want to disable only individual instances of this check |
| 115 | you can add a more specific comment. |
| 116 | |
| 117 | ```yaml |
| 118 | # pint disable labels/conflict($prometheus) |
| 119 | ``` |
| 120 | |
| 121 | Where `$prometheus` is the name of Prometheus server to disable. |
| 122 | |
| 123 | Example: |
| 124 | |
| 125 | ```yaml |
| 126 | # pint disable labels/conflict(prod) |
| 127 | ``` |
| 128 | |
| 129 | ## How to snooze it |
| 130 | |
| 131 | You can disable this check until given time by adding a comment to it. Example: |
| 132 | |
| 133 | ```yaml |
| 134 | # pint snooze $TIMESTAMP labels/conflict |
| 135 | ``` |
| 136 | |
| 137 | Where `$TIMESTAMP` is either use [RFC3339](https://www.rfc-editor.org/rfc/rfc3339) |
| 138 | formatted or `YYYY-MM-DD`. |
| 139 | Adding this comment will disable `labels/conflict` *until* `$TIMESTAMP`, after that |
| 140 | check will be re-enabled. |
| 141 | |