What to measure
| Method | For | Signals |
|---|---|---|
| RED | Request-driven services | Rate, Errors, Duration |
| USE | Resources | Utilization, Saturation, Errors |
| Golden signals | Any service | Latency, traffic, errors, saturation |
Use percentiles, not averages, for latency.
Metric types
| Type | Behaviour | Query with |
|---|---|---|
| Counter | Only increases | rate(), increase() |
| Gauge | Up and down | Direct, avg_over_time, predict_linear |
| Histogram | Bucketed observations | histogram_quantile over rate of buckets |
PromQL patterns
# Requests per second by route
sum by (route) (rate(http_requests_total[5m]))
# Error ratio
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))
# p95 latency per route
histogram_quantile(0.95,
sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
# Targets that are down
up == 0
# Node memory used percent
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
# Disk will fill within 4 hours
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 4 * 3600) < 0
# Pod restarts in the last hour
increase(kube_pod_container_status_restarts_total[1h]) > 3
# Top 5 series by value
topk(5, sum by (pod) (rate(container_cpu_usage_seconds_total[5m])))Rules of thumb:
- Never graph a raw counter; wrap it in
rate() - Use a
rate()window of at least four scrape intervals - Keep
lewhen aggregating histogram buckets - Do not average percentiles
- Avoid unbounded labels such as user IDs and raw URLs
Scrape configuration
global:
scrape_interval: 15s
scrape_configs:
- job_name: shop-api
metrics_path: /metrics
static_configs:
- targets: ["shop-api-1:8080", "shop-api-2:8080"]Recording and alert rules
groups:
- name: shop-api
rules:
- record: route:http_requests:rate5m
expr: sum by (route) (rate(http_requests_total[5m]))
- alert: ShopApiHighErrorRatio
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.02
for: 10m
labels:
severity: page
annotations:
summary: "Error ratio above 2% for 10 minutes"
runbook: "https://runbooks.example.com/shop-api/high-errors"Alertmanager
route:
receiver: team-chat
group_by: [alertname, service]
group_wait: 30s
repeat_interval: 4h
routes:
- matchers: ['severity="page"']
receiver: on-call-pager
receivers:
- name: team-chat
- name: on-call-pageramtool alert query --active
amtool silence add alertname=ShopApiHighErrorRatio --duration=2h --comment="planned work"
promtool check rules rules.yml
promtool check config prometheus.ymlAlert design
- Must a human act now?
- Is it a symptom users feel?
- Does the page link to a runbook?
Page on symptoms; ticket for slow-burn causes; dashboards for the rest.
Logs and traces
| Practice | Why |
|---|---|
| Structured JSON logs | Fields can be queried |
| Request ID and trace ID in every line | Connects logs, metrics and traces |
| Never log secrets or personal data | Logs are widely copied |
| Tail sampling for traces | Keeps slow and failing requests |
| Log to stdout in containers | The platform collects it |
{app="shop-api"} | json | level="error" | route="/checkout"Cardinality check
curl -s localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[0:5]'Query patterns to know
sum by (status) (rate(http_requests_total[5m]))
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24 * 3600) < 0
increase(kube_pod_container_status_restarts_total[1h]) > 3
up == 0
count by (job) (up)
topk(5, sum by (pod) (container_memory_working_set_bytes))
absent(up{job="api"})Operate the server
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/*.yml
promtool test rules tests/alerts_test.yml
promtool query instant http://localhost:9090 'up'
curl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health:.health, err:.lastError}'
curl -s localhost:9090/api/v1/alerts | jq '.data.alerts[].labels'
curl -X POST localhost:9090/-/reload
curl -s localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:5]'
amtool alert query
amtool silence add alertname=HighLatency --duration=2h --comment="deploy in progress" --author=me
amtool config routes test --config.file=alertmanager.yml severity=page team=shopAlert rule skeleton
groups:
- name: api
rules:
- alert: HighErrorRatio
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.02
for: 10m
labels: {severity: page, team: shop}
annotations:
summary: "More than 2 percent of requests fail"
runbook_url: "https://runbooks.example.com/high-error-ratio"