Many operators still run an SNMP-based NMS for alarm management. If your Kubernetes monitoring is built on kube-prometheus-stack, you can forward Prometheus alerts to that NMS as SNMP traps using snmp_notifier. This post walks through the full configuration, shows how to test it with fake alerts, and collects every troubleshooting command I used, together with real (sanitized) output, so you can compare against your own system.
1. How it fits together
PrometheusRule (alert has label oid=...)
│
▼
Prometheus ──fires──► Alertmanager
│ route: matchers oid=~".+"
▼
webhook receiver (HTTP POST /alerts)
│
▼
alertmanager-snmp-notifier (:9464)
│ SNMPv2c trap, UDP 162
▼
NMS / trap receiver(s)Key ideas:
- Only alerts that carry an
oidlabel are sent to SNMP. The label value becomes the trap's base OID. - Alertmanager does not speak SNMP. It sends a normal webhook to snmp_notifier, which converts it into a trap.
- With
continue: trueon the route, the same alert also reaches your other receivers (Slack, e-mail, …).
2. Prerequisites
- kube-prometheus-stack deployed with Helm (here in the
monitoringnamespace, release namekube-prometheus-stack). - The alertmanager-snmp-notifier Helm chart. In my setup it is bundled as a dependency of kube-prometheus-stack, enabled by
snmpAlertNotifier.enabled. With upstream kube-prometheus-stack, installprometheus-community/alertmanager-snmp-notifierseparately in the same namespace instead. - An enterprise OID for your traps. Below I use
1.3.6.1.4.1.XXXXX.41. ReplaceXXXXXwith your own Private Enterprise Number. - UDP 162 open from the Kubernetes nodes to the trap receivers.
3. Configuration
3.1 Enable the SNMP notifier and set the trap destinations
snmpAlertNotifier:
enabled: true # turns on the alertmanager-snmp-notifier sub-chart
oid: 1.3.6.1.4.1.XXXXX.41 # oid label added to the built-in node rules
type: service
alertmanager-snmp-notifier:
snmpNotifier:
extraArgs: []
# - --log.level=debug # handy while troubleshooting
snmpDestinations:
- 192.0.2.10:162 # NMS 1
- 192.0.2.20:162 # NMS 2
# The default community is "public". Prefer a secret:
# snmpCommunitySecret:
# name: snmp-community
# key: community
service:
type: ClusterIP
port: 94643.2 Trap content: description and extra varbinds
snmp_notifier lets you template the trap description and add extra varbinds. These templates put the pod, instance, namespace, host and container of the first alert into sub-OIDs 4–8:
alertmanager-snmp-notifier:
snmpNotifier:
snmpTemplates:
description: |
{{- if .Alerts -}}
{{ len .Alerts }}/{{ len .DeclaredAlerts }} alerts are firing:
{{ range $severity, $alerts := (groupAlertsByLabel .Alerts "severity") -}}
Status: {{ $severity }}
{{- range $index, $alert := $alerts }}
- Alert: {{ $alert.Labels.alertname }}
Summary: {{ $alert.Annotations.summary }}
Description: {{ $alert.Annotations.description }}
{{ end }}
{{ end }}
{{ else -}}
Status: OK
{{- end -}}
extraFields:
- subId: 4
template: |
{{- if .Alerts -}}{{ (index .Alerts 0).Labels.pod }}{{- else -}}{{- "" -}}{{- end -}}
- subId: 5
template: |
{{- if .Alerts -}}{{ (index .Alerts 0).Labels.instance }}{{- else -}}{{- "" -}}{{- end -}}
- subId: 6
template: |
{{- if .Alerts -}}{{ (index .Alerts 0).Labels.namespace }}{{- else -}}{{- "" -}}{{- end -}}
- subId: 7
template: |
{{- if .Alerts -}}{{ or ((index .Alerts 0).Labels.host) .CommonLabels.host "" }}{{- else -}}{{- "" -}}{{- end -}}
- subId: 8
template: |
{{- if .Alerts -}}{{ or ((index .Alerts 0).Labels.container) .CommonLabels.container "" }}{{- else -}}{{- "" -}}{{- end -}}3.3 Alertmanager receiver and route
Add a webhook receiver that points at the notifier service, and a route that matches any alert with an oid label:
alertmanager:
enabled: true
config:
global:
resolve_timeout: 1m
receivers:
- name: slack-notifications
slack_configs:
- api_url: '<your-slack-webhook-url>'
channel: 'alerts'
send_resolved: true
- name: snmp-notifications
webhook_configs:
- send_resolved: true
url: http://kube-prometheus-stack-alertmanager-snmp-notifier:9464/alerts
route:
receiver: slack-notifications # default receiver
group_by: ['...']
routes:
- matchers:
- oid=~".+" # only alerts that carry an oid label
receiver: snmp-notifications
group_by: ['alertname', 'pod', 'instance', 'namespace', 'host', 'container']
continue: true # keep evaluating, so Slack gets it too
- receiver: slack-notifications
group_by: ['...']Note: the webhook URL is
<release>-alertmanager-snmp-notifier. If your release name is different, check the real service name with kubectl get svc -n monitoring | grep snmp.3.4 Give your alerts an oid label
No oid label, no trap. In my chart, the built-in node CPU and memory rules get the label automatically when snmpAlertNotifier.enabled is true. They live in the general rule group, so that group must be enabled:
defaultRules:
rules:
general: true # if false, the oid-labelled node rules are not rendered at allFor your own rules, just add the labels:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: my-app-snmp-rules
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
groups:
- name: my-app
rules:
- alert: MyAppDown
expr: up{job="my-app"} == 0
for: 1m
labels:
severity: critical
oid: 1.3.6.1.4.1.XXXXX.41
type: service
annotations:
summary: "my-app is down"
description: "Target {{ $labels.instance }} has been down for 1 minute."3.5 Deploy
helm upgrade kube-prometheus-stack ./kube-prometheus-stack -f ./kube-prometheus-stack/my-values.yaml -n monitoring4. Testing with a fake alert (amtool)
You don't need to wait for a real problem. amtool is included in the Alertmanager image, so you can inject an alert directly. The commands below are for bash (Linux/WSL).
4.1 Fire the alert
END=$(date -u -d '+1 hour' +%Y-%m-%dT%H:%M:%SZ)
kubectl exec -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
amtool alert add alertname=SnmpTestAlert severity=critical oid=1.3.6.1.4.1.XXXXX.41 type=service \
instance=test-instance namespace=monitoring pod=test-pod \
'--annotation=summary="SNMP test alert"' '--annotation=description="Manual SNMP trap test"' \
--end=$END --alertmanager.url=http://localhost:90934.2 Check that it is active
kubectl exec -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
amtool alert query alertname=SnmpTestAlert --alertmanager.url=http://localhost:9093Output:
Alertname Starts At Summary State
SnmpTestAlert 2026-11-02 07:29:27 UTC SNMP test alert active4.3 Resolve the alert
Send the alert again with exactly the same labels and an end time of now:
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo $NOW # must be the current UTC time, NOT one hour ahead
kubectl exec -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
amtool alert add alertname=SnmpTestAlert severity=critical oid=1.3.6.1.4.1.XXXXX.41 type=service \
instance=test-instance namespace=monitoring pod=test-pod \
'--annotation=summary="SNMP test alert"' '--annotation=description="Manual SNMP trap test"' \
--end=$NOW --alertmanager.url=http://localhost:9093Query again. An empty result means it is resolved (amtool alert query only shows active alerts):
Alertname Starts At Summary State4.4 Timing to expect
- Firing trap: about 30 s after you send the alert (
group_wait, default 30s). - Resolved trap: up to about 5 minutes after the resolve (
group_interval, default 5m). - No end time: with
resolve_timeout: 1m, an alert added without--endresolves itself after about a minute.
5. Verifying and troubleshooting, step by step
Work along the chain: route → webhook → notifier → network. Each step below has the command and what a healthy result looks like.
5.1 Are the pods and the service there?
kubectl get pods,svc -n monitoring | grep -E "snmp|alertmanager"You should see a running …-alertmanager-snmp-notifier pod and a service on port 9464 whose name matches the webhook URL.
5.2 Does the alert match the SNMP route?
Test the routing tree against the config Alertmanager actually loaded:
kubectl exec -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
amtool config routes test --config.file=/etc/alertmanager/config_out/alertmanager.env.yaml \
alertname=SnmpTestAlert oid=1.3.6.1.4.1.XXXXX.41 severity=criticalHealthy output (both receivers, thanks to continue: true):
snmp-notifications,slack-notificationsIf you only see slack-notifications, your route was not applied. Show the loaded tree with:
kubectl exec -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
amtool config routes show --config.file=/etc/alertmanager/config_out/alertmanager.env.yaml5.3 Did Alertmanager deliver the webhook?
Alertmanager only logs failed notifications, so an empty log is a good sign:
kubectl logs -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 -c alertmanager --since=1h \
| grep -iE "snmp|webhook|notify|error"The metrics are more reliable:
kubectl exec -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
wget -qO- http://localhost:9093/metrics | grep -E 'alertmanager_notifications_(total|failed_total)\{.*webhook'Healthy output:
alertmanager_notifications_failed_total{integration="webhook",reason="clientError"} 0
alertmanager_notifications_failed_total{integration="webhook",reason="contextCanceled"} 0
alertmanager_notifications_failed_total{integration="webhook",reason="contextDeadlineExceeded"} 0
alertmanager_notifications_failed_total{integration="webhook",reason="other"} 0
alertmanager_notifications_failed_total{integration="webhook",reason="serverError"} 0
alertmanager_notifications_total{integration="webhook"} 3Run it before and after firing a test alert. The notifications_total counter should go up by one (3 → 4 in my test), and every failed_total should stay at 0.
5.4 What does the notifier say?
kubectl logs -n monitoring deploy/kube-prometheus-stack-alertmanager-snmp-notifier --since=1h
kubectl exec -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
wget -qO- http://kube-prometheus-stack-alertmanager-snmp-notifier:9464/metrics | grep -i snmpIf there isn't enough detail, add --log.level=debug to snmpNotifier.extraArgs and run helm upgrade again.
5.5 Bypass Alertmanager: call the notifier directly
This tells you whether the notifier and network work on their own:
kubectl exec -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 -c alertmanager -- \
wget -qO- --header='Content-Type: application/json' --post-data='{"receiver":"snmp-notifications","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"SnmpDirectTest","oid":"1.3.6.1.4.1.XXXXX.41","severity":"critical","instance":"test"},"annotations":{"summary":"direct test","description":"direct webhook test"},"startsAt":"2026-11-02T08:00:00Z"}],"groupLabels":{"alertname":"SnmpDirectTest"},"commonLabels":{"alertname":"SnmpDirectTest","oid":"1.3.6.1.4.1.XXXXX.41","severity":"critical"},"commonAnnotations":{},"externalURL":"http://alertmanager","version":"4","groupKey":"test"}' \
http://kube-prometheus-stack-alertmanager-snmp-notifier:9464/alerts5.6 Capture the traps with ksniff
ksniff runs tcpdump in a pod's network namespace. Capture both the incoming webhook (TCP 9464) and the outgoing traps (UDP 162):
POD=$(kubectl get pods -n monitoring -o name | grep snmp-notifier | head -1 | cut -d/ -f2)
kubectl sniff $POD -n monitoring -p -f "udp port 162 or tcp port 9464" -o snmp_trap_test.pcap-p(privileged mode) is needed because the notifier image is minimal and non-root, so ksniff can't upload a static tcpdump into it.-owrites to a file. Without it, ksniff tries to open Wireshark, which won't work in WSL.- On containerd clusters you may need
--socket /run/containerd/containerd.sock. On air-gapped clusters, point--imageand--tcpdump-imageat your own registry.
Start the capture first, then fire the test alert, wait about a minute, resolve it, wait about 5 minutes, and stop with Ctrl+C.
In Wireshark, filter out the health-probe noise with:
http.request.method == "POST" || udp.port == 162A successful capture looks like this:
No. Source Destination Protocol Info
147 10.244.1.20 10.244.1.30 HTTP/JSON POST /alerts HTTP/1.1
149 10.244.1.30 192.0.2.10 SNMP snmpV2-trap 1.3.6.1.2.1.1.3.0 ...
150 10.244.1.30 192.0.2.20 SNMP snmpV2-trap 1.3.6.1.2.1.1.3.0 ...Alertmanager (10.244.1.20) posts the alert to the notifier (10.244.1.30). About 3 ms later, the notifier sends one trap to each destination.
Don't be fooled by probes: if your capture only shows
GET / HTTP/1.1 every 10 seconds from the node IP, that's the kubelet's liveness/readiness probes, not alerts. Alert deliveries are always POST /alerts.6. Anatomy of the trap
Expanding the trap in Wireshark:
Simple Network Management Protocol
version: v2c (1)
community: public
data: snmpV2-trap (7)
variable-bindings: 10 items
1.3.6.1.2.1.1.3.0: 3629193000
1.3.6.1.6.3.1.1.4.1.0: 1.3.6.1.4.1.XXXXX.41.1
1.3.6.1.4.1.XXXXX.41.2.1: "1.3.6.1.4.1.XXXXX.41[alertname=SnmpTestAlert,instance=test-instance,namespace=monitoring,pod=test-pod]"
1.3.6.1.4.1.XXXXX.41.2.2: "critical"
1.3.6.1.4.1.XXXXX.41.2.3: 312f3120616c657274732061726520666972696e673a...
1.3.6.1.4.1.XXXXX.41.2.8: <MISSING>
1.3.6.1.4.1.XXXXX.41.2.4: "test-pod"
1.3.6.1.4.1.XXXXX.41.2.5: "test-instance"
1.3.6.1.4.1.XXXXX.41.2.6: "monitoring"
1.3.6.1.4.1.XXXXX.41.2.7: <MISSING>| Varbind | Meaning |
|---|---|
1.3.6.1.2.1.1.3.0 | sysUpTime |
1.3.6.1.6.3.1.1.4.1.0 | snmpTrapOID: the alert's oid label + .1 |
<oid>.2.1 | Alert ID: the OID plus the route's group_by labels |
<oid>.2.2 | Severity (severity label) |
<oid>.2.3 | Description from the description template. Wireshark shows it as hex because it contains newlines; it decodes to "1/1 alerts are firing: …" |
<oid>.2.4 – .2.8 | Extra fields: pod, instance, namespace, host, container |
<MISSING> just means the value is empty. The test alert had no host or container label. Add them to the amtool alert add command and fields 7 and 8 are filled in.
7. Troubleshooting cheat sheet
| Symptom | Likely cause | Fix / check |
|---|---|---|
| No traps from real alerts, but test alerts work | Your rules have no oid label, or the rule group carrying it is disabled (e.g. defaultRules.rules.general: false) |
Enable the group, or add oid/type labels to your PrometheusRules |
routes test returns only the default receiver |
The SNMP route isn't in the loaded config, or the matcher is wrong | amtool config routes show (§5.2) |
alertmanager_notifications_failed_total{integration="webhook"} increasing |
Wrong service name or port in the webhook URL, or the notifier pod is down | kubectl get svc -n monitoring | grep snmp; Alertmanager logs |
| Webhook counter goes up, POST visible in pcap, no UDP 162 | Notifier problem (template error, bad destination) | Notifier logs with --log.level=debug |
Pcap shows only GET / every 10 s |
Those are kubelet probes. The capture started after the alert was delivered | Start ksniff before firing the alert |
| Traps leave the pod but the NMS sees nothing | Firewall/routing between the nodes and the NMS, or a community mismatch | tcpdump -nn udp port 162 on the NMS; set snmpCommunity/snmpCommunitySecret |
Alert stays active after "resolving" |
--end was in the future (e.g. NOW computed with -d '+1 hour'), or a label differs |
echo $NOW must be the current UTC time; labels must match exactly |
| Test alert disappears by itself after ~1 minute | No --end given; resolve_timeout: 1m resolves it |
Pass --end one hour ahead |
-bash: syntax error near unexpected token `(' |
A PowerShell command ((Get-Date)…) was pasted into bash |
Use the bash date -u … versions above |
level=warn msg="Alertmanager is moving to a new parser…" |
Only a warning: annotation values with spaces aren't double-quoted | Wrap as '--annotation=summary="SNMP test alert"' |
| Resolved trap "missing" | It is sent at the next group_interval (default 5 min) |
Keep capturing for about 5 minutes after resolving |
Summary: if
amtool config routes test lists snmp-notifications, the webhook counter goes up with no failures, and a capture shows POST /alerts followed by snmpV2-trap packets to your NMS addresses, the kube-prometheus-stack side is working end to end. Anything left is between the cluster and the NMS.
No comments:
Post a Comment