AKS Observability with Managed Prometheus and Grafana: Metrics, Alerts, and Dashboards

You promoted a microservices workload to production on AKS last quarter. Since then, the on-call rotation has become a guessing game: is the latency spike a noisy neighbour on a node, a memory-pressured pod, or a throttled container hitting its CPU limit? Without structured metrics collection and alert routing, every incident starts at zero.

Azure's Managed Prometheus (Azure Monitor managed service for Prometheus) and Azure Managed Grafana solve the infrastructure half of that problem — Microsoft operates the Prometheus-compatible remote-write endpoint, the long-term store, and the Grafana workspace, so your team owns signal design rather than scraper maintenance. This tutorial walks you through enabling the full stack on an existing AKS cluster, writing alert rules that page on real conditions, and building dashboards that survive a 3 AM incident.

---

Prerequisites

Install the required CLI extensions if they're missing:

az extension add --name aks-preview --upgrade
az extension add --name amg --upgrade   # Azure Managed Grafana

Cost note: Azure Monitor managed service for Prometheus charges per time series ingested per month. Review the pricing page and set a data collection rule (DCR) cap before enabling on large clusters.

---

Architecture Overview

Before touching the CLI, understand what you're wiring together. Three distinct Azure services collaborate; getting the data-flow direction wrong is the most common first-hour mistake.

flowchart LR
    subgraph AKS Cluster
        A[Node Exporter DaemonSet] -->|scrape| P[ama-metrics pods\nManaged Prometheus Agent]
        B[kube-state-metrics] -->|scrape| P
        C[Your App /metrics] -->|scrape via PodMonitor| P
    end

    P -->|remote_write HTTPS| AMW[(Azure Monitor\nWorkspace\nPrometheus endpoint)]

    AMW -->|PromQL data source| G[Azure Managed Grafana]
    AMW -->|Alert rules| AR[Azure Monitor\nAlert Rules]
    AR -->|Action Group| N[Email / PagerDuty /\nWebhook]

    style AMW fill:#0072C6,color:#fff
    style G fill:#F46800,color:#fff

Key insight: ama-metrics (the Azure Monitor Agent for metrics) runs inside the cluster and pushes to the Azure Monitor Workspace via remote_write. You never expose a Prometheus port externally. Grafana queries the workspace via a native Prometheus-compatible data source that Microsoft pre-wires when you use the portal or CLI integration.

---

Step 1 — Create the Azure Monitor Workspace

An Azure Monitor Workspace is the managed Prometheus backend. One workspace can receive metrics from multiple AKS clusters, which matters for multi-cluster dashboards.

RESOURCE_GROUP="rg-aks-observability"
LOCATION="eastus2"
AMW_NAME="amw-aks-prod"

az monitor account create \
  --name $AMW_NAME \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --query id --output tsv

Save the returned resource ID — you'll reference it in the next step.

---

Step 2 — Enable Managed Prometheus on the AKS Cluster

The --enable-azure-monitor-metrics flag installs the ama-metrics DaemonSet and wires its remote_write target to your workspace. This is a non-destructive operation on a running cluster — existing workloads are not restarted.

CLUSTER_NAME="aks-prod-eastus2"
AMW_ID="/subscriptions/<sub-id>/resourceGroups/rg-aks-observability/providers/microsoft.monitor/accounts/amw-aks-prod"

az aks update \
  --name $CLUSTER_NAME \
  --resource-group $RESOURCE_GROUP \
  --enable-azure-monitor-metrics \
  --azure-monitor-workspace-resource-id $AMW_ID

The operation takes 3–5 minutes. Confirm the agent pods are running:

kubectl get pods -n kube-system -l rsName=ama-metrics
# Expected: 1 pod per node (DaemonSet) + 2 replica pods for the collector

As of the AKS 1.27 release notes, the ama-metrics agent uses the prometheus.io/scrape annotation convention by default, meaning any pod annotated with prometheus.io/scrape: "true" is automatically included in the default scrape config. Verify which targets are being scraped by querying the agent's debug port from a local port-forward:

kubectl port-forward -n kube-system ds/ama-metrics-node 9090:9090 &
curl http://localhost:9090/metrics | grep scrape_samples_scraped | head -20

---

Step 3 — Create and Link Azure Managed Grafana

GRAFANA_NAME="grafana-aks-prod"

az grafana create \
  --name $GRAFANA_NAME \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --sku Standard

GRAFANA_ID=$(az grafana show \
  --name $GRAFANA_NAME \
  --resource-group $RESOURCE_GROUP \
  --query id --output tsv)

Now link the Grafana instance to the Azure Monitor Workspace so that the Prometheus data source is auto-configured:

az aks update \
  --name $CLUSTER_NAME \
  --resource-group $RESOURCE_GROUP \
  --enable-azure-monitor-metrics \
  --azure-monitor-workspace-resource-id $AMW_ID \
  --grafana-resource-id $GRAFANA_ID

Running the az aks update with both flags is idempotent — if Prometheus was already enabled, this only adds the Grafana link. Azure creates a system-managed data source named Azure Monitor Workspace - <workspace-name> inside Grafana automatically; you don't configure a URL or API key manually.

---

Step 4 — Scraping Custom Application Metrics with PodMonitor

The default scrape config covers Kubernetes infrastructure metrics (node CPU/memory, kube-state-metrics, cAdvisor). To scrape your application's /metrics endpoint, use a PodMonitor custom resource — the Prometheus Operator CRD spec that ama-metrics natively understands.

# podmonitor-myapp.yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: myapp-metrics
  namespace: production
  labels:
    app: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  podMetricsEndpoints:
    - port: metrics          # named port in your Pod spec
      path: /metrics
      interval: 30s
      scheme: http
  namespaceSelector:
    matchNames:
      - production

Apply it and verify the target appears in the remote_write stream within 60 seconds:

kubectl apply -f podmonitor-myapp.yaml

# Spot-check: query the workspace for a known metric from your app
az monitor metrics list \
  --resource $AMW_ID \
  --metric "myapp_http_requests_total" \
  --output table

Pitfall: If the PodMonitor has no labels that match the agent's podMonitorSelector, it will be silently ignored. The managed agent's default selector is {} (match all), but check if a custom ConfigMap named ama-metrics-settings-configmap in kube-system overrides this — a previous admin may have locked the selector.

---

Step 5 — Writing Prometheus Alert Rules

Azure Monitor Workspace alert rules are defined as PrometheusRuleGroup resources (ARM/Bicep) or via the CLI. Rules are evaluated server-side by the workspace, not inside the cluster, which means they fire even if the cluster is completely down — a critical advantage over in-cluster Alertmanager.

Create an Action Group First

az monitor action-group create \
  --name "ag-aks-oncall" \
  --resource-group $RESOURCE_GROUP \
  --short-name "aks-oncall" \
  --action email oncall-eng oncall@example.com

Deploy Alert Rules via Bicep

Using Bicep (or ARM) rather than the portal gives you version-controlled, repeatable alert definitions:

// aks-alerts.bicep
param location string = resourceGroup().location
param amwId string
param actionGroupId string

resource prometheusRuleGroup 'Microsoft.AlertsManagement/prometheusRuleGroups@2023-03-01' = {
  name: 'aks-critical-alerts'
  location: location
  properties: {
    description: 'Critical AKS cluster and workload alerts'
    scopes: [ amwId ]
    interval: 'PT1M'
    rules: [
      {
        record: 'job:node_cpu_utilisation:avg1m'
        expression: '1 - avg(rate(node_cpu_seconds_total{mode="idle"}[1m])) by (node)'
      }
      {
        alert: 'NodeCPUSaturation'
        expression: 'job:node_cpu_utilisation:avg1m > 0.90'
        for: 'PT5M'
        severity: 2
        annotations: {
          summary: 'Node {{ $labels.node }} CPU > 90% for 5 min'
          description: 'Current value: {{ $value | humanizePercentage }}'
        }
        resolveConfiguration: {
          autoResolved: true
          timeToResolve: 'PT10M'
        }
        actions: [
          { actionGroupId: actionGroupId }
        ]
      }
      {
        alert: 'PodCrashLooping'
        expression: 'rate(kube_pod_container_status_restarts_total[15m]) * 60 > 1'
        for: 'PT5M'
        severity: 1
        annotations: {
          summary: 'Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash-looping'
        }
        resolveConfiguration: {
          autoResolved: true
          timeToResolve: 'PT15M'
        }
        actions: [
          { actionGroupId: actionGroupId }
        ]
      }
      {
        alert: 'ContainerMemoryNearLimit'
        expression: |
          (container_memory_working_set_bytes{container!=""} 
            / on(namespace,pod,container)
          kube_pod_container_resource_limits{resource="memory",unit="byte"}) > 0.85
        for: 'PT10M'
        severity: 3
        annotations: {
          summary: 'Container {{ $labels.container }} in {{ $labels.namespace }}/{{ $labels.pod }} is using >85% of its memory limit'
        }
        resolveConfiguration: {
          autoResolved: true
          timeToResolve: 'PT15M'
        }
        actions: [
          { actionGroupId: actionGroupId }
        ]
      }
    ]
  }
}

Deploy it:

az deployment group create \
  --resource-group $RESOURCE_GROUP \
  --template-file aks-alerts.bicep \
  --parameters amwId=$AMW_ID actionGroupId=$(az monitor action-group show \
      --name ag-aks-oncall \
      --resource-group $RESOURCE_GROUP \
      --query id --output tsv)

The record rule at the top creates a recording rule — a pre-aggregated time series stored back into the workspace. Pre-computing expensive aggregations as recording rules is the single biggest dashboard query performance win available at this layer.

---

Step 6 — Building Useful Grafana Dashboards

Microsoft publishes a set of pre-built AKS dashboards that are automatically imported when you link Grafana via the CLI. These cover node utilisation, pod resource usage, and persistent volume capacity. Use them as a foundation, not a final answer.

Import Community Dashboards via Grafana CLI

The grafana CLI (available inside Grafana's built-in shell or via the HTTP API) lets you pull community dashboards from grafana.com:

# Use the Managed Grafana API endpoint
GRAFANA_ENDPOINT=$(az grafana show \
  --name $GRAFANA_NAME \
  --resource-group $RESOURCE_GROUP \
  --query properties.endpoint --output tsv)

# Import Kubernetes cluster overview (dashboard ID 15520 on grafana.com)
az grafana dashboard import \
  --name $GRAFANA_NAME \
  --resource-group $RESOURCE_GROUP \
  --definition 15520 \
  --overwrite true

A Practical Application SLO Panel (JSON Model Snippet)

Rather than import someone else's dashboard verbatim, augment it with panels tied to your own SLOs. The following panel definition queries a 99th-percentile latency metric and renders it against a 300ms SLO threshold:

{
  "title": "p99 Request Latency vs SLO",
  "type": "timeseries",
  "datasource": {
    "type": "prometheus",
    "uid": "${DS_AZURE_MONITOR_WORKSPACE}"
  },
  "targets": [
    {
      "expr": "histogram_quantile(0.99, sum(rate(myapp_http_request_duration_seconds_bucket[5m])) by (le, route))",
      "legendFormat": "p99 - {{ route }}"
    },
    {
      "expr": "0.300",
      "legendFormat": "SLO: 300ms"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "s",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 0.200 },
          { "color": "red", "value": 0.300 }
        ]
      }
    }
  }
}

Paste this into Dashboard → Panel → Edit → JSON model to add it to any existing dashboard.

---

Common Pitfalls to Avoid

1. Scraping too many high-cardinality labels.
Adding pod_ip or request_id as metric labels creates an unbounded number of time series. Each unique label combination is a separate series in the workspace. A single misconfigured histogram with a high-cardinality label can double your ingestion cost overnight. Audit label cardinality before enabling any new exporter.

2. Alert rules with no for duration on fast-moving metrics.
rate(node_cpu_seconds_total[1m]) > 0.90 without a for: PT5M clause fires on momentary spikes during deployments and generates alert fatigue within days. Every severity-1 alert should have a for window that reflects the actual time-to-customer-impact for that signal.

3. Forgetting the kube-system namespace exclusion in cost-sensitive configs.
The managed agent scrapes kube-system by default, including noisy control-plane metrics. If you're cost-optimising, add a metricsToDrop list in the ama-metrics-settings-configmap to filter metrics you'll never query.

4. Using the Grafana admin password for automation.
Azure Managed Grafana enforces Azure AD authentication. Service principals and managed identities should be granted the Grafana Editor or Grafana Viewer role via RBAC — never use the local admin account for CI/CD pipelines.

5. No recording rules on cross-namespace aggregations.
sum(container_memory_working_set_bytes) by (namespace) computed at dashboard load time across 500+ pods is slow and wastes query compute. Record it at 1-minute resolution and query the recording rule instead.

---

Verifying the Full Pipeline

Run this end-to-end health check once your configuration is complete:

#!/usr/bin/env bash
# verify-observability.sh

set -euo pipefail

echo "=== 1. ama-metrics agent health ==="
kubectl get pods -n kube-system -l rsName=ama-metrics \
  --no-headers | awk '{print $1, $3}' | column -t

echo ""
echo "=== 2. Recent remote_write errors (should be 0) ==="
kubectl logs -n kube-system -l rsName=ama-metrics --tail=50 \
  2>/dev/null | grep -i "remote_write\|error\|failed" || echo "No errors found"

echo ""
echo "=== 3. PrometheusRuleGroup status ==="
az resource list \
  --resource-type "Microsoft.AlertsManagement/prometheusRuleGroups" \
  --resource-group $RESOURCE_GROUP \
  --query "[].{name:name, state:properties.provisioningState}" \
  --output table

echo ""
echo "=== 4. Spot-check a metric exists in the workspace ==="
az monitor metrics list \
  --resource $AMW_ID \
  --metric "kube_node_status_condition" \
  --output table 2>/dev/null | head -5 \
  || echo "Metric not yet available — wait 2 minutes after agent start"

echo ""
echo "=== 5. Grafana endpoint reachable ==="
curl -sI "$GRAFANA_ENDPOINT" | head -1

A clean run produces no remote_write errors, shows Succeeded provisioning state for the rule group, and returns rows from the metric query.

---

Key Takeaways

• Managed Prometheus removes the scraper maintenance burden but doesn't eliminate the need for deliberate label design and cardinality control. Treat metric naming as an API.
• Alert rules evaluated server-side (in the Azure Monitor Workspace) are more reliable than in-cluster Alertmanager for cluster-level signals — they survive node failures and cluster upgrades.
• PodMonitor CRDs are the correct mechanism for application metric scraping. The prometheus.io/scrape annotation route works but bypasses structured selector logic and gets harder to audit as workload count grows.
• Recording rules are not optional at production scale. Pre-aggregate anything that appears in a dashboard panel or a cross-namespace alert expression.
• Grafana RBAC via Azure AD keeps your observability plane inside the same identity boundary as your cluster. Don't create a shadow identity system with Grafana local accounts.

---

What to Do This Week

1. Enable Managed Prometheus on your dev cluster first and review the default scrape targets with kubectl port-forward before touching production.
2. Open the Azure Monitor Workspace metrics explorer and run count by(__name__) to see your current time series count — this is your cardinality baseline.
3. Deploy the three alert rules from the Bicep template above; tune the thresholds against your last 30-day metric history before wiring them to PagerDuty.
4. Import dashboard 15520 and replace the hardcoded namespace filters with template variables ($namespace) so the dashboard is reusable across environments without duplication.
5. Set a workspace ingestion cap in the Azure Monitor Workspace Overview → Data cap blade to prevent a cardinality explosion from triggering an unexpected bill.

Related Articles

• Azure Kubernetes Service Cost Optimization: Right-Sizing Node Pools and Autoscaling Policies
• Kubernetes Pod Disruption Budgets: Preventing Downtime During Node Drains and Upgrades
• Kubernetes Admission Controllers: Enforcing Policy with OPA Gatekeeper vs Kyverno