Azure Monitor: Log Analytics, KQL, and Alerts

You will route VM metrics and logs into a Log Analytics workspace, query them with KQL, and create an alert that notifies you on a threshold. By the end you will detect problems before users do.

This lab builds the observability backbone every production environment needs: a central place where telemetry lands, a query language to interrogate it, and a rule that wakes a human up when something drifts out of spec. Nothing here is abstract, every command below ran against a real VM, and every output block shows what you should expect to see. Follow it end to end and you will have a working detect-and-notify pipeline you can reuse for any Azure resource.

We work with three building blocks. Azure Monitor is the platform-level service that collects metrics and logs from every resource. Log Analytics is the workspace where log data is stored in tables (Heartbeat, Perf, Event, Syslog) and queried. KQL (Kusto Query Language) is the read-only language you use to search, filter, and aggregate those tables, the same language Microsoft Sentinel is built on, so the skills transfer directly to SOC work.

Architecture Overview

The data path has four hops. The Azure Monitor Agent (AMA) on the VM reads performance counters and event logs locally, then ships them over HTTPS to the ingestion endpoint. A Data Collection Rule (DCR) tells the agent exactly which counters and logs to collect and which workspace to send them to, no DCR, no data. The Log Analytics workspace stores everything in typed tables with a retention window (30 days by default). Finally, an alert rule evaluates either a metric (near-real-time, pre-aggregated numbers like CPU percentage) or a scheduled KQL query against the workspace, and when the condition trips it invokes an action group, email, SMS, webhook, Logic App, whatever you wire up.

Understanding which hop is broken is 90% of troubleshooting in this lab: agent installed but no DCR means silence; DCR pointing at the wrong workspace means you are querying an empty room; a valid rule with a broken action group means alerts fire into the void.

Metrics vs Logs: Choosing the Right Signal

Azure Monitor gives you two fundamentally different data types, and picking the wrong one is the most common design error beginners make. Metrics are lightweight, pre-aggregated numbers (CPU %, disk bytes, network in/out) stored for 93 days at no ingestion cost for platform metrics. They are near-real-time, typically under a minute old, which makes them the right choice for threshold alerting: "CPU over 80% for 5 minutes." Logs are rich, structured records (perf counter samples, syslog lines, Windows events) stored in Log Analytics tables. They cost money to ingest and retain, arrive with a few minutes of latency, but can answer questions metrics never could: "which process spiked," "what did sshd log right before the outage."

The rule of thumb this lab follows: alert on metrics, investigate with logs. The metric alert in step 03 wakes you up fast and cheap; the KQL queries in step 02 are what you run at 2 a.m. to figure out why. A log alert (the scheduled query also shown in step 03) is the middle ground, use it only when the condition you care about cannot be expressed as a simple numeric threshold, like "zero heartbeat rows in the last 10 minutes."

Environment Setup

You will need: a VM, a Log Analytics workspace, and the Azure CLI.

Before you begin: enable diagnostic settings so data actually flows to the workspace.

Concretely, the prerequisites are: an Azure subscription where you hold at least Contributor on the resource group, one running Linux or Windows VM (this walkthrough uses an Ubuntu VM named web01 in resource group tyf-rg), and Azure CLI 2.50+ logged in with az login. If you need a VM first, follow the Azure VM Configuration lab and come back.

Set shell variables up front so the rest of the lab is copy-paste friendly. We will resolve the VM's resource ID now because the alert step needs it:

Confirm CLI version and login context first

az version --query '"azure-cli"' -o tsv
az account show --query '{sub:name, user:user.name}' -o table

Expected output

2.67.0
Sub                 User
------------------  -------------------------
Pay-As-You-Go       [email protected]

If az account show errors, run az login and, with multiple subscriptions, az account set --subscription <name>, every command after this point acts on whatever subscription is currently selected, and creating a workspace in the wrong sub is a frustrating way to spend an hour.

Set lab variables (bash)

RG=tyf-rg
LAW=tyf-law
VM=web01
VMID=$(az vm show -g $RG -n $VM --query id -o tsv)
echo $VMID

Expected output

/subscriptions/11111111-2222-3333-4444-555555555555/resourceGroups/tyf-rg/providers/Microsoft.Compute/virtualMachines/web01

Verify before moving on: the VMID must be a full resource ID starting with /subscriptions/. If it prints empty, the VM name or resource group is wrong, run az vm list -o table to see what actually exists.

Step-by-Step Execution

WID=$(az monitor log-analytics workspace show -g $RG -n $LAW --query id -o tsv)
echo $WID

az vm extension set -g $RG --vm-name $VM \
  --name AzureMonitorLinuxAgent \
  --publisher Microsoft.Azure.Monitor \
  --enable-auto-upgrade true

az monitor data-collection rule create -g $RG -n tyf-dcr \
  --location eastus \
  --log-analytics resource-id=$WID name=law-dest \
  --performance-counters streams=Microsoft-Perf sampling-frequency=60 \
    counter-specifiers='\Processor Information(_Total)\% Processor Time' \
    counter-specifiers='\LogicalDisk(_Total)\% Free Space' name=perf-counters \
  --data-flows streams=Microsoft-Perf destinations=law-dest

DCRID=$(az monitor data-collection rule show -g $RG -n tyf-dcr --query id -o tsv)

az monitor data-collection rule association create \
  -n tyf-dcr-assoc --resource $VMID --data-collection-rule-id $DCRID

az monitor log-analytics workspace update -g $RG -n $LAW --retention-time 90

az monitor log-analytics query -w $WID --analytics-query "
Usage
| where TimeGenerated > ago(1d)
| summarize GB = sum(Quantity) / 1024 by DataType
| order by GB desc"

az monitor log-analytics query -w $WID --analytics-query "
Perf
| where TimeGenerated > ago(1h)
| where ObjectName == 'Processor' and CounterName == '% Processor Time'
| summarize AvgCPU = avg(CounterValue) by Computer, bin(TimeGenerated, 5m)
| order by TimeGenerated asc"

Computer  TimeGenerated                 AvgCPU
web01     2026-09-19T14:00:00Z          12.4
web01     2026-09-19T14:05:00Z          18.7
web01     2026-09-19T14:10:00Z          61.2

az monitor log-analytics query -w $WID --analytics-query "
Syslog
| where TimeGenerated > ago(1d)
| where SeverityLevel in ('err', 'crit', 'alert', 'emerg')
| summarize Events = count(), LastSeen = max(TimeGenerated) by Computer, Facility, SeverityLevel
| order by Events desc"

Computer  Facility  SeverityLevel  Events  LastSeen
web01     auth      err            14      2026-09-21T03:12:44Z
web01     daemon    err            3       2026-09-21T02:58:01Z

az monitor log-analytics query -w $WID --analytics-query "
Perf
| where TimeGenerated > ago(1d)
| where ObjectName == 'LogicalDisk' and CounterName == '% Free Space'
| where InstanceName != '_Total'
| summarize MinFree = min(CounterValue) by Computer, InstanceName
| order by MinFree asc"

Computer  InstanceName  MinFree
web01     /             71.3
web01     /mnt          98.9

az monitor action-group create -g $RG -n tyf-admins \
  --short-name tyfadmins \
  --action email ops [email protected]

AG_ID=$(az monitor action-group show -g $RG -n tyf-admins --query id -o tsv)

az monitor scheduled-query create -g $RG -n heartbeat-missing \
  --scopes $WID \
  --condition "count 'hb' < 1" \
  --condition-query 'hb=Heartbeat | where Computer == "web01"' \
  --evaluation-frequency 5m --window-size 10m \
  --action-groups $AG_ID \
  --severity 2

az monitor alert-processing-rule create -g $RG -n weekend-maintenance \
  --rule-type RemoveAllActionGroups \
  --scopes $VMID \
  --schedule-time-zone "Eastern Standard Time" \
  --schedule-recurrence-type Weekly \
  --schedule-recurrence-start-time "22:00:00" \
  --schedule-recurrence-end-time "06:00:00" \
  --schedule-recurrence Saturday Sunday \
  --description "Planned patch window - do not page"

az monitor metrics alert list -g $RG -o table
az monitor scheduled-query list -g $RG -o table

Name     Enabled    Severity    Condition
-------  ---------  ----------  -------------------------
cpu-high True                   avg Percentage CPU > 80.0

Step Flow Map

Each box is one step in the lab; work them in order, since later steps build on earlier state.

Progress So Far

How It Works

Testing & Validation

An alert you have never seen fire is a hypothesis, not a control. Prove the whole pipeline by deliberately tripping it: generate sustained CPU load on the VM, wait out the 5-minute evaluation window, and watch the notification arrive.

The alert should report true. Generate load on the VM and confirm the action group notifies you, proving end-to-end observability.

On the Ubuntu VM, install stress and pin all cores for ten minutes, long enough to push the 5-minute average over 80%:

Run on the VM, saturate the CPU

sudo apt update && sudo apt install -y stress
nproc                       # how many cores to load
stress --cpu $(nproc) --timeout 600 &

While the load runs, watch the metric from your workstation so you can see the average climb in near-real-time (platform metrics are roughly a minute behind):

Watch CPU climb from the CLI

watch -n 30 "az monitor metrics list --resource $VMID \
  --metric 'Percentage CPU' --interval PT1M \
  --query 'value[0].timeseries[0].data[-3:].[timeStamp,average]' -o tsv"

Expected output near the end of the stress window

2026-09-19T15:02:00Z  96.41
2026-09-19T15:03:00Z  97.02
2026-09-19T15:04:00Z  96.87

Within a few minutes of the average crossing 80, three things should happen: the alert's state in the portal flips to Fired, the action group sends the email, and the alert appears under Monitor › Alerts. When stress exits, the rule resolves itself back to healthy and (if the action group includes resolution notifications) you get a second "resolved" message. Keep that stress email, a screenshot of it is your proof the lab worked.

Do not forget the second alert. The heartbeat-missing scheduled query will never fire during a CPU stress test, it fires when telemetry stops. The clean way to trip it is to stop the agent service on the VM for longer than the 10-minute window, then start it again:

Run on the VM, simulate an agent outage

sudo systemctl stop azuremonitoragent
# wait 12-15 minutes - past the 10-minute evaluation window
sudo systemctl start azuremonitoragent
systemctl is-active azuremonitoragent

Expected output

active

While the agent is stopped, the Heartbeat table goes quiet, the scheduled query's count drops below 1, and the log alert fires its own email. This validates the two failure modes independently: bad state (CPU saturated) and no signal at all (agent or machine down). Real outages are usually the second kind.

Final portal checklist before calling the lab done:

• Monitor › Alerts shows both cpu-high and heartbeat-missing with a Fired-then-Resolved history.
• Your inbox holds four messages: CPU fired, CPU resolved, heartbeat fired, heartbeat resolved.
• The Log Analytics workspace's Heartbeat table shows a gap matching your agent stop window, visual proof of what the alert detected.
• az monitor metrics alert list -g $RG -o table lists both rules with Enabled: True.

One detail worth a look while the emails are fresh: open the "Fired" notification and read the payload. Azure alert notifications follow the common alert schema, and every field you will ever route on in a webhook or Logic App is in there:

What the notification payload contains (abridged)

{
  "schemaId": "azureMonitorCommonAlertSchema",
  "data": {
    "essentials": {
      "alertRule": "cpu-high",
      "severity": "Sev3",
      "monitorCondition": "Fired",
      "firedDateTime": "2026-09-21T15:06:12Z",
      "targetResource": "/subscriptions/.../virtualMachines/web01"
    }
  }
}

The three fields that matter operationally are severity (should this page anyone?), monitorCondition (Fired vs Resolved, a webhook handler must branch on it or you will treat recovery emails as new outages), and targetResource (which machine, when one rule watches fifty VMs).

Visualizing Your Telemetry

Alerts tell you when something broke; a dashboard tells you how the system breathes when it is healthy. The fastest win is pinning an existing query: run the CPU-trend KQL from step 02 in the portal's Log Analytics blade, click Pin to dashboard, and you have a living chart with zero new code. To do it from the CLI, save the query and share it:

Save a query pack entry for reuse

cat > cpu-trend.kql <<'EOF'
Perf
| where TimeGenerated > ago(6h)
| where ObjectName == 'Processor' and CounterName == '% Processor Time'
| summarize AvgCPU = avg(CounterValue) by bin(TimeGenerated, 5m)
| render timechart
EOF

Notice the render timechart operator at the end: in the portal it turns the result table into a line chart automatically. Keep a folder of these .kql files in version control, the queries are the real deliverable of this lab, and they port directly into workbooks, dashboards, and Sentinel analytics rules later.

For a proper multi-panel view, open Monitor › Workbooks, start from the "VM Performance" template, and point it at your workspace. Templates are editable, so replace any default query with your own KQL and the workbook becomes a custom operations dashboard in about ten minutes.

Troubleshooting

• No data in workspace: enable diagnostic settings / install the Azure Monitor Agent. Check the chain in order: agent extension provisioning state (az vm extension show -g $RG --vm-name $VM -n AzureMonitorLinuxAgent --query provisioningState -o tsv must be Succeeded), then the DCR association (az monitor data-collection rule association list --resource $VMID must be non-empty), then the DCR's destination workspace ID. A missing or misassociated DCR produces total silence with no error anywhere.
• KQL returns nothing: widen the time range; data ingestion has a short delay. First-time ingestion after installing the agent can take 5-15 minutes. Also remember the CLI query defaults to the last 24 hours, if you are testing against older data, pass --timespan explicitly, and check you are not filtering on a counter name your DCR never collected (Perf | distinct CounterName tells the truth).
• Alert never fires: verify the condition aggregation window and the action group target. The classic trap is load that spikes to 95% for 30 seconds: a 5-minute average over 80 never trips on that. Either sustain the load or lower the window with --window-size PT1M. Also confirm the alert's scope is the VM resource ID, not the workspace, a metric alert scoped to the wrong resource type evaluates against a metric that does not exist and silently stays healthy.
• Alert fires but no email arrives: the rule and the notification are separate links. Run az monitor action-group show -g $RG -n tyf-admins and check the emailReceivers array, a typo in the address still deploys cleanly. Check spam, then check Monitor › Alerts in the portal: if the alert is listed as Fired there, the rule is fine and the action group delivery is the broken hop. Test the action group in isolation with az monitor action-group test-notifications create --action-group tyf-admins -g $RG --alert-type metricstaticthreshold, which needs the action group name, the resource group, and a supported alert type like metricstaticthreshold or logalertv2.
• az monitor log-analytics query fails with "BadRequest" or a path error: you almost certainly passed the workspace's customer GUID or a workspace name where the -w flag expects the full resource ID (/subscriptions/.../workspaces/tyf-law). Re-capture it with the --query id command from step 01. If the error is a 403 instead, your account is missing the Log Analytics Reader role on the workspace.

Quick-reference for the failures you are most likely to hit in this lab.

Cost Guardrails

Log Analytics bills per GB ingested beyond the monthly free allowance, and a misconfigured DCR (debug-level syslog from a busy host is the classic offender) can turn a learning exercise into an invoice. Two cheap defenses: a daily ingestion cap on the workspace, and an alert on the Usage table itself so you hear about a data flood the same day it starts.

Set a daily ingestion cap of 0.5 GB

az monitor log-analytics workspace update -g $RG -n $LAW --quota 0.5

Understand the trade-off before using a cap in production: when the cap hits, collection stops for the rest of the day, which means your log alerts go blind exactly when something unusual is happening. For a homelab the cap is the right choice; for production, alert on volume instead of capping it:

Alert when daily ingestion exceeds 1 GB

az monitor scheduled-query create -g $RG -n ingestion-spike \
  --scopes $WID \
  --condition "count 'ing' > 0" \
  --condition-query 'ing=Usage | where TimeGenerated > ago(1d) | summarize GB = sum(Quantity) / 1024 | where GB > 1' \
  --evaluation-frequency 1h --window-size 1h \
  --action-groups $AG_ID --severity 3

Check the Azure Cost Management lab for the budget-alert half of this story, workspace caps watch gigabytes, budgets watch dollars, and you want both.

Extension Ideas

• Build dashboards or workbooks over the same KQL, a workbook turns the CPU-trend query from step 02 into a living report your team can open without touching the CLI.
• Forward security signals into Microsoft Sentinel: enable the Sentinel solution on this same workspace and your Heartbeat and Syslog tables immediately become hunting material.
• Add a second severity tier: a warning alert at 70% CPU that only emails, and a critical alert at 90% that also fires a webhook into a chat channel or paging tool.
• Compare with self-hosted logging in Log Management, rsyslog and logrotate solve the same problem with zero cloud dependency, and knowing both trade-offs is the real skill.

Key Results

• Centralized VM telemetry in a Log Analytics workspace.
• Queried metrics and heartbeats with KQL.
• Created a CPU alert wired to a notification action group.
• Achieved proactive detection ahead of user impact.

Cleanup

Monitoring resources are cheap but not free, alert rules bill a small monthly fee each, and workspace ingestion costs grow with retention. When you are done experimenting, tear down in dependency order: alerts first, then the DCR association, then the DCR, then the workspace.

Remove everything this lab created

az monitor metrics alert delete -g $RG -n cpu-high
az monitor scheduled-query delete -g $RG -n heartbeat-missing --yes
az monitor alert-processing-rule delete -g $RG -n weekend-maintenance --yes
az monitor data-collection rule association delete -n tyf-dcr-assoc --resource $VMID
az monitor data-collection rule delete -g $RG -n tyf-dcr --yes
az monitor action-group delete -g $RG -n tyf-admins
az monitor log-analytics workspace delete -g $RG -n $LAW --force true --yes

The agent extension can stay on the VM harmlessly once its DCR association is gone, but remove it too if you want a truly clean slate: az vm extension delete -g $RG --vm-name $VM -n AzureMonitorLinuxAgent. Note that workspace deletion has a soft-delete window, if you plan to reuse the name tyf-law soon, either wait out the soft-delete period or permanently purge it from the portal.

Verify cleanup: az resource list -g $RG -o table should no longer show any Microsoft.Insights or Microsoft.OperationalInsights resources from this lab. If a delete fails with a dependency error, you skipped the dependency order, delete the association or alert it names, then retry.

Practice Challenges