Layered Network Troubleshooting with ping, mtr, and ss
I built a layered troubleshooting methodology that isolates faults bottom-up across the OSI model using ping, traceroute, mtr, and ss. Working from link to application layer localizes a fault to a specific layer in minutes instead of guessing.
Objective & Context
Random tool-spraying wastes time. This lab imposes a disciplined order, physical/link, then IP reachability, then routing path, then transport and DNS, so each test eliminates a layer and narrows the fault deterministically.
Learning Objectives
- Apply a disciplined bottom-up order: link, IP, path, transport, application.
- Verify link state and local addressing with ip commands.
- Trace and interpret the path with mtr, distinguishing real loss from ICMP rate-limiting.
- Check transport state and DNS with ss and dig.
- Document findings so each test eliminates one layer.
Environment & Prerequisites
Prerequisites
- Linux host with iputils, mtr, traceroute, and iproute2.
- A reachable gateway, an internal host, and an external target.
- Access to interface and route state.
- Linux host with iputils-ping, mtr, traceroute, iproute2, and dnsutils installed.
- A reachable gateway, an internal host, and an external target for layered tests.
- Access to interface and route state (ip link, ip route).
- Basic understanding of the OSI model layers.
Step-by-Step Execution
1. Confirm link and IP reachability
ip -br link && ping -c3 192.168.10.1Expected output:
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0 UP 52:54:00:12:34:56 <BROADCAST,MULTICAST,UP,LOWER_UP>
PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.
64 bytes from 192.168.10.1: icmp_seq=1 ttl=64 time=0.412 ms
64 bytes from 192.168.10.1: icmp_seq=2 ttl=64 time=0.388 ms
64 bytes from 192.168.10.1: icmp_seq=3 ttl=64 time=0.401 ms
--- 192.168.10.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet lossWhat this teaches: confirming the link and the first hop eliminates the entire physical and local-IP layer in seconds.
2. Trace the path with mtr
mtr -rwzbc 20 1.1.1.1Expected output:
HOST Loss% Avg
1. 192.168.10.1 0.0% 0.4
2. isp-gw 0.0% 8.1
3. 1.1.1.1 0.0% 9.3What this teaches: mtr combines ping and traceroute so you can see exactly which hop starts dropping or delaying packets.
3. Check transport and DNS
ss -tunap | grep :443 ; dig +short tyfsadik.orgExpected output:
(ss prints nothing when no :443 sockets exist; the dig line is the DNS verdict)
203.0.113.10What this teaches: ss shows what the transport layer is actually doing, and dig separates DNS failures from connectivity failures.
Use a semicolon (or two commands) rather than &&: grep exits nonzero when no :443 socket exists, which would skip the DNS test exactly when you need it. ss -p needs sudo to show process names; without it the Process column stays blank.
How It Works
Bottom-up troubleshooting works because each layer depends on the one below it. If the link is down, no IP test can pass; if the gateway is unreachable, no path test matters; if DNS fails, the application fails even with perfect connectivity. Testing in dependency order means the first failing test points at the faulty layer.
ping tests IP reachability with ICMP echo, but many networks rate-limit or block ICMP, so a ping failure is a clue, not a verdict. mtr sends a continuous stream of probes with increasing TTLs and reports loss and latency per hop, which reveals whether loss starts at a specific router or is an artifact of ICMP policing.
ss reads socket state directly from the kernel: ESTABLISHED, SYN-SENT, or nothing at all tells you whether the transport handshake completed. dig queries DNS independently of the application, so comparing dig output against curl behavior isolates name resolution from the service itself.
Validation & Testing
Inject a fault (wrong gateway, blocked port, bad resolver) and confirm the methodology localizes it to the correct layer. Pass criteria: each scenario is isolated to one layer with a single confirming test.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| ping works but the application fails | DNS or transport-layer problem above IP | Check ss for socket state and dig for resolution before blaming the network |
| mtr shows 100% loss at one hop but the target responds | That hop rate-limits ICMP while forwarding normally | Trust the final hop's statistics; intermediate loss with a healthy target is cosmetic |
| Intermittent failures with no pattern | Flapping link, duplicate IP, or asymmetric routing | Watch ip -s link for error counters and check for duplicate addresses with arping |
| Everything works except one destination | A host route, firewall rule, or policy routing exception for that prefix | Run ip route get <dst> and check nftables/iptables for destination-specific drops |
| DNS resolves but connections hang | Firewall allows UDP 53 but blocks the TCP service port | Test with nc -zv <host> <port> to separate port filtering from service failure |
| ss shows the socket but no process name | ss -p was run without privileges, so the kernel hides process ownership | Rerun with sudo; combine with lsof -i :443 for a second opinion |
Advanced: Troubleshooting
- Ping works, app fails: the issue is transport/application; check ports with ss and firewall rules.
- Loss mid-path: mtr loss only at the final hop is often ICMP rate-limiting, not a real drop.
- Name fails, IP works: a DNS problem; test with dig against an alternate resolver.
Key Results
- Localized injected faults to a single OSI layer in under 5 minutes each.
- Standardized a 5-step bottom-up runbook used across incidents.
- Distinguished real packet loss from ICMP rate-limiting in mtr output.
- Reduced mean diagnosis time versus unstructured tool use.
Key Takeaways
- Always work bottom-up: link, IP, path, transport, then application.
- One failing test at the lowest failing layer is the diagnosis; everything above is noise.
- Distrust ICMP-only evidence; confirm with TCP-level tests.
- Write down each result; a runbook beats memory during an outage.
Practice Challenges
Challenge 1: Break it on purpose
Goal: Have a partner inject one fault (wrong gateway, blocked port, bad DNS) and localize it to a single layer in under 5 minutes.
Hint: Start at ip -br link and move up only when the current layer passes.
Challenge 2: Script the runbook
Goal: Write a bash script that runs the five checks in order and prints PASS/FAIL per layer.
Hint: Use ping -c1 -W1 for speed and check exit codes rather than parsing output.