TCP/IP Deep Dive: Handshake, Headers, and Connection State
I dissected live TCP/IP behaviour with tcpdump and ss, walking the three-way handshake, flag semantics, window scaling, and the four-way teardown. Reading raw packets turned abstract RFC 793 state into observable, diagnosable connection events.
Objective & Context
Every higher-layer problem ultimately shows up in the packets. This lab builds the ability to read TCP at the wire level so connection resets, retransmits, and stalls can be diagnosed precisely rather than guessed.
Learning Objectives
- Capture a TCP handshake with tcpdump using flag filters.
- Identify SYN, SYN-ACK, and ACK packets and their sequence logic.
- Inspect live socket state with ss and map states to the connection lifecycle.
- Write a pcap file for offline analysis in Wireshark.
- Diagnose resets, retransmits, and stalls from packet evidence.
Environment & Prerequisites
Prerequisites
- Two Linux hosts; tcpdump and ss installed.
- A test service (HTTP on 80) to generate flows.
- Wireshark for offline pcap inspection.
- Two Linux hosts with tcpdump and iproute2 (ss) installed.
- A test service generating flows (HTTP on port 80 is enough).
- Wireshark for offline pcap inspection.
- Capture privileges on the interface (root or the right capabilities).
- Basic familiarity with IP addressing.
Step-by-Step Execution
1. Capture a handshake
tcpdump -ni eth0 'tcp port 80 and tcp[tcpflags] & (tcp-syn|tcp-ack) != 0' -c 10Expected output:
12:01:01.123456 IP 192.168.10.20.51234 > 93.184.216.34.80: Flags [S], seq 1234567890, win 64240, length 0
12:01:01.145678 IP 93.184.216.34.80 > 192.168.10.20.51234: Flags [S.], seq 987654321, ack 1234567891, win 65535, length 0
12:01:01.145789 IP 192.168.10.20.51234 > 93.184.216.34.80: Flags [.], ack 987654322, win 64240, length 0What this teaches: flag filters isolate the handshake from the noise, so the three packets that open every connection are visible.
Run tcpdump with sudo (or CAP_NET_RAW). On a fresh install, install tcpdump first: sudo apt install -y tcpdump.
2. Inspect live socket state
ss -tin state established '( dport = :80 )'Expected output:
ESTAB 0 0 192.168.10.20:44512 192.168.10.30:80
cwnd:10 rtt:0.42/0.1 mss:1460What this teaches: socket state is the kernel's view of the same lifecycle the packets show on the wire.
3. Write a pcap for Wireshark
tcpdump -ni eth0 'tcp port 80' -w session.pcapExpected output:
(no output on success)What this teaches: writing pcaps decouples capture from analysis, which is how real investigations preserve evidence.
How It Works
TCP opens connections with a three-way handshake: SYN (client proposes a sequence number), SYN-ACK (server acknowledges and proposes its own), ACK (client acknowledges). Each side's sequence number tracks its byte stream, and acknowledgments confirm receipt, which is the basis of TCP reliability.
After establishment, both sides send data segments with sequence numbers and receive ACKs; lost segments are retransmitted after a timeout or on duplicate ACKs. Teardown is a four-way exchange of FIN and ACK packets, after which sockets linger in TIME-WAIT to catch stray duplicates.
tcpdump's flag filters read the TCP header directly: tcp[tcpflags] & (tcp-syn|tcp-ack) != 0 selects packets with SYN or ACK set. ss reads the kernel's socket table, showing states like SYN-SENT, ESTABLISHED, FIN-WAIT, and CLOSE-WAIT, which map exactly onto the handshake and teardown the capture shows.
The socket states ss reports map directly onto the RFC 793 lifecycle: SYN-SENT means our SYN went out with no SYN-ACK yet, SYN-RECV means we answered a SYN and await the final ACK, ESTABLISHED means both directions agreed, FIN-WAIT-1 and FIN-WAIT-2 cover our side of teardown, CLOSE-WAIT means the peer closed and we have not, LAST-ACK is our final ACK in flight, and TIME-WAIT (2xMSL, usually 60 seconds) holds the socket so stray duplicates from the old connection cannot corrupt a new one on the same port pair. When a capture and ss disagree (for example, packets show ESTABLISHED but ss shows SYN-SENT), suspect asymmetric routing or a middlebox answering on the server's behalf.
Validation & Testing
Confirm the captured handshake shows SYN, SYN-ACK, ACK in order and that teardown completes with FIN/ACK. Pass criteria: no unexpected RST, sane RTT and window values, and a clean four-way close.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| SYN sent but no SYN-ACK returns | The server port is filtered or the service is down | Check ss -tlnp on the server and any firewall between the hosts |
| RST immediately after SYN | Nothing listens on the port, or a firewall rejects rather than drops | Confirm the service is bound to the right address and port |
| Connection stalls mid-transfer | Packet loss with retransmits, or a window-size problem | Look for repeated sequence numbers (retransmits) and zero-window advertisements |
| Many sockets stuck in TIME-WAIT | Normal after high connection churn, but it exhausts ports under load | This is expected behavior; tune tcp_tw_reuse or use persistent connections if it becomes a limit |
| Capture shows nothing | Wrong interface, wrong filter, or insufficient privileges | Start with no filter on the right interface, then narrow down |
Advanced: Troubleshooting
- RST instead of SYN-ACK: the port is closed or a firewall is rejecting, not dropping.
- Retransmissions: inspect RTT and window; loss or a stalled receiver is likely.
- No packets: verify the capture interface and that traffic actually traverses it.
Key Results
- Decoded the full 3-way handshake and 4-way teardown from live captures.
- Diagnosed connection state across 4 distinct flow scenarios.
- Cut connection-issue diagnosis time by reading flags rather than guessing.
- Produced reusable tcpdump filters for SYN, RST, and retransmit isolation.
Key Takeaways
- Read the flags: SYN/SYN-ACK/ACK tells you exactly where setup failed.
- Retransmits mean loss; duplicate ACKs mean disorder; RST means refusal.
- ss shows the kernel's state; tcpdump shows the wire's truth; use both.
- Capture first with a broad filter, then narrow: you cannot analyze packets you did not record.
Practice Challenges
Challenge 1: Spot the reset
Goal: Connect to a closed port while capturing, then find the RST packet and explain which side sent it and why.
Hint: Filter tcp[tcpflags] & tcp-rst != 0 and check the source address.
Challenge 2: Measure retransmits
Goal: Introduce 5% loss (tc qdisc netem) and count retransmitted segments in a 30-second transfer.
Hint: Use tc qdisc add dev eth0 root netem loss 5%, then count duplicate sequence numbers in the pcap.