Concept Focus: the OSI model
Security engineering across a network only makes sense once you can name where a problem lives. The OSI model gives you seven layers to reason in, each with its own job, its own unit of data, and its own class of attacks.
| Layer | Unit (PDU) | What it does |
|---|---|---|
| 7. Application | Data | User-facing protocols and semantics (HTTP, SMTP, DNS); where business logic and most app security issues live (XSS, SQLi) |
| 6. Presentation | Data | Formatting, compression, encryption; turns app data into an on-the-wire form (TLS records, character sets, serialization) |
| 5. Session | Data | Starts, tracks, and ends dialogues between endpoints; checkpoints, resumes, half-open cleanups |
| 4. Transport | Segment / Datagram | End-to-end delivery: sequencing, acknowledgments, flow and congestion control, ports |
| 3. Network | Packet | Logical addressing and routing between networks; chooses a path, fragments if needed (IP datagrams) |
| 2. Data Link | Frame | Hop-to-hop delivery on a single link: MAC addressing, framing, error detection (CRC), medium access |
| 1. Physical | Bits | Transmits raw bits as voltage, light, or radio; connectors, wavelengths, modulation, timing |
Deep dive, one layer at a time
7. Application
Provides networked services to applications and users, defining request and response semantics, headers, and data models. A browser builds an HTTP request, a mail client crafts SMTP commands, a resolver sends a DNS query. Typical protocols: HTTP/HTTPS, SMTP/IMAP/POP3, DNS, DHCP, SNMP, NTP, SSH, SIP. Security gotchas: input validation failures (SQLi, XSS), cookie and session mishandling, weak auth flows, sensitive data exposure. In TCP/IP, Application often subsumes Presentation and Session, but the OSI split is still useful for precise thinking.
6. Presentation
Makes data portable and safe in transit via formatting, compression, and encryption. Data is serialized (JSON, ASN.1, Protobuf), possibly compressed, then encrypted and authenticated (TLS), producing a byte stream with its own record boundaries. Typical: TLS/SSL, MIME, XDR, ASN.1, Base64. Gotchas: deprecated ciphers or protocol versions, certificate validation errors, SNI/ALPN mismatches. People often say “TLS is L4,” but TLS wraps the app stream before TCP moves it, which is very Presentation in OSI terms.
5. Session
Creates, maintains, and gracefully closes conversations between endpoints, coordinating checkpoints and resumes. In modern stacks this is frequently implemented by the app framework or folded into Transport and Application. Historical: NetBIOS Session Service, RPC bind, PPTP control. Modern analogues: WebSocket handshake and control frames, TLS session tickets. Gotchas: session fixation, token replay, idle timeout misconfigurations.
4. Transport
Delivers bytes end-to-end with ordering, reliability, and rate control. TCP does a three-way handshake (SYN, SYN-ACK, ACK), then transfers data with sequence numbers, ACKs, retransmissions, flow control (windowing), and congestion control. UDP is fire-and-forget. Ports identify application endpoints. Typical: TCP, UDP (plus SCTP, DCCP). Gotchas: blind RST/FIN injection, SYN floods, reflection and amplification with UDP. TLS sits above this to encrypt and authenticate streams.
3. Network
Moves packets between networks using logical addresses and routing. IPv4/IPv6 add source and destination IPs, TTL or Hop-Limit, and flags; routers consult routing tables to forward each hop. If a packet exceeds a link's MTU, IPv4 may fragment while IPv6 expects endpoints to size correctly; Path MTU Discovery relies on ICMP to signal when to shrink. Typical: IPv4/IPv6, ICMP, IPsec AH/ESP, ARP/ND, plus routing protocols OSPF, BGP, IS-IS, EIGRP. Gotchas: ICMP filtering that breaks PMTUD, BGP route leaks and hijacks, IP spoofing, weak IPsec policies.
2. Data Link
Delivers frames hop-to-hop on a single link, arbitrates who speaks, and detects errors. Frames get source and destination MACs, VLAN tags (802.1Q), and a Frame Check Sequence (CRC). Switches learn MAC tables and segment collision domains; STP/RSTP prevents L2 loops; ARP maps IP to MAC. Typical: Ethernet (802.3), Wi-Fi (802.11), PPP, HDLC, VLANs, STP/RSTP/MSTP, LACP. Gotchas: ARP spoofing, rogue DHCP, MAC flooding, VLAN hopping. Mitigations: DHCP snooping, dynamic ARP inspection, port security, correct trunk and native VLAN configs.
1. Physical
Turns bits into signals and back, reliably and on time: encoding, modulation, timing, connector pinouts. Media define distance and speed limits and noise behaviour. Copper: 1000BASE-T, 10GBASE-T, UTP/STP categories. Fiber: 10GBASE-SR/LR/ER, single-mode (long), multimode (short). Wireless: 802.11 a/b/g/n/ac/ax, cellular, microwave. Gotchas: evil-maid taps, exposed patch panels, and cheap cables causing errors that masquerade as app issues.
How data moves, encapsulation and decapsulation
Going down the stack, the Application creates a message, the Presentation layer may serialize, compress, and encrypt it, the Session layer labels the dialogue, the Transport layer segments the stream and adds ports and sequence logic, the Network layer wraps each segment with IP headers, the Data Link layer encapsulates packets into frames with MACs and a CRC, and the Physical layer turns frames into bits. At the receiver the reverse happens, bit by bit up to the application, exactly as written.
| Layers | Data unit (PDU) |
|---|---|
| L7 / L6 / L5 | Data or Message |
| L4 | Segment (TCP) or Datagram (UDP) |
| L3 | Packet (IP datagram) |
| L2 | Frame |
| L1 | Bits |
Where TCP/IP maps onto OSI
- Application (OSI 7 to 5) maps to TCP/IP Application: HTTP, DNS, SMTP, SSH, DHCP, SNMP.
- Transport (OSI 4) maps to TCP/IP Transport: TCP, UDP.
- Internet (TCP/IP) is roughly Network (OSI 3): IPv4/IPv6, ICMP, IPsec, routing protocols.
- Link and Physical (TCP/IP) are roughly OSI 2 and 1: Ethernet, Wi-Fi, PPP, cabling, optics, radio.
You fix things with TCP/IP knobs, but you diagnose with OSI.
Protocols by layer
- L7 Application: HTTP/S, DNS, SMTP/IMAP/POP3, SSH, FTP/SFTP, DHCP, SNMP, NTP, SIP.
- L6 Presentation: TLS/SSL, MIME, ASN.1, XDR, compression (DEFLATE/GZIP), Base64.
- L5 Session: NetBIOS Session, RPC session, PPTP control; app-level sessions, WebSocket control.
- L4 Transport: TCP, UDP (plus SCTP, DCCP).
- L3 Network: IPv4/IPv6, ICMP, IPsec (AH/ESP), ARP/ND, OSPF, BGP, IS-IS, EIGRP.
- L2 Data Link: Ethernet, Wi-Fi, PPP, HDLC, VLANs (802.1Q), STP/RSTP/MSTP, LACP.
- L1 Physical: 1000BASE-T/10GBASE-T, 10GBASE-SR/LR, optics, UTP/STP categories, 802.11 PHYs.
Brain Ticklers
Q1. Large file downloads fail midway after a WAN change, while small API calls work. TCP handshakes succeed and TLS negotiates fine. What should Anya check first?
- DNS TTLs for the new provider
- Path MTU and ICMP “Fragmentation Needed” behaviour; clamp TCP MSS if needed
- Regenerate TLS certificates with ECDSA
- Increase HTTP keepalive timeout
Q2. A red team pulls off ARP spoofing and session hijacking on a flat office LAN. Which control sits at the most relevant OSI layer to blunt this class of attack?
- WAF rules for cookie signatures (L7)
- Disallow TLS 1.0 (L6)
- DHCP snooping + Dynamic ARP Inspection + port security on switches (L2)
- Enforce 10GBASE-SR optics (L1)
Q3. Captures show repeated TCP retransmissions and a sawtooth throughput pattern despite zero application errors. What best explains this?
- Session timeouts at L5
- TCP congestion control and receive window dynamics at L4
- VLAN mis-tagging at L2
- DSCP remarking at L3
Q4. A review demands “encrypt data in transit.” Where, in OSI terms, is this most commonly implemented for web apps without changing routing or switching?
- L3 IPsec tunnel mode only
- L6 TLS between client and server
- L2 MACsec across access switches
- L1 optical layer encryption
Q5. A site-to-site VPN works, but certain apps break when traversing a provider who blocks ICMP. Which OSI concept is most likely impacted?
- Session resumption tickets (L5)
- Path MTU Discovery signaling (L3)
- Port exhaustion (L4)
- HTTP header normalization (L7)