What does Linux do with a lost TCP connection?

We can simulate a dropped TCP connection using two virtual machines. I have two machines, tcpserver and tcpclient. On tcpserver, I run nc -l 0.0.0.0 -p 12345, listening on TCP port 12345 for a new connection. Then from tcpclient, I run tcpdump to spy on our connection:

jim@tcpclient:~$ sudo tcpdump -i eth1 -n 'tcp port 12345'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes

From tcpclient, I start a new connection:

jim@tcpclient:~$ nc 192.168.33.10 12345
hello
hi

I immediately see the TCP three-way handshake:

14:47:25.710372 IP 192.168.33.11.60739 > 192.168.33.10.12345: Flags [S], seq 3041775903, win 29200, options [mss 1460,sackOK,TS val 4637 ecr 0,nop,wscale 6], length 0
14:47:25.710675 IP 192.168.33.10.12345 > 192.168.33.11.60739: Flags [S.], seq 2649418967, ack 3041775904, win 28960, options [mss 1460,sackOK,TS val 5194 ecr 4637,nop,wscale 6], length 0
14:47:25.710695 IP 192.168.33.11.60739 > 192.168.33.10.12345: Flags [.], ack 1, win 457, options [nop,nop,TS val 4637 ecr 5194], length 0

With netstat, I can see the TCP connection state on the client. It’s ESTABLISHED, and the TCP buffers are empty. All is healthy:

jim@tcpclient:~$ sudo netstat -tpn
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 192.168.33.11:60739     192.168.33.10:12345     ESTABLISHED 2060/nc

Next, I pause the tcpserver VM (in VirtualBox). This simulates the server dying, or the network dying, or the server cutting off the client uncleanly. tcpdump and netstat detect nothing! No data is being transferred, so all seems normal. Now send some bytes from the client connection, e.g. is anyone there?. nc gives no indication that the bytes are not sent, but we can see this from netstat ...

jim@tcpclient:~$ sudo netstat -tpn
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0     17 192.168.33.11:60739     192.168.33.10:12345     ESTABLISHED 2060/nc

The connection is still ESTABLISHED, but notice that the Send-Q has 17 bytes in it, i.e. the string "is anyone there?\n". With tcpdump, we can see the OS repeatedly trying to send these bytes:

14:53:10.384563 IP 192.168.33.11.60739 > 192.168.33.10.12345: Flags [P.], seq 22:39, ack 6, win 457, options [nop,nop,TS val 90806 ecr 87171], length 17
14:53:10.589005 IP 192.168.33.11.60739 > 192.168.33.10.12345: Flags [P.], seq 22:39, ack 6, win 457, options [nop,nop,TS val 90857 ecr 87171], length 17
14:53:10.792272 IP 192.168.33.11.60739 > 192.168.33.10.12345: Flags [P.], seq 22:39, ack 6, win 457, options [nop,nop,TS val 90908 ecr 87171], length 17
14:53:11.201529 IP 192.168.33.11.60739 > 192.168.33.10.12345: Flags [P.], seq 22:39, ack 6, win 457, options [nop,nop,TS val 91010 ecr 87171], length 17
14:53:12.015841 IP 192.168.33.11.60739 > 192.168.33.10.12345: Flags [P.], seq 22:39, ack 6, win 457, options [nop,nop,TS val 91214 ecr 87171], length 17
14:53:13.653392 IP 192.168.33.11.60739 > 192.168.33.10.12345: Flags [P.], seq 22:39, ack 6, win 457, options [nop,nop,TS val 91623 ecr 87171], length 17
14:53:16.922019 IP 192.168.33.11.60739 > 192.168.33.10.12345: Flags [P.], seq 22:39, ack 6, win 457, options [nop,nop,TS val 92440 ecr 87171], length 17
14:53:23.464003 IP 192.168.33.11.60739 > 192.168.33.10.12345: Flags [P.], seq 22:39, ack 6, win 457, options [nop,nop,TS val 94076 ecr 87171], length 17
14:53:36.535794 IP 192.168.33.11.60739 > 192.168.33.10.12345: Flags [P.], seq 22:39, ack 6, win 457, options [nop,nop,TS val 97344 ecr 87171], length 17

Notice the timestamps: it tries again after 204ms, 203ms, 409ms, 814ms, 1637ms, 3268ms, 6541ms, 13071ms ... this is exponential backoff, doubling each time.

Linux eventually gives up after 15 retries. This works out at around 30 minutes. The number of retries is configurable as /proc/sys/net/ipv4/tcp_retries2. One can edit it with echo 5 > /proc/sys/net/ipv4/tcp_retries2.

When Linux gives up, the connection goes directly from ESTABLISHED to disappearing entirely. This transition seems to be omitted from the TCP state transition diagrams I’ve seen.

Tagged #programming, #unix, #networking.
👋 I'm Jim, a full-stack product engineer. Want to build an amazing product and a profitable business? Read more about me or Get in touch!

Similar posts

How to make a webserver with netcat (nc)
Use nc (netcat) to create a web server that returns custom HTTP responses. 2018-12-31
Creating a UDP connection with netcat
nc can create UDP connections. UDP connections are established when the first data packet is sent, and terminated when the server becomes unreachable. ICMP messages notify the client of an unreachable server. 2018-03-04
Running tcpdump on a TCP connection
tcpdump captures network traffic. Capture and analysis of a TCP connection, including the 3-way handshake and message exchange. 2018-02-23
What is DHCP?
DHCP dynamically assigns IP addresses to hosts, using a client-server protocol over UDP. It involves a sequence of DORA (Discover, Offer, Request, Acknowledge) messages to obtain and configure a network lease. 2018-02-06
How to run Redis Sentinel
Redis Sentinel provides high availability for Redis. We start a Redis master, then three Redis Sentinel instances. They discover each other, then we trigger a failover. 2019-01-08
What is HTTP keep-alive? What is HTTP request pipelining?
HTTP keep-alive allows reusing the same TCP connection for multiple requests, avoiding the overhead of opening a new connection. HTTP request pipelining sends multiple requests without waiting for their responses. 2018-03-27

Similar posts

How to make a webserver with netcat (nc)
Use nc (netcat) to create a web server that returns custom HTTP responses. 2018-12-31
Creating a UDP connection with netcat
nc can create UDP connections. UDP connections are established when the first data packet is sent, and terminated when the server becomes unreachable. ICMP messages notify the client of an unreachable server. 2018-03-04
Running tcpdump on a TCP connection
tcpdump captures network traffic. Capture and analysis of a TCP connection, including the 3-way handshake and message exchange. 2018-02-23
What is DHCP?
DHCP dynamically assigns IP addresses to hosts, using a client-server protocol over UDP. It involves a sequence of DORA (Discover, Offer, Request, Acknowledge) messages to obtain and configure a network lease. 2018-02-06
What is tcpdump?
tcpdump captures and displays network traffic. An example inspecting DNS requests and responses. 2018-02-01
osquery: UNIX as a SQL database
osquery allows querying a UNIX system using SQL. 2017-12-11

More by Jim

What does the dot do in JavaScript?
foo.bar, foo.bar(), or foo.bar = baz - what do they mean? A deep dive into prototypical inheritance and getters/setters. 2020-11-01
Smear phishing: a new Android vulnerability
Trick Android to display an SMS as coming from any contact. Convincing phishing vuln, but still unpatched. 2020-08-06
A probabilistic pub quiz for nerds
A “true or false” quiz where you respond with your confidence level, and the optimal strategy is to report your true belief. 2020-04-26
Time is running out to catch COVID-19
Simulation shows it’s rational to deliberately infect yourself with COVID-19 early on to get treatment, but after healthcare capacity is exceeded, it’s better to avoid infection. Includes interactive parameters and visualizations. 2020-03-14
The inception bar: a new phishing method
A new phishing technique that displays a fake URL bar in Chrome for mobile. A key innovation is the “scroll jail” that traps the user in a fake browser. 2019-04-27
The hacker hype cycle
I got started with simple web development, but because enamored with increasingly esoteric programming concepts, leading to a “trough of hipster technologies” before returning to more productive work. 2019-03-23
Project C-43: the lost origins of asymmetric crypto
Bob invents asymmetric cryptography by playing loud white noise to obscure Alice’s message, which he can cancel out but an eavesdropper cannot. This idea, published in 1944 by Walter Koenig Jr., is the forgotten origin of asymmetric crypto. 2019-02-16
How Hacker News stays interesting
Hacker News buried my post on conspiracy theories in my family due to overheated discussion, not censorship. Moderation keeps the site focused on interesting technical content. 2019-01-26
My parents are Flat-Earthers
For decades, my parents have been working up to Flat-Earther beliefs. From Egyptology to Jehovah’s Witnesses to theories that human built the Moon billions of years in the future. Surprisingly, it doesn’t affect their successful lives very much. For me, it’s a fun family pastime. 2019-01-20
The dots do matter: how to scam a Gmail user
Gmail’s “dots don’t matter” feature lets scammers create an account on, say, Netflix, with your email address but different dots. Results in convincing phishing emails. 2018-04-07
The sorry state of OpenSSL usability
OpenSSL’s inadequate documentation, confusing key formats, and deprecated interfaces make it difficult to use, despite its importance. 2017-12-02
I hate telephones
I hate telephones. Some rational reasons: lack of authentication, no spam filtering, forced synchronous communication. But also just a visceral fear. 2017-11-08
The Three Ts of Time, Thought and Typing: measuring cost on the web
Businesses often tout “free” services, but the real costs come in terms of time, thought, and typing required from users. Reducing these “Three Ts” is key to improving sign-up flows and increasing conversions. 2017-10-26
Granddad died today
Granddad died. The unspoken practice of death-by-dehydration in the NHS. The Liverpool Care Pathway. Assisted dying in the UK. The importance of planning in end-of-life care. 2017-05-19
How do I call a program in C, setting up standard pipes?
A C function to create a new process, set up its standard input/output/error pipes, and return a struct containing the process ID and pipe file descriptors. 2017-02-17
Your syntax highlighter is wrong
Syntax highlighters make value judgments about code. Most highlighters judge that comments are cruft, and try to hide them. Most diff viewers judge that code deletions are bad. 2014-05-11

This page copyright James Fisher 2018. Content is not associated with my employer. Found an error? Edit this page.

Jim Fisher
CV
Speaking
Blogroll
RSS
TigYog
Kickabout