How do peer-to-peer programs discover each other?

In peer-to-peer systems like BitTorrent and Bitcoin, you start your local program on a networked computer, and you become connected to other instances of the same program running on lots of other computers. The local process starts listening on a local port, and finds IP:port pairs for those other processes. How does your process discover those “peer” addresses?

If the peer-to-peer system is purely decentralized, there is no better way to find a peer than to search. Decide on a specific port for your application. Have the new process send a “ping” packet to every address on the internet, until one replies with a “pong”.

Once you find one peer, you can stop port-scanning: peers can share address information with each other. The peer you find can reply with all the addresses of the peers it knows. You can then connect to these peers, and repeat the process until you have “enough” peers.

This approach has one big problem: to find just one peer, you may have to send millions of “hello” packets. This is not good for your bandwidth, or for start-up latency, or for the network generally.

The only way to fix the “initial search” problem is to centralize. BitTorrent does this with “trackers”. A .torrent file contains the URL of a tracker server for that file:

$ wget http://releases.ubuntu.com/17.04/ubuntu-17.04-desktop-amd64.iso.torrent
$ npm install --save bencode
$ node
> let bencode = require("bencode")
> bencode.decode(fs.readFileSync("ubuntu-17.04-desktop-amd64.iso.torrent")).announce.toString('utf8')
'http://torrent.ubuntu.com:6969/announce'

A GET request to that URL, with some info about the file you wish to download, gives you a list of peers you can connect to.

Bitcoin uses other centralized methods for peer discovery. An early mechanism was to connect to known IRC channels. Later, Bitcoin began using “DNS seeds”. The Bitcoin client is hardcoded with a list of domain names. The owners of these domain names maintain a set of A records which point to long-running Bitcoin peers. For example, seed.bitcoinstats.com resolves to a whole bunch of IPs:

$ dig seed.bitcoinstats.com

; <<>> DiG 9.8.3-P1 <<>> seed.bitcoinstats.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6518
;; flags: qr rd ra; QUERY: 1, ANSWER: 25, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;seed.bitcoinstats.com.		IN	A

;; ANSWER SECTION:
seed.bitcoinstats.com.	60	IN	A	104.247.230.28
seed.bitcoinstats.com.	60	IN	A	220.245.196.37
seed.bitcoinstats.com.	60	IN	A	138.197.109.21
seed.bitcoinstats.com.	60	IN	A	47.94.45.184
seed.bitcoinstats.com.	60	IN	A	14.52.209.177
seed.bitcoinstats.com.	60	IN	A	122.13.2.80
...
seed.bitcoinstats.com.	60	IN	A	138.68.64.102
seed.bitcoinstats.com.	60	IN	A	88.150.192.17
seed.bitcoinstats.com.	60	IN	A	76.187.118.123
seed.bitcoinstats.com.	60	IN	A	82.197.194.198
seed.bitcoinstats.com.	60	IN	A	78.40.244.243
seed.bitcoinstats.com.	60	IN	A	163.172.161.49

;; Query time: 269 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Sun Aug 13 23:40:52 2017
;; MSG SIZE  rcvd: 439
Tagged .
👋 I'm Jim, a full-stack product engineer. Want to build an amazing product and a profitable business? Read more about me or Get in touch!

Similar posts

Shape typing in Python
2024-04-12
Automatic differentiation with dual numbers
Differentiation asks: how does tweaking the inputs change the output? To accurately differentiate an arbitrary function, perhaps the simplest way is using “dual numbers”. 2024-04-02
What is numerical differentiation?
Differentiation asks: how does tweaking the inputs change the output? Numerical differentiation is a simple, general way to do this. But it’s not very accurate, and it’s inefficient when there are many inputs. 2024-04-01
Tell the LLM the business context
Tell LLM why its task is needed and how the output will be used. You’ll get better results, but programmers often don’t do this because it’s anti-modular. 2024-03-30
Auto-summarizing my blog posts
Summarizing my blog posts using Claude Haiku. The benefits of providing business context. The benefits of iterating. 2024-03-26
It might not need a label
Often, content is self-labelling. Removing its label can make the content more scannable. Programmers often over-label. 2024-03-20

Similar posts

Shape typing in Python
2024-04-12
Automatic differentiation with dual numbers
Differentiation asks: how does tweaking the inputs change the output? To accurately differentiate an arbitrary function, perhaps the simplest way is using “dual numbers”. 2024-04-02
What is numerical differentiation?
Differentiation asks: how does tweaking the inputs change the output? Numerical differentiation is a simple, general way to do this. But it’s not very accurate, and it’s inefficient when there are many inputs. 2024-04-01
Tell the LLM the business context
Tell LLM why its task is needed and how the output will be used. You’ll get better results, but programmers often don’t do this because it’s anti-modular. 2024-03-30
Auto-summarizing my blog posts
Summarizing my blog posts using Claude Haiku. The benefits of providing business context. The benefits of iterating. 2024-03-26
It might not need a label
Often, content is self-labelling. Removing its label can make the content more scannable. Programmers often over-label. 2024-03-20

More by Jim

What does the dot do in JavaScript?
foo.bar, foo.bar(), or foo.bar = baz - what do they mean? A deep dive into prototypical inheritance and getters/setters. 2020-11-01
Smear phishing: a new Android vulnerability
Trick Android to display an SMS as coming from any contact. Convincing phishing vuln, but still unpatched. 2020-08-06
A probabilistic pub quiz for nerds
A “true or false” quiz where you respond with your confidence level, and the optimal strategy is to report your true belief. 2020-04-26
Time is running out to catch COVID-19
Simulation shows it’s rational to deliberately infect yourself with COVID-19 early on to get treatment, but after healthcare capacity is exceeded, it’s better to avoid infection. Includes interactive parameters and visualizations. 2020-03-14
The inception bar: a new phishing method
A new phishing technique that displays a fake URL bar in Chrome for mobile. A key innovation is the “scroll jail” that traps the user in a fake browser. 2019-04-27
The hacker hype cycle
I got started with simple web development, but because enamored with increasingly esoteric programming concepts, leading to a “trough of hipster technologies” before returning to more productive work. 2019-03-23
Project C-43: the lost origins of asymmetric crypto
Bob invents asymmetric cryptography by playing loud white noise to obscure Alice’s message, which he can cancel out but an eavesdropper cannot. This idea, published in 1944 by Walter Koenig Jr., is the forgotten origin of asymmetric crypto. 2019-02-16
How Hacker News stays interesting
Hacker News buried my post on conspiracy theories in my family due to overheated discussion, not censorship. Moderation keeps the site focused on interesting technical content. 2019-01-26
My parents are Flat-Earthers
For decades, my parents have been working up to Flat-Earther beliefs. From Egyptology to Jehovah’s Witnesses to theories that human built the Moon billions of years in the future. Surprisingly, it doesn’t affect their successful lives very much. For me, it’s a fun family pastime. 2019-01-20
The dots do matter: how to scam a Gmail user
Gmail’s “dots don’t matter” feature lets scammers create an account on, say, Netflix, with your email address but different dots. Results in convincing phishing emails. 2018-04-07
The sorry state of OpenSSL usability
OpenSSL’s inadequate documentation, confusing key formats, and deprecated interfaces make it difficult to use, despite its importance. 2017-12-02
I hate telephones
I hate telephones. Some rational reasons: lack of authentication, no spam filtering, forced synchronous communication. But also just a visceral fear. 2017-11-08
The Three Ts of Time, Thought and Typing: measuring cost on the web
Businesses often tout “free” services, but the real costs come in terms of time, thought, and typing required from users. Reducing these “Three Ts” is key to improving sign-up flows and increasing conversions. 2017-10-26
Granddad died today
Granddad died. The unspoken practice of death-by-dehydration in the NHS. The Liverpool Care Pathway. Assisted dying in the UK. The importance of planning in end-of-life care. 2017-05-19
How do I call a program in C, setting up standard pipes?
A C function to create a new process, set up its standard input/output/error pipes, and return a struct containing the process ID and pipe file descriptors. 2017-02-17
Your syntax highlighter is wrong
Syntax highlighters make value judgments about code. Most highlighters judge that comments are cruft, and try to hide them. Most diff viewers judge that code deletions are bad. 2014-05-11

This page copyright James Fisher 2017. Content is not associated with my employer. Found an error? Edit this page.

Jim Fisher
CV
Speaking
Blogroll
RSS
TigYog
Kickabout