1. What problem are you solving?

Incognito Nodes serve as simple plug-and-play devices so they often sit behind restricted home-networks. Without manual configurations, these nodes cannot talk directly to each other because of firewalls or protocols such as Network Address Translation (NAT). This proposal investigates a number of promising solutions to help alleviate this problem. For the uninitiated, we give a high-level overview of NAT in the paragraphs below.

Each device on the internet has an IP address. This address helps devices find out where others are in the network and connect to them. There’s only one problem: unique IP addresses are limited. This leads to the emergence of protocols like NAT. In simple terms, a router supporting NAT connects to multiple devices at once and gives each of them a unique local IP address. All of these devices can be dialed from other devices on the Internet by referring to the router’s global IP address.

To draw a parallel picture, imagine 2 people (devices) who need to send a letter to each other. On the envelope, they write the address of the receiver (e.g., 325 S. Santa Claus Lane North Pole, Alaska 99705) to tell the postman where to deliver it. When NAT is used, it’s like they’re asking the postman: “please deliver my letter to the house with the red door”. The house with the red door (local IP address) might be unique if we know which street it is on (global IP address). But in this scenario, there is no way to deliver the letter if only the local IP address is given. To make matters worse, in this scenario, the receiver doesn’t know what their address is, and therefore, they cannot give it to the postman or to the sender.

2. What is the solution?

There are multiple established solutions for NAT traversal, all of which are heavily used in industrial products (such as peer-to-peer file sharing and real-time communication). According to our initial research, here are the promising protocols:

STUN (hole-punching): reuse the same port for inbound and outbound connections

TURN: relay data through another (public IP) server

ICE: sequentially run the above protocols and use the one with the highest performance

These solutions aren’t deployed in any blockchains that we know of. Therefore, this proposal presents a plan to investigate the protocols in these ways:

Assess if these protocols are suitable for the Incognito blockchain. Will they completely solve the problem of NAT traversal?

Test these protocols by implementing small samples/demo applications

Assess the degree of change to the Incognito blockchain codebase if we want to implement them

3. Which solutions do people resort to because this doesn’t exist yet?

Currently, the NAT problem is bypassed by using highway. As they are devices with public global IP, they act as brokers that connect to all nodes and help exchange information between them. As noted in the article above, highway introduces its own problem:

There is a centralized aspect to this design. Highway is currently run by the Core Development Team. We are working on a plan to enable other teams/individuals to run and connect their highways to the Incognito network.

Incognito chain is fully decentralized, but the network topology isn’t, since all data must flow through highways. Therefore, to achieve operational decentralization, the problem of NAT must be solved so that nodes can connect directly to each other.

4. Who are you?

The initial research phase will be performed by engineers from the Incognito Scalability team (@lam). The results will be verified by other engineers and researchers in the same team (@0xkumi, @dungtran).

5. Why do you care?

We take Incognito’s decentralization plan very seriously. Solving this problem would be a big step toward a more decentralized blockchain for everyone. Also, if we pull this off, we will be the first blockchain to solve this problem.

6. What’s your plan? What’s your schedule?

This proposal is the first step toward a peer-to-peer (p2p) network topology for Incognito. There are 3 steps:

Research and design how to do NAT traversal (this proposal)

Research and design how to build the p2p network and the broadcast algorithm given that the NAT traversal protocols work (future work)

Implement NAT traversal and the broadcast algorithm (future work)

This proposal is scheduled for 4 weeks:

First 3 weeks: write demo and test on different NAT routers for all the above protocols. Deliverables will be demo programs.

Fourth week: document and design how to implement these protocols into current Incognito codebase. Deliverables will be a document and the design of the NAT traversal solution.

If Nodes can find each other does that mean there is a possibility for attacks as well? There was some concern about IP addresses being exposed to all network participants, it was then said only the Highway had this information. Is that going to be changed?

For the 1st question:
Yes, however, depends on which concern being mentioned the possibility is different.

If the concern is about the Incognito network reliability, then the possibility of an attack on Highway is very high if it is the only mean of network communication. So as mentioned in the proposal, to eliminate this bottleneck, we going to rely on other nodes on the network to do the job and put Highway to be the last option when direct p2p isn’t possible. It’s hard to take down a lot of small nodes than a few big nodes.

if the concern is about the user home network’s security, then it heavily depends on how the user and/or ISP (Internet Service Provider) setup their home network. For now, we are only looking at solutions that don’t require the user to change their router configuration (by default ISP should’ve already setup a secure home network against outsider for the user)

For the 2nd question:
Yes, your external IP address will be broadcast to participants on the network.

oh, about that, hate to say this but you can’t stay incognito with your ISP hence your government. The IP addresses of Highway is public so the government technically can track whether you connect to the Incognito network or not.

We currently thinking of some solutions to stay incognito on the network but that is for another proposal in the future.

If you switch to fully P2P connection, the number of messages that each validator must receive during the BFT consensus will be really large (about O(N^2) message with N is the size of the committee), this makes it difficult for the network to achieve scalability. Do you have a solution to this problem?

We not actually switching to full p2p, Highway still have an important role. Chain committee will have priority access to Highway to communicate with each others. This solution is aiming at node that isn’t in chain committee and as a backup plan for scenarios where Highway isn’t working or got blocked.

BFT-Vote message size is relatively small so that shouldn’t be a problem. BFT-Propose message will likely to be slower when using this solution than Highway.

I think this is an important feature, and that might worth it. So far, In our infrastructure, HW is the weakest component. Anything makes HW harmful, the message can not communicate from node to node and the chain should be stopped. I see that someone user also consider about it like here. And NAT traversal solution is the way to find out better p2p solution for our network.

So far, In our infrastructure, HW is the weakest component. Anything makes HW harmful, the message can not communicate from node to node and the chain should be stopped

I think this is quite difficult to evaluate. Although the Highway may be weak, it can be overcome by expanding the Highway network.

lam:

We not actually switching to full p2p, Highway still have an important role. Chain committee will have priority access to Highway to communicate with each others. This solution is aiming at node that isn’t in chain committee and as a backup plan for scenarios where Highway isn’t working or got blocked.

Can you sketch the network architecture when applying Highway and P2P in parallel so that we can easily visualize what you want to do?

I think this is quite difficult to evaluate. Although the Highway may be weak, it can be overcome by expanding the Highway network.

Expanding Highway as Centralized system is a economic issue for core team or when core-team is not survive, how to maintain this highway is hard to answer. And how is enough for expanding Highway network.

@lam great post and very comprehensive explanation. HW is now centralize. A plan for decentralized network and a good topology is desirable. This proposal is very much worth doing.
My question is the same as @hyng as well

How HW and P2P work together? It’s would be complicated, and need carefully integration.

BFT network cost is O(N^2), it’s a matter if Incognito consider increase committee size from 32 to 300. So plz take this into consideration in research and implementation phase, it could help take on network complexity problem if needed

Could you provide some information about how Ethereum/Bitcoin (big old blockchain) and Harmony/Near (new sharding blockchain) nodes communicate with each others. Do they encounter this problem as well? Especially Harmony or Near?

“How HW and P2P work together?” Yeah, as you said it would be complicated. As of the time when writing this reply we still exploring possible approaches to this matter, we will post an update as soon as we have something concrete.

“BFT network cost is O(N^2),…” thank you for raising this concern, the increase in committee size is in the future plan of Incognito so we will take this into consideration.

Ethereum & Bitcoin are depending on big mining pools which have setup large infrastructure of public IP nodes so they didn’t really have to worry much about this kind of problem. Harmony documentation is only about hosting validator node on the cloud, no user-own device is mentioned. As the case for Near, they use something like a TURN/tunneling method, the performance of method depends on how good their routing algorithm is if all devices are behind NAT.

So yes, public blockchain networks do encounter this problem as well but depending on which kind of community they want to build this issue will have big or small impact.

Ethereum & Bitcoin are depending on big mining pools which have setup large infrastructure of public IP nodes so they didn’t really have to worry much about this kind of problem. Harmony documentation is only about hosting validator node on the cloud, no user-own device is mentioned. As the case for Near, they use something like a TURN/tunneling method, the performance of method depends on how good their routing algorithm is if all devices are behind NAT.

Thanks @lam for your information. As you mentioned, big blockchain doesn’t encounter this problem, they use public IP. Private IP or IP behind NAT it a hard problem to tackle, so maybe the bypass it. If we continue to take on this problem it would be very hard. What if we accomplish NAT traversal solution then we find out in reality it doesn’t work for private IP node (slow message transmission, small bandwidth)?
Does your solution work for public IP node? If it does, then we can use p2p for public ip then highway for private ip.

This is a proposal for NAT Traversal and public IP nodes don’t use NAT so they can just connect directly to each other easily. We have a lot of users using Incognito node in the home, this solution mainly for these devices, if we depend on Highway for these devices which will be many, we have to build and maintain a lot of Highway servers as they will need to grow when more and more devices join the network.

I believe that reserve Highway for nodes that are in committee and use p2p for other kinds of nodes is a more cost-effective and simpler solution. Plus it will be more decentralized.

Here is a summary of all what we have discussed above and what we have researched this week:

We are not switching to fully using P2P connection, both Highway and NAT traversal methods will be used. Highway will be reserved for committee nodes, other nodes will use NAT traversal for communication.

As researched, other blockchain networks are encountering the same problem when it comes to p2p, but the impact on their network is different.

There are several methods used by other software to overcome this problem, some require the user to manually set up and some don’t.

We are still exploring solutions that not require the user to manually change their router setting to allow inbound connection to a device behind NAT.

We currently try to simulate a network of devices behind different types of NATs.

That’s a nice idea but Tor/I2P is slow as it has to traverse many nodes and because of the current way of how node in Incognito network must communicate, using Tor/I2P will slow the network way down.
In the meantime, we have our focus on reducing dependency on Highway. We will explore solutions that hide user device’s IP address in a future proposal.