Blog Stats

Pages

Tag: algorithm

In my last post, we had a look at NEO-CLI. Today, we will have a look at the basic concept of consensus NEO implements inside. There are a lot of ways to implement consensus in any blockchain platform you see around. All of them tends to solve the historical problem named “The Byzantine Generals’ Problem”. If you want to read the original paper on the problem, here it is.

The generals and their problems

The simplest way one could possibly describe this problem would be to describe it as the problem of determining the true outcome of a vote. One might wonder how that can be a problem since counting the votes and tallying them up should easily tell the result. Let’s imagine a scenario where we have 5 generals. They want to attack a city in the middle but all of them are in different geographic location. In a simplistic picture, the setup would look something like:

They need to attack or retreat in sync or according to a predetermined plan. To act in unison, they proposed a way to vote for the next move. They decided to vote daily and the majority vote wins. Since they are not in the same geographic place, they use couriers to convey their vote amongst each other.

One doesn’t need to think for long to land on the conclusion that this whole plan has some inherent flaws. First, the couriers are humans and not that reliable, they can get captured, bribed to work for the enemy. They can get killed, lose the message or deliver the wrong message. Secondly, the generals themselves can be bribed in the same way to betray each other. And last but not the least, any general can make an inappropriate decision due to stupidity and failure to assess the situation or both.

Distributed computing by default inherits this problem since it works with a similar setup. Actors in the system can fall into the same trap as these generals. They can malfunction or act in an untrustworthy way to threaten or destabilize the system.

Delegated Byzantine Fault Tolerance

As it was mentioned before that there are numerous ways to solve this problem. Bitcoin paved a way of solving this problem through its Proof Of Work mechanism. Hyperledger Fabric improved upon the same principles through Practical Byzantine Fault Tolerance mechanism. NEO, on the other hand, proposes Delegated Byzantine Fault Tolerance.

Let’s assume we are living in Zion and there is a leader, preferably NEO. In this case, NEO is a delegate of Zion. In this world, he just can’t assume this role by his heroics, he has to gain votes from the citizen of Zion. All citizen votes for delegates like NEO and there are multiple delegates. The delegates are responsible for making the laws of Zion.

In any moment of time, if the citizen of Zion is not satisfied with a delegates proposal of laws, they can vote for a different delegate next time. Delegates are constantly notified of the demands of the citizen. They document it on a ledger, and yes you thought it right, this ledger is the analogy of our blockchain. These demands are added one after another and laws are passed focused towards keeping the citizens of Zion happy.

But a single delegate can’t pass a law anytime he wishes to. The laws are passed in a periodic manner. When it is time to pass a law, a random delegate is chosen, let’s call him the speaker. The speaker proposes the law based on the demands of the citizens. The job of the speaker is to show how the proposed law increases the happiness index of Zion. And then he shares the law he is proposing to the other delegates. The delegates then have the job to decide whether the speaker’s calculations were correct and the laws are aligned to the common goal. They also verify the newly improved happiness index calculation. If 66% of the delegates believe that the calculation is correct and it is an improvement, the law gets finalized and passed.

What if the new law has failed to get 66% of delegates approval? Then the whole process starts over and a new speaker is selected.

How this is translated in NEO blockchain

This whole concept is reflected in NEO blockchain. Anyone who owns NEO is a stakeholder, thus a citizen. They want to create transactions, transfer, exchange assets. They do not participate in the bookkeeping of the blockchain. To do that one node needs to be a delegate as in NEO blockchain world, a bookkeeping node. There are certain criteria to be fulfilled to be a bookkeeping node in NEO blockchain. This includes dedicated internet connections and a certain amount of GAS. The demands are analogous to transactions. When they are put together by a bookkeeping node, a new block is created, which is analogous to a law. The happiness number is the hash of the aforementioned block.

Common Pitfalls: The dishonest speaker

What if the speaker is dishonest? What if he sends wrong versions of the law to some of the delegates?

In this scene, the evil speaker sends different versions of the law to different delegates. So, when the delegates communicate with each other they send different versions of the law. In this case, the leftmost delegate will get the accurate version of the law and will be able to calculate the happiness index properly. The rest of the delegates will fail to do so, failing the consensus.

Common Pitfalls: The dishonest delegate

The dishonest delegate scenario should be self-explanatory now.

In this scenario, we have an honest speaker who sends the same version of the law to every delegate but the rightmost delegate is dishonest. He sends a different version B to everyone else. In this case, 2 of the delegates will be able to calculate the happiness index properly. But the will only be able to verify it from the speaker’s version. It will reach 66% consensus and the law will be passed. But the result of this session will deem the rightmost delegate to be faulty. This data can be used for the next time of citizen vote where they chose their delegates.

By the definition here the minimum threshold for faulty delegates in the system is f=(n-1)/3 where n is the number of active delegate nodes.

In my previous post, we started with a small NEO private net. Today, we will take a quick look into NEO-CLI and what it offers. Although it is named NEO-CLI, in practicality, this is a full blown NEO blockchain node instead of just a CLI tool to communicate with it. NEO offers two node types – GUI and CLI. I think the suffix comes from that and I wanted to explicitly mention it since it is a tad confusing.

At first, we will try to connect to our newly created private net. To do that, we will start with installing a separate installation of NEO-CLI. Installing NEO-CLI is pretty straight forward. You will need .Net core installed in your machine. If you don’t follow the instruction here.

Installation

I’m currently using an Ubuntu 16.04 as a reference OS. After installing .Net core framework you will need to install the NEO-CLI package. And since Im on a debian it was quite easy to do so the following way:

sudo apt-get install libleveldb-dev sqlite3 libsqlite3-dev

Configuration

We are testing and my local machine doesn’t have a frame of reference of the test private net we just created over Azure. To give this node a frame of reference we need to configure its SeedList to point to our own private net. What is a seed list? Simply put, it is nothing more than a list of URLs as described in the official NEO documentation. This is the first set of nodes NEO-CLI will try to connect to when it boots up.

To configure the aforementioned SeedList, we will modify the protocol.json file, under the neo-cli directory.

We need to update the SeedList section of the configuration the following way:

“SeedList”: [
"IP_or_FQDN_of_Azure_Private_Net_Host:20333”
],

If you opt to use the public test net, rename the protocol.testnet.json to protocol.json and you should be good to go.

Booting up the node

Now, it is time to start the node, we are going to invoke:

dotnet neo-cli.dll --log --nopeers

The log option will log the smart contract information and nopeers makes the node only connect to the seed nodes from the configuration file. this is something we want since this is a private network.

Creating a new wallet

Let’s create a new wallet then.

neo> create wallet mywallet.db3

NEO-CLI will ask for password twice for the wallet, pick your desired password. And copy the address and pubkey to keep it a safe place. If you forget the public key you can use list key command to see it.

More on protocol.json

Before we end this one, we will have one last look at the protocol.json configuration file for our node.

The Magic field contains a uint value that denotes the source network of the message.

The StandbyValidators field are the validating nodes in the private node. It is the list of public keys of aforementioned validating nodes. We created 4 wallet here in this specific example and thus we have 4 entries here. 4 is the minimum number of nodes here to be listed for reaching a consensus.

SeedList is configured to localhost in this example configuration since NEO-CLI is booting up against the localhost node.

SystemFee section is the section that defines the system fee. As the configuration states, the registration fee for assets is 100000 GAS depicted by the RegisterTransaction field. EnrollmentTransaction field defines the registration fee for book-keepers. IssueTransaction is the fee for distributing assets. Finally the PublishTransaction is the fee for smart contracts.

That sums it up for this time. Next, we are going to have a look at how consensus works in NEO. And finally we will write a smart contract on NEO in C#. 🙂

Usually I talk about pretty current stuffs, let’s talk future now. And the future is needed to be talked now. Because after some days in this young app industry simple rss readers won’t be considered a quality app anymore. If you look around, Windows Phone in Bangladesh has walked a pretty silent but significant path. And I’m quiet happy that I walked a little too.

Now, let’s start talking how this was made. When I started doing this, the first thing that came into my mind is the actual scenario of Dhaka, if you really expect a local bus to come in the very right time exactly every day, I guess you have optimism as your profession. So, along with finding the best route and other basic operations, I had to believe someday I would face the challenge which will “actually” solve the problem.

Traffic jams are usually one of the common phenomena here, very common as in like brushing your teeth everyday. If it’s not there, you get surprised. But it’s not the only thing that can keep you getting on the right bus. There’s bad road situations, there’s bus shortage, possibly anything that you can’t predict or manage.

I thought big big plans, like putting devices on buses and tracking down each one of them. But think once, just once. When nobody is even curious about putting proper fitness checks on a bus, maintaining a device would be a overkill for them, and of course will lead my dream service to perish.

What I had it mind was a pretty simple one. If you ever heard of Ant Colony Optimization, I believe in case of Dhaka, it would come to a greater use . Because the shortest path is not always the “right path”.

I wanted to follow the traditional Ant Colony Optimization trait here. Let’s put a small city like Dhaka with us like below:

Pretty simple for Dhaka, eh? 😀 Let’s keep it simple for understanding. You will see Node D is kind of the hub of the city, much like our motijheel, mohakhali or other big bus junctions, right? Now, in this city there can be a number of buses, right? Essentially a Bus Route is nothing but a number of these nodes ordered by bus stops.

Let’s make some bus routes, shall we?
1. A-C-E

2. C-D-E-B-A

3. D-E-C-A

For simplicity lets assume all these bus goes and comes back in the same order. Now, I know what you guys are thinking, if I put “distance metrics” on these paths, it’s easy to find a suitable bus route. Like, if you want to take A trip from A to D you can chose bus number 2 or 3. Question is which one should you pick? What if there is more traffic jam in B-E junction and less in C-E? These data changes over time, even in fraction of ten minutes. You can’t possibly decide every time.

Here ant colony comes so handy. The very basics of Ant Colony Optimization is based on how ants find best paths. In the natural world, ants wander randomly and after finding food return to their colony while they continuously lay down pheromone trails. If other ants find such a path, they are likely not to keep traveling at random, but to instead follow the trail, returning and reinforcing it if they eventually find food. So, the fact is who knows the best path right now, the bus that just has gone to that path a few minutes ago. Just like the ant. The only deficit we have here is pheromone trail left by ants. Now, remember what we have in hand. We cant put too much on a local bus as they wont take care of it properly.

But what about known bus counters? If we manage to find only time-stamps of a bus arriving at a bus-stoppage somehow, we don’t need to know which one came, just spotting one on a counter, much like spotting an ant on a node. And based on the frequency of that timestamp, you can actually verify the frequency of buses coming down there. If the adjacent roads are not doing well with traffic jams, you are bound to have a lesser frequency and if it’s smooth sailing you will have a solid frequency. Even if that sounds awkward we can even track ticket sales, although that would be far off, still a usable heuristic metric to understand whether people are actually getting up on a bus or not.

Now, based on that. We want to go to D from A.

We have two bus routes, 2 and 3. Now which to pick? All we have to do is check the pheromone/our very own heuristic cost metric trails and the route that currently has a more favorable reinforcement/feedback. No matter what comes in the road, traffic jams or stuff, within minutes, these would be updated. And as the buses are continuously traveling, the metric will always be updated. And further more, you can even predict a possible next bus time from this too. Or predict a best possible route to go from one place to another on a car. I know, for Dhaka, it might be a bit long shot, still…worth a try.

So, if you believe every bus is a ant running on a direction (although these are defined rather than random). Actually that makes us one step closer to goal. Because logically ant colony optimization finds a best path. As our possible paths are already defined, we don’t have to go through the first state.

There should be usually two steps of ant colony optimization:
1. Edge selection
2. Pheromone Update

As our paths are already defined, edge selection is basically finding eligible bus routes to go from one place to another, it could be a multiple bus journey but I will write about that later.

procedure BMD_Heuristic
while(not_termination)
TraverseRoutes()
RouteActions()
UpdateMetrics()
end while
end procedure

Let me know what you guys think of it and how can you help NerdCats take it to the next level. See ya!