What is behind the Steem bandwidth issues...

The last bandwidth issues inspired me to start my own debugging. I'm an OPS guy and when I'm facing issue I need to understand it and find the root cause.

Steem Blockchain provides a bunch of raw data, we can easily get details for any block we want, but we are people, we can't read and process millions lines of text on the fly... but... we're better at reading charts ;-)

I've built a system which gather data from Steem blockchain and generate dynamic charts for transactions and bandwidth details. The system is build on Open Source software such as Grafana, Graphite and few Python scripts I wrote for parsing and collecting data.

current_reserve_ratio it's a kind of anti spam parameter, whose value decrease when the average size of the new blocks is greater than 25% of the maximum_block_size (65536 * 0.25 = 16384).

Every transaction generates data

more transactions = more data,

more data = bigger blocks,

bigger blocks = lower current_reserve_ratio,

lower current_reserve_ratio = lower bandwidth,

lower bandwidth = MOAR! issues...

Let's take a look on the charts

This is a snapshot of data I made today, the charts show how many transactions are processed, what is the current_reserve_ratio, average block size, and how the Steem blockchain automatically reduce the bandwidth when average_block_size hit the limit.

We can see the most popular transactions are,

custom_json (blue) - when we follow/unfollow somebody

votes (red) - when we upvote post or comment

comment (orange) - when we add comment

The average_block_size hit the limit (red horizontal line set on 25% of max_block_size = 16kB) after which the current_reserve_ratio value is decreased.

And finally the bandwidth is reduced as well. The lower value of current_reserve_ration, the lower value of available blockchain badwidth.

Spikes

From time to time we see spikes caused by operations like votes, custom_json, delegate.

custom_json

Votes

Dozen of minions upvoting the same post at the same time... probably voting bots.

My conclusion

In the next few weeks/months something needs to be improved to make enough capacity for all new Steemians, it could be the algorithm itself or the max_block_size parameter set by Witnesses. We're hitting the limit which is set to just 25% of the max block size... :)

In the next few weeks/months something needs to be improved to make enough capacity for all new Steemians, it could be the algorithm itself or the max_block_size parameter set by Witnesses.

Longer term if the usage continues to grow there will certainly need to be an increase to the block size. The other thing that is needed is more visibility of bandwidth usage and limits so people can be more aware of them and not get cut off so frequently or for such long periods of time. When the reserve ratio is reduced, the accounts which are cut off first are those which are using the highest percentage of their maximum allowance (which can be increased by powering up for SP). Each user has to make a bit of a tradeoff between higher usage during low-usage periods, being cut off during high-usage periods, and powering up more SP. Unfortunately this tradeoff is difficult to make now in practice because bandwidth usage is not even visible at all in the UI.

This probably should be an advanced option though. Surely, if regular users will have to take even more parameters into account in the future, this will not help us attract normal, nongeek, population. It would greatly put me off as well.

So this is how I see it, @steemit holds the largest stake which means he has the largest bandwidth and since there are high volumes of transactions inside issues like these occur. There is also a chain of effect of this issue, like anyone who has been delegated by @steemit, and then lost the privilege to do things in steemit i. e upvoting, commenting, posting etc. That is why also @steemit is undelegating his sp to newbies before but what about those who havent got enough sp?

So this is how I see it, @steemit holds the largest stake which means he has the largest bandwidth

Sort of. Part of the purpose of the reserve ratio is to account for unused and underused stake like the steemit account (and in practice most other large accounts). Since those accounts are not using their full share of bandwidth, the reserve ratio grows (or stays) higher, and that unused portion becomes available to every else via a (temporarily) increased allowance.

That is why also @steemit is undelegating his sp to newbies before but what about those who havent got enough sp?

As I understand it, the account is supposed to be undelegating from accounts which have been misbehaving (in this case many accounts controlled by the same person/group and mostly spam self voting). To the extent those accounts are cut off from the delegation and more limited in bandwidth, more bandwidth should then be left for everyone else.

Wait a minute, its not just me?! I thought I was just doing something wrong using up all my bandwidth in the mornings and then having tons of bandwidth in the afternoon, with morning bandwidth in the kilobytes and afternoon in the megabytes, I thought it was because I was new and needed more steem power.

Thank you!
Finally someone who could find one cause for the bandwidth drop.
If it is true, what you are writing here, it seems, that steem is currently vunerable for some kind of DoS.
Just one user did bring down the bandwidth for everybody?
That will be quite interesting to see, how the blockchain reacts to more user and more transactions.
Not even to mention the upcoming SMTs

That was an excellent analysis. Theoretically, this issue should drive up the price for Steem, since you need more to Steem Power to still post something.

But obviously, this is an artificial bottleneck that shouldn't be there. So it actually is a matter that needs to be dealt with, or it might in fact hurt the price of Steem due to the unsystematic nature of the bandwidth shortage.

Witnesses. Proof of stake. But big picture, you can see it as the same shit. Instead of mining pools, you get these witnesses that are voted by the rest of steemit users. They pay the cost of running the server; they produce new blocks. Once new blocks are created, Steem magically appears. Some goes to witnesses, the rest goes to authors, curators, and some goes to people who own a lot of Steem.

The current block size is 65536 (64kB) and algorithm reduces bandwidth when new blocks are filled in 25% (16384). I think the long term solution is algorithm update, the blocks seem to be big enough for now.

Congratulations @jamzed, this post is the most rewarded post (based on pending payouts) in the last 12 hours written by a User account holder (accounts that hold between 0.1 and 1.0 Mega Vests). The total number of posts by User account holders during this period was 2912 and the total pending payments to posts in this category was $8187.38. To see the full list of highest paid posts across all accounts categories, click here.

If you do not wish to receive these messages in future, please reply stop to this comment.

As I guessed the bandwidth problems that Steemit was having was because "peek" hour's, but I can understand that is something that is going beyond that.

I think this will be a bigger issue in the future, and I don't think it would be so easily fix, decreasing available bandwidth per user, would be a better solution for the long run.

Yes yes I know that reducing bandwidth per user to increase bandwidth per user sounds wrong, but hear me out.

We are having problems because there are accounts using to much bandwidth while us done use near the 90% of the available bandwidth we had, and we can't prevent someone voting for the same post at the same time so the only solutions are to increase the block size and decrease the available bandwidth per user.

Also, Thank you for your post, it is has very detailed information that I wouldn't had known if not for you posting it, it also made me realize that yes, I also took noticed the same problems with the bandwidth but I did not stop to question why beyond it is "peek" hours, and even worse, to look for a solution for the problem.

Hallo @jamzed It would be quite nice to be able to select multiple things to track at once, which may be possible but was not immediately obvious to me how to do.Very keen, though. Nicely done.thanks for sharing this post. Upvote and resteem done now

Its good that people who have technical process are participating in the improvement of Steemit. I have heard that there are currently 6 lakh steemian and growing. I have heard that a lot of new people are joining steemit. This is why i think it is important that people with technical information help steem out.

I am happy that someone could find one cause for the bandwidth drop.
If it was just a user that did bring down the bandwidth for everybody?
That will be quite interesting to see, how the blockchain reacts to more user and more transactions.
Good job.
You sure do have my vote.