For miners, including too many transactions into a mined block increases the risk of collision with a block generated by another miner at approximately the same fime, resulting in one of the blocks being orphaned and that miner losing both the block reward and the fee. In the even of a collision between 2 blocks, it is more likely for the larger block to get orphaned.

There are a bunch of posts over at the mining threads complaining about SatoshiDice being responsible for the recent increase in the number of orphaned blocks. Even with high Tx fees, there is a limit to how many transactions a miner will risk including into a block due to this.

That seems like a big problem... could we change the protocol to favor the larger block?

It seems like a nonexistant problem, unless I've totally misunderstood how mining works. If so, can anyone explain to me why including more transactions in a block increases the chance that the another block is found at the same time? The time spend hashing transactions and transmitting data is negligable compared to the time spent calculating the block header hash. And why would the block with the most transactions be more likely to be orphaned? Doesn't that depend entirely on which fork just happens to be the first to be built on, which is essentially random and completely independant of the transactions in the block?

including more txns in a block does not increase the chance of another block being found at the same time. It is an unusual occurrence normally.BUT at the moment IF 2 blocks are found close together, the smaller block (the one with less txns) can be propagated (uploaded to the network) and verified quicker (by other nodes) than the "big " block, this means the small block can propagate across the network quicker and thus has a higher likelihood of being the 1st block built on leaving the larger block to be orphaned.

essentially random , maybe, but if more nodes hear about one block and start building on it (through faster propagation), there is more likelihood it will the winner in the orphan "race"

So... I'm a little confused on why miners are having a problem with the SatoshiDice "polluting" the chain. The solution isn't that difficult or complex. Yes, SatoshiDice is paying the .0005 fee, which is good and as designed. However, if, as a miner, you feel that is inadequate because of the stress SatoshiDice is putting on the network, then refuse transactions from SatoshiDice at the .0005 fee and only include them if they have a .01 fee (or whatever amount you feel is adequate for the stress they are placing on your system(s)). If the rest of the network is in agreeance with your arbitrary fee requirement for that type of transaction, and they all start limiting SatoshiDice transactions to .01 or higher, then SatoshiDice won't be processed until they increase their fee... which they will do or basically go out of business.

In the short term, it may increase the queue quite a bit, which is bad, however I think market pressures will rapidly correct the problem on the SatoshiDice side. What am I missing here?

If you're searching these lines for a point, you've probably missed it. There was never anything there in the first place.

Actually the older bitcoin gets and the block reward goes down then TX fee will need to increase as incentives to miners to keep operating.

Will the TX fee need to increase, or will the volume of transactions make up for it? The cost of including a transaction is what... hashing it once for inclusion in the Merkle tree? Given enough transactions, it doesn't seem like that should cost more than 1/4 penny (~what it is now).

For miners, including too many transactions into a mined block increases the risk of collision with a block generated by another miner at approximately the same fime, resulting in one of the blocks being orphaned and that miner losing both the block reward and the fee. In the even of a collision between 2 blocks, it is more likely for the larger block to get orphaned.

There are a bunch of posts over at the mining threads complaining about SatoshiDice being responsible for the recent increase in the number of orphaned blocks. Even with high Tx fees, there is a limit to how many transactions a miner will risk including into a block due to this.

I don't under stand why there Is a larger chance with more TXs, can you elaborate?

I have seen any miners complaining. The whinning and doom/gloom is from people to cheap to pay a fractional penny in fees and thus having lower priority tx. Nothing more.

Actually, no. There is a huge problem with block validation times vastly increasing and a big developer discussion on scalability occurring now. Bitcoin can be broken. I personally think that the 50 BTC block reward is too high and acting as an artificial subsidy on tx fees.

Actually the older bitcoin gets and the block reward goes down then TX fee will need to increase as incentives to miners to keep operating.

Will the TX fee need to increase, or will the volume of transactions make up for it? The cost of including a transaction is what... hashing it once for inclusion in the Merkle tree? Given enough transactions, it doesn't seem like that should cost more than 1/4 penny (~what it is now).

For miners, including too many transactions into a mined block increases the risk of collision with a block generated by another miner at approximately the same fime, resulting in one of the blocks being orphaned and that miner losing both the block reward and the fee. In the even of a collision between 2 blocks, it is more likely for the larger block to get orphaned.

There are a bunch of posts over at the mining threads complaining about SatoshiDice being responsible for the recent increase in the number of orphaned blocks. Even with high Tx fees, there is a limit to how many transactions a miner will risk including into a block due to this.

I don't under stand why there Is a larger chance with more TXs, can you elaborate?

Each TX is included in the block so...The more TXs, the larger the block.The larger the block, the more data that has to be sent the the other connected clients.More data causes a longer time to transmit.The longer the transmit time, the more downtime/rejects around a block change/long poll.The more rejects/down time, the lower the efficiency.Lower efficiency, lower returns for miners and pool operators.Lower returns for miners and operators, less of them on the network.Lower network security, etc....

I could go on and on and on.

Also, a side effect is that smaller blocks get propagated through the network faster than larger ones, just because they contain less data. As a result, a block that holds more transactions is more likely to get orphaned than one with less if they are reported as solved at relatively the same time. Orphaned blocks don't make anyone happy, especially pool operators, so pool operators are likely to limit the number of transactions in a block. And that's not good.

Remember the asshat "Mystery miner" that was only pumping out 0 transaction blocks, orphaning blocks that were full of transactions?

Actually the older bitcoin gets and the block reward goes down then TX fee will need to increase as incentives to miners to keep operating.

Will the TX fee need to increase, or will the volume of transactions make up for it? The cost of including a transaction is what... hashing it once for inclusion in the Merkle tree? Given enough transactions, it doesn't seem like that should cost more than 1/4 penny (~what it is now).

For miners, including too many transactions into a mined block increases the risk of collision with a block generated by another miner at approximately the same fime, resulting in one of the blocks being orphaned and that miner losing both the block reward and the fee. In the even of a collision between 2 blocks, it is more likely for the larger block to get orphaned.

There are a bunch of posts over at the mining threads complaining about SatoshiDice being responsible for the recent increase in the number of orphaned blocks. Even with high Tx fees, there is a limit to how many transactions a miner will risk including into a block due to this.

I don't under stand why there Is a larger chance with more TXs, can you elaborate?

Each TX is included in the block so...The more TXs, the larger the block.The larger the block, the more data that has to be sent the the other connected clients.More data causes a longer time to transmit.The longer the transmit time, the more downtime/rejects around a block change/long poll.The more rejects/down time, the lower the efficiency.Lower efficiency, lower returns for miners and pool operators.Lower returns for miners and operators, less of them on the network.Lower network security, etc....

I could go on and on and on.

Also, a side effect is that smaller blocks get propagated through the network faster than larger ones, just because they contain less data. As a result, a block that holds more transactions is more likely to get orphaned than one with less if they are reported as solved at relatively the same time. Orphaned blocks don't make anyone happy, especially pool operators, so pool operators are likely to limit the number of transactions in a block. And that's not good.

Remember the asshat "Mystery miner" that was only pumping out 0 transaction blocks, orphaning blocks that were full of transactions?

i mine but don't pretend to understand the subtleties or mathematics behind your claims here but i have seen theymos claim that the larger data blocks are trivial when it comes down to efficiency (aren't they only in the KB or MB range?). obviously there seems to be some disagreement.

Each TX is included in the block so...The more TXs, the larger the block.The larger the block, the more data that has to be sent the the other connected clients.More data causes a longer time to transmit.The longer the transmit time, the more downtime/rejects around a block change/long poll.The more rejects/down time, the lower the efficiency.Lower efficiency, lower returns for miners and pool operators.Lower returns for miners and operators, less of them on the network.Lower network security, etc....

I could go on and on and on.

Also, a side effect is that smaller blocks get propagated through the network faster than larger ones, just because they contain less data. As a result, a block that holds more transactions is more likely to get orphaned than one with less if they are reported as solved at relatively the same time. Orphaned blocks don't make anyone happy, especially pool operators, so pool operators are likely to limit the number of transactions in a block. And that's not good.

Remember the asshat "Mystery miner" that was only pumping out 0 transaction blocks, orphaning blocks that were full of transactions?

i mine but don't pretend to understand the subtleties or mathematics behind your claims here but i have seen theymos claim that the larger data blocks are trivial when it comes down to efficiency (aren't they only in the KB or MB range?). obviously there seems to be some disagreement.

As far as I understand from watching the IRC discussions, it is not slow because of sending the block from node A to node B. It's the actual validation of the block and all of the transactions on each node that is done before relaying the block.

Bitcoin is a p2p network. No pool has a direct connection to every node. So a pool broadcasts a new block to immediate peers who VALIDATE IT FIRST and then broadcast it to their peers who VALIDATE IT FIRST and then broadcast it to their peers WHO VALIDATE IT FIRST, etc, etc. Usually within 4-5 hops every node on the network knows about it. However due to the randomness of block solving more than 1 miner/pool will solve the same block within 6 seconds roughly 1% of the time.

The issue is that the validation is a non-trivial step. Larger blocks take longer to validate and across multiple hops that latency adds up. Normally this doesn't matter but for a small % of blocks there will be a competing solution. Higher latency = higher % of orphaned blocks. Now if there was NO 50 BTC subsidy it wouldn't really matter. The value of the tx would be worth more than the latency they cause (if you "lose" 1% of blocks but each block is worth 5% more it is still optimal to seek large blocks). The 50 BTC distorts the economics. Say hypothetically that adding 1000+ tx nets you an extra 0.2 BTC in fees but increases your chance of an orphaned block by 0.5%. Well 50.20 / 50 = 0.4% increase in 0.2 BTC revenue but 0.5% of the time you will lose the entire block (-50 BTC). Under such a (not so) hypothetical scenario you would the long run by including all tx, even all paying tx. The token amount of fees don't justify the risk of losing the massive 50 BTC subsidy.

The good news is this is only a temporary problem. Competition for space in block will drive fees higher even if it is only marginally higher (say 0.5 BTC per full block). That combined with falling reward will shift the economics. 25.5 / 25 = 2% boost in revenue. Even if a full block results in more oprhans unless the orphan rate increases by >2% the pool is still coming out ahead. In 4 years when we are looking at fees > 1BTC per block and block reward of 12.5BTC it becomes a "no brainer" to include all/most paying tx in every block.

As far as I understand from watching the IRC discussions, it is not slow because of sending the block from node A to node B. It's the actual validation of the block and all of the transactions on each node that is done before relaying the block.

The 0.7 release optimizes transaction validation, so most transactions that were validated when broadcast/relayed across the network don't have to be re-validated when they're included in a block. That will speed up block verification/relaying a lot.

How often do you get the chance to work on a potentially world-changing project?

Bitcoin is a p2p network. No pool has direction connection to every node. So a pool broadcasts a new block to immediate peers who VALIDATE IT FIRST and then broadcast it to their peers whoVALIDATE IT FIRST and then broadcast it to their peers WHO VALIDATE IT FIRST, etc, etc. usually within 4-5 hops every node on the network knows about it.

The issue is that the validation is a non-trivial step. Larger blocks take longer to validate and across multiple hops that latency adds up. Higher latency = higher % of orphaned blocks. Now if there was NO 50 BTC subsidy it wouldn't really matter. The value of the tx would be worth more than the latency they cause. The 50 BTC distorts the economics. Say hypothetically that adding 1000+ tx nets you 0.2 BTC more fees but increases your chance of an orphaned block by 0.5%. Well 50.20 / 50 = 0.4% increase in revenue but 0.5% of the time you will lose the entire block. You lose revenue in the long run by including all tx, even all paying tx. The token amount of fees don't justify the risk of losing the massive 50 BTC subsidy.

The good news is this is only a temporary problem. Competition for space in block will drive fees higher even if it is only marginally higher (say 0.5 BTC per full block). That combined with falling reward will shift the economics. 25.5 / 25 = 2% boost in revenue. Even if a full block results in more oprhans unless the orphan rate increases by >2% the pool is still coming out ahead. In 4 years when we are looking at fees > 1BTC per block and block reward of 12.5BTC it becomes a "no brainer" to include all/most paying tx in every block.

As far as I understand from watching the IRC discussions, it is not slow because of sending the block from node A to node B. It's the actual validation of the block and all of the transactions on each node that is done before relaying the block.

The 0.7 release optimizes transaction validation, so most transactions that were validated when broadcast/relayed across the network don't have to be re-validated when they're included in a block. That will speed up block verification/relaying a lot.

Could you explain why that also safe in the presence of malicious nodes?

It is non-trivial but it isn't inherently compute-intensive. More of a architectural issue in the Satoshi bitcoin client. Transations are verified while getting included in the mempool. So the verification of the block should consist mostly of querying the mempool and pruning it. Only rare, previously unseen, transactions should require doing the the full cryptographic check.

Doing a full database-querying and signature-verifying check could be done in the background as a sort of self-test for the software implementation.

Edit: I see that Gavin Andersen above said that the upcoming version will incorporate the optimization above. Thank you very much.

Could you explain why that also safe in the presence of malicious nodes?

Current bitcoin versions check signatures twice, which is the slowest/most expensive part of checking a transaction: once when it's broadcast, and again when the block containing it is downloaded. Gavin has changed this to only do the check once, avoiding the duplicate check later.

When 0.7 comes out, upgrade! You will help the health of the network and reduce orphan blocks.