I would like to discuss a way of computing a UTXO set hash that isvery efficient to update, but does not support any compact proofs ofexistence or non-existence.

Much has been written on the topic of various data structures andderived hashes for the UTXO/TXO set before (including Alan Reiner'strust-free lite nodes [1], Peter Todd's TXO MMR commitments [2] [3],or Bram Cohen's TXO bitfield [4]). They all provide interesting extrafunctionality or tradeoffs, but require invasive changes to the P2Pprotocol or how wallets work, or force nodes to maintain theirdatabase in a normative fashion. Instead, here I focus on an efficienthash that supports nothing but comparing two UTXO sets. However, it isnot incompatible with any of those other approaches, so we can gainsome of the advantages of a UTXO hash without adopting something thatmay be incompatible with future protocol enhancements.

1. Incremental hashing

Computing a hash of the UTXO set is easy when it does not needefficient updates, and when we can assume a fixed serialization with anormative ordering for the data in it - just serialize the whole thingand hash it. As different software or releases may use differentdatabase models for the UTXO set, a solution that is order-independentwould seem preferable.

This brings us to the problem of computing a hash of unordered data.Several approaches that accomplish this through incremental hashingwere suggested in [5], including XHASH, AdHash, and MuHash. XHASHconsists of first hashing all the set elements independently, andXORing all those hashes together. This is insecure, as Gaussianelimination can easily find a subset of random hashes that XOR to agiven value. AdHash/MuHash are similar, except addition/multiplicationmodulo a large prime are used instead of XOR. Wagner [6] showed thatattacking XHASH or AdHash is an instance of a generalized birthdayproblem (called the k-sum problem in his paper, with unrestricted k),and gives a O(2^(2*sqrt(n)-1)) algorithm to attack it (for n-bithashes). As a result, AdHash with 256-bit hashes only has 31 bits ofsecurity.

Thankfully, [6] also shows that the k-sum problem cannot beefficiently solved in groups in which the discrete logarithm problemis hard, as an efficient k-sum solver can be used to compute discretelogarithms. As a result, MuHash modulo a sufficiently large safe primeis provably secure under the DL assumption. Common guidelines onsecurity parameters [7] say that 3072-bit DL has about 128 bits ofsecurity. A final 256-bit hash can be applied to the 3072-bit resultwithout loss of security to reduce the final size.

An alternative to multiplication modulo a prime is using an ellipticcurve group. Due to the ECDLP assumption, which the security ofBitcoin signatures already relies on, this also results in securityagainst k-sum solving. This approach is used in the Elliptic CurveMultiset Hash (ECMH) in [8]. For this to work, we must "hash onto acurve point" in a way that results in points without known discretelogarithm. The paper suggests using (controversial) binary ellipticcurves to make that operation efficient. If we only considersecp256k1, one approach is just reading potential X coordinates from aPRNG until one is found that has a corresponding Y coordinateaccording to the curve equation. On average, 2 iterations are needed.A constant time algorithm to hash onto the curve exists as well [9],but it is only slightly faster and is much more complicated toimplement.

AdHash-like constructions with a sufficiently large intermediate hashcan be made secure against Wagner's algorithm, as suggested in [10].4160-bit hashes would be needed for 128 bits of security. Whenrepetition is allowed, [8] gives a stronger attack against AdHash,suggesting that as much as 400000 bits are needed. While repetition isnot directly an issue for our use case, it would be nice ifverification software would not be required to check for duplicatedentries.

2. Efficient addition and deletion

Interestingly, both ECMH and MuHash not only support adding setelements in any order but also deleting in any order. As a result, wecan simply maintain a running sum for the UTXO set as a whole, andadd/subtract when creating/spending an output in it. In the case ofMuHash it is slightly more complicated, as computing an inverse isrelatively expensive. This can be solved by representing the runningvalue as a fraction, and multiplying created elements into thenumerator and spent elements into the denominator. Only when the finalhash is desired, a single modular inverse and multiplication is neededto combine the two.

As the update operations are also associative, H(a)+H(b)+H(c)+H(d) canin fact be computed as (H(a)+H(b)) + (H(c)+H(d)). This implies thatall of this is perfectly parallellizable: each thread can process anarbitrary subset of the update operations, allowing them to beefficiently combined later.

3. Comparison of approaches

Numbers below are based on preliminary benchmarks on a single threadof a i7-6820HQ CPU running at 3.4GHz.

(2) (ECMH) Adding secp256k1 EC points* Much more complicated than the previous approaches whenimplementing from scratch, but almost no extra complexity when ECDSAsecp256k1 signature validation is already implemented.* Using SHA512 + libsecp256k1's point decompression for generatingthe points takes 11us per element on average.* Addition/subtracting of N points takes 5.25us + 0.25us*N.* 64 bytes for a running sum.* Identical security assumption as Bitcoin's signatures.

Using the numbers above, we find that:* Computing the hash from just the UTXO set takes (1) 2m15s (2) 9m20s* Processing all creations and spends in an average block takes (1)24ms (2) 100ms* Processing precomputed per-transaction aggregates in an averageblock takes (1) 3ms (2) 0.5ms

Note that while (2) has higher CPU usage than (1) in general, it haslower latency when using precomputed per-transaction aggregates. Usingsuch aggregates is also more feasible as they're only 64 bytes ratherthan 768. Because of simplicity, (1) has my preference.

Overall, these numbers are sufficiently low (note that they can beparallellized) that it would be reasonable for full nodes and/or othersoftware to always maintain one of them, and effectively have arolling cryptographical checksum of the UTXO set at all times.

4. Use cases

* Replacement for Bitcoin Core's gettxoutsetinfo RPC's hashcomputation. This currently requires minutes of I/O and CPU, as itserializes and hashes the entire UTXO set. A rolling set hash wouldmake this instant, making the whole RPC much more usable for sanitychecking.* Assisting in implementation of fast sync methods with known goodblocks/UTXO sets.* Database consistency checking: by remembering the UTXO set hash ofthe past few blocks (computed on the fly), a consistency check can bedone that recomputes it based on the database.

I wanted to say that I thought this write-up was excellent! And efficiently hashing the UTXO set in this rolling fashion is a very exciting idea!!

Peter R

Post by Pieter Wuille via bitcoin-devHello all,I would like to discuss a way of computing a UTXO set hash that isvery efficient to update, but does not support any compact proofs ofexistence or non-existence.Much has been written on the topic of various data structures andderived hashes for the UTXO/TXO set before (including Alan Reiner'strust-free lite nodes [1], Peter Todd's TXO MMR commitments [2] [3],or Bram Cohen's TXO bitfield [4]). They all provide interesting extrafunctionality or tradeoffs, but require invasive changes to the P2Pprotocol or how wallets work, or force nodes to maintain theirdatabase in a normative fashion. Instead, here I focus on an efficienthash that supports nothing but comparing two UTXO sets. However, it isnot incompatible with any of those other approaches, so we can gainsome of the advantages of a UTXO hash without adopting something thatmay be incompatible with future protocol enhancements.1. Incremental hashingComputing a hash of the UTXO set is easy when it does not needefficient updates, and when we can assume a fixed serialization with anormative ordering for the data in it - just serialize the whole thingand hash it. As different software or releases may use differentdatabase models for the UTXO set, a solution that is order-independentwould seem preferable.This brings us to the problem of computing a hash of unordered data.Several approaches that accomplish this through incremental hashingwere suggested in [5], including XHASH, AdHash, and MuHash. XHASHconsists of first hashing all the set elements independently, andXORing all those hashes together. This is insecure, as Gaussianelimination can easily find a subset of random hashes that XOR to agiven value. AdHash/MuHash are similar, except addition/multiplicationmodulo a large prime are used instead of XOR. Wagner [6] showed thatattacking XHASH or AdHash is an instance of a generalized birthdayproblem (called the k-sum problem in his paper, with unrestricted k),and gives a O(2^(2*sqrt(n)-1)) algorithm to attack it (for n-bithashes). As a result, AdHash with 256-bit hashes only has 31 bits ofsecurity.Thankfully, [6] also shows that the k-sum problem cannot beefficiently solved in groups in which the discrete logarithm problemis hard, as an efficient k-sum solver can be used to compute discretelogarithms. As a result, MuHash modulo a sufficiently large safe primeis provably secure under the DL assumption. Common guidelines onsecurity parameters [7] say that 3072-bit DL has about 128 bits ofsecurity. A final 256-bit hash can be applied to the 3072-bit resultwithout loss of security to reduce the final size.An alternative to multiplication modulo a prime is using an ellipticcurve group. Due to the ECDLP assumption, which the security ofBitcoin signatures already relies on, this also results in securityagainst k-sum solving. This approach is used in the Elliptic CurveMultiset Hash (ECMH) in [8]. For this to work, we must "hash onto acurve point" in a way that results in points without known discretelogarithm. The paper suggests using (controversial) binary ellipticcurves to make that operation efficient. If we only considersecp256k1, one approach is just reading potential X coordinates from aPRNG until one is found that has a corresponding Y coordinateaccording to the curve equation. On average, 2 iterations are needed.A constant time algorithm to hash onto the curve exists as well [9],but it is only slightly faster and is much more complicated toimplement.AdHash-like constructions with a sufficiently large intermediate hashcan be made secure against Wagner's algorithm, as suggested in [10].4160-bit hashes would be needed for 128 bits of security. Whenrepetition is allowed, [8] gives a stronger attack against AdHash,suggesting that as much as 400000 bits are needed. While repetition isnot directly an issue for our use case, it would be nice ifverification software would not be required to check for duplicatedentries.2. Efficient addition and deletionInterestingly, both ECMH and MuHash not only support adding setelements in any order but also deleting in any order. As a result, wecan simply maintain a running sum for the UTXO set as a whole, andadd/subtract when creating/spending an output in it. In the case ofMuHash it is slightly more complicated, as computing an inverse isrelatively expensive. This can be solved by representing the runningvalue as a fraction, and multiplying created elements into thenumerator and spent elements into the denominator. Only when the finalhash is desired, a single modular inverse and multiplication is neededto combine the two.As the update operations are also associative, H(a)+H(b)+H(c)+H(d) canin fact be computed as (H(a)+H(b)) + (H(c)+H(d)). This implies thatall of this is perfectly parallellizable: each thread can process anarbitrary subset of the update operations, allowing them to beefficiently combined later.3. Comparison of approachesNumbers below are based on preliminary benchmarks on a single threadof a i7-6820HQ CPU running at 3.4GHz.(1) (MuHash) Multiplying 3072-bit hashes mod 2^3072 - 1103717 (thelargest 3072-bit safe prime).* Needs a fast modular multiplication/inverse implementation.* Using SHA512 + ChaCha20 for generating the hashes takes 1.2us per element.* Modular multiplication using GMP takes 1.5us per element (2.5uswith a 60-line C+asm implementation).* 768 bytes for maintaining a running sum (384 for numerator, 384for denominator)* Very common security assumption. Even if the DL assumption wouldbe broken (but no k-sum algorithm faster than Wagner's is found), thisstill maintains 110 bits of security.(2) (ECMH) Adding secp256k1 EC points* Much more complicated than the previous approaches whenimplementing from scratch, but almost no extra complexity when ECDSAsecp256k1 signature validation is already implemented.* Using SHA512 + libsecp256k1's point decompression for generatingthe points takes 11us per element on average.* Addition/subtracting of N points takes 5.25us + 0.25us*N.* 64 bytes for a running sum.* Identical security assumption as Bitcoin's signatures.* Computing the hash from just the UTXO set takes (1) 2m15s (2) 9m20s* Processing all creations and spends in an average block takes (1)24ms (2) 100ms* Processing precomputed per-transaction aggregates in an averageblock takes (1) 3ms (2) 0.5msNote that while (2) has higher CPU usage than (1) in general, it haslower latency when using precomputed per-transaction aggregates. Usingsuch aggregates is also more feasible as they're only 64 bytes ratherthan 768. Because of simplicity, (1) has my preference.Overall, these numbers are sufficiently low (note that they can beparallellized) that it would be reasonable for full nodes and/or othersoftware to always maintain one of them, and effectively have arolling cryptographical checksum of the UTXO set at all times.4. Use cases* Replacement for Bitcoin Core's gettxoutsetinfo RPC's hashcomputation. This currently requires minutes of I/O and CPU, as itserializes and hashes the entire UTXO set. A rolling set hash wouldmake this instant, making the whole RPC much more usable for sanitychecking.* Assisting in implementation of fast sync methods with known goodblocks/UTXO sets.* Database consistency checking: by remembering the UTXO set hash ofthe past few blocks (computed on the fly), a consistency check can bedone that recomputes it based on the database.[1] https://bitcointalk.org/index.php?topic=88208.0[2] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-May/012715.html[3] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-February/013591.html[4] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-March/013928.html[5] https://cseweb.ucsd.edu/~mihir/papers/inchash.pdf[6] https://people.eecs.berkeley.edu/~daw/papers/genbday.html[7] https://www.keylength.com/[8] https://arxiv.org/pdf/1601.06502.pdf[9] https://www.di.ens.fr/~fouque/pub/latincrypt12.pdf[10] http://csrc.nist.gov/groups/ST/hash/sha-3/Aug2014/documents/gligoroski_paper_sha3_2014_workshop.pdfCheers,--Pieter_______________________________________________bitcoin-dev mailing listhttps://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Post by Pieter Wuille via bitcoin-dev4. Use cases* Replacement for Bitcoin Core's gettxoutsetinfo RPC's hashcomputation. This currently requires minutes of I/O and CPU, as itserializes and hashes the entire UTXO set. A rolling set hash wouldmake this instant, making the whole RPC much more usable for sanitychecking.* Assisting in implementation of fast sync methods with known goodblocks/UTXO sets.* Database consistency checking: by remembering the UTXO set hash ofthe past few blocks (computed on the fly), a consistency check can bedone that recomputes it based on the database.

Another use case I can think of is a potential "chain-flip" hard fork of block header formats, where the UTXO hash rather than merkle tree root of transactions is in the header, which would let lite nodes download a UTXO set from any full node and verify it by verifying only block headers starting from genesis.

Post by ZmnSCPxj via bitcoin-devtransactions is in the header, which would let lite nodes download a UTXOset from any full node and verify it by verifying only block headersstarting from genesis.

Ya, lite nodes with UTXO sets are one of the the oldest observedadvantages of a commitment to the UTXO data:

https://bitcointalk.org/index.php?topic=21995.0

But it requires a commitment. And for most of the arguments for thoseyou really want compact membership proofs. The recent rise ininterest in full block lite clients (for privacy reasons), perhapscomplements the membership proofless usage.

Pieter describes some uses for doing something like this without acommitment. In my view, it's more interesting to first gainexperience with an operation without committing to it (which is aconsensus change and requires more care and consideration, which areeasier if people have implementation experience).

For audibility and engineering reasons it would need to be be inaddition to rather than rather than, because the proof of work needsto commit to the witness data (in that kind of flip, the transactionsthemselves become witnesses for UTXO deltas) or you get trivial DOSattacks where people provide malleated blocks that have invalidwitnesses.

Post by ZmnSCPxj via bitcoin-devtransactions is in the header, which would let lite nodes download a UTXOset from any full node and verify it by verifying only block headersstarting from genesis.

Ya, lite nodes with UTXO sets are one of the the oldest observedhttps://bitcointalk.org/index.php?topic=21995.0But it requires a commitment. And for most of the arguments for thoseyou really want compact membership proofs. The recent rise ininterest in full block lite clients (for privacy reasons), perhapscomplements the membership proofless usage.Pieter describes some uses for doing something like this without acommitment. In my view, it's more interesting to first gainexperience with an operation without committing to it (which is aconsensus change and requires more care and consideration, which areeasier if people have implementation experience).

To be clear, *none* of the previous (U)TXO commitment schemes require *miners*to participate in generating a commitment. While that was previously thought tobe true by many, I've seen no counter-arguments to the argument I published Ifew months ago(1) that (U)TXO commitments did not require a soft-fork todeploy.

1) "[bitcoin-dev] TXO commitments do not need a soft-fork to be useful",Peter Todd, Feb 23 2017,https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-February/013591.html

Post by Peter Todd via bitcoin-devTo be clear, *none* of the previous (U)TXO commitment schemes require *miners*to participate in generating a commitment. While that was previously thought tobe true by many, I've seen no counter-arguments to the argument I published Ifew months ago(1) that (U)TXO commitments did not require a soft-fork todeploy.1) "[bitcoin-dev] TXO commitments do not need a soft-fork to be useful",Peter Todd, Feb 23 2017,https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-February/013591.html

I'm aware, I agree, and I even referenced that mail in my original post.

However, all of those approaches still require a network wide choiceto be useful. A validating node that does not maintain a UTXO X mustget a proof of its unspentness from somewhere for at least the blockwhich contains a spend of X. In a world where such a model is deployednetwork-wide, that proof information is generated by the wallet andrelayed wherever needed. In a partial deployment however, you neednodes that can produce the proof for other nodes, and the ability toproduce a proof is significantly more expensive than running either anold or a new full node.

This ability to produce proofs becomes even harder when there aredifferent models deployed at once. Even just having a differentcriterion for which UTXOs need a proof (eg. "only outputs created morethan 1000 blocks ago") may already cause compatibility issues. Combinethat with the multitude of ideas about this (insertion-ordered TXOtrees, txid-ordered UTXO Patricia tries, AVL+ trees, append-onlybitfield, ...) with different trade-offs (in CPU, RAM for validators,complexity for wallets/index services, ...), I don't think we're quiteready to make that choice.

To be clear: I'm very much in favor of moving to a model where theresponsibilities of full nodes are reduced in the long term. Butbefore that can happen there will need to be implementations,experiments, analysis, ...

Because of that, I think it is worthwhile to investigate solutions tothe "how can we efficiently compare UTXO sets" problem separately fromthe "how do we reduce full node costs by sending proofs instead of itmaintaining the data". And rolling UTXO set hashes are a solution forjust the first - and one that has very low costs and no normativedatastructures at all.

I do prefer the (2) approach, BTW, as it reuses existing primitives, butI know "simpler" means a different thing to mathier brains :)

Since it wasn't explicit in the proposal, I think the txout informationplaced in the hash here is worth discussing.

I prefer a simple txid||outnumber[1], because it allows simple validationwithout knowing the UTXO set itself; even a lightweight node can assertthat UTXOhash for block N+1 is valid if the UTXOhash for block N isvalid (and vice versa!) given block N+1. And miners can't really usethat even if they were to try not validating against UTXO (!) becausethey need to know input amounts for fees (which are becomingsignificant).

If I want to hand you the complete validatable UTXO set, I need to handyou all the txs with any unspent output, and some bitfield to indicatewhich ones are unspent.

OTOH, if you serialize more (eg. ...||amount||scriptPubKey ?), then the UTXOset size needed to validate the utxohash is a little smaller: you needto send the txid, but not the tx nVersion, nLocktime or inputs. But in aSegWit world, that's actually *bigger* AFAICT.

Thanks,Rusty.

[1] I think you could actually use txid^outnumber, and if that's not acurve point SHA256() again, etc. But I don't think that saves anyreal time, and may cause other issues.

Post by ZmnSCPxj via bitcoin-devtransactions is in the header, which would let lite nodes download a UTXOset from any full node and verify it by verifying only block headersstarting from genesis.

Ya, lite nodes with UTXO sets are one of the the oldest observedhttps://bitcointalk.org/index.php?topic=21995.0But it requires a commitment. And for most of the arguments for thoseyou really want compact membership proofs. The recent rise ininterest in full block lite clients (for privacy reasons), perhapscomplements the membership proofless usage.Pieter describes some uses for doing something like this without acommitment. In my view, it's more interesting to first gainexperience with an operation without committing to it (which is aconsensus change and requires more care and consideration, which areeasier if people have implementation experience).

For audibility and engineering reasons it would need to be be inaddition to rather than rather than, because the proof of work needsto commit to the witness data (in that kind of flip, the transactionsthemselves become witnesses for UTXO deltas) or you get trivial DOSattacks where people provide malleated blocks that have invalidwitnesses.

Another thought I have, is that instead of committing to the UTXO of the block, to commit to the UTXO of the previous block, and the merkle tree root of the transactions in the current block.

My thought is that this would help reduce SPV mining, as a miner would need to actually scan any received new blocks in order to create the UTXO set of the previous block. An empty block would make things easier for the next block's miner, not the current block's miner. However, I'm not sure if my understanding is correct, or if there is some subtlety I missed in this regard.