This proposal is backward compatible. Non-supporting peers will ignore

the encinit messages.

This statement is incorrect. Sending content that existing nodes do notexpect is clearly an incompatibility. An implementation that ignoresinvalid content leaves itself wide open to DOS attacks. The versionhandshake must be complete before the protocol level can be determined.While it may be desirable for this change to precede the versionhandshake it cannot be described as backward compatible.

This proposal is backward compatible. Non-supporting peers will ignore

the encinit messages.

This statement is incorrect. Sending content that existing nodes do notexpect is clearly an incompatibility. An implementation that ignoresinvalid content leaves itself wide open to DOS attacks. The versionhandshake must be complete before the protocol level can be determined.While it may be desirable for this change to precede the versionhandshake it cannot be described as backward compatible.

The worst possible effect of ignoring unknown messages is a waste ofdownstream bandwidth. The same is already possible by being sent addrmessages.

Using the protocol level requires a strict linear progression of (allowed)network protocol features, which I expect to become harder and harder tomaintain.

Using otherwise ignored messages for determining optional features iselegant, simple and opens no new attack vectors. I think it's very muchpreferable over continued increments of the protocol version.

This proposal is backward compatible. Non-supporting peers will ignore

the encinit messages.This statement is incorrect. Sending content that existing nodes do notexpect is clearly an incompatibility. An implementation that ignoresinvalid content leaves itself wide open to DOS attacks. The versionhandshake must be complete before the protocol level can be determined.While it may be desirable for this change to precede the versionhandshake it cannot be described as backward compatible.The worst possible effect of ignoring unknown messages is a waste ofdownstream bandwidth. The same is already possible by being sent addrmessages.Using the protocol level requires a strict linear progression of(allowed) network protocol features, which I expect to become harder andharder to maintain.Using otherwise ignored messages for determining optional features iselegant, simple and opens no new attack vectors. I think it's very muchpreferable over continued increments of the protocol version.

As I said, it *may* be desirable, but it is *not* backward compatible,and you do not actually dispute that above.

There are other control messages that qualify as "optional messages" butthese are only sent if the peer is at a version to expect them -explicit in their BIPs. All adopted BIPs to date have followed thispattern. This is not the same and it is not helpful to imply that it isjust following that pattern.

As for DOS, waste of bandwidth is not something to be ignored. If a peeris flooding a node with addr message the node can manage it because itunderstands the semantics of addr messages. If a node is required toallow any message that it cannot understand it has no recourse. Itcannot determine whether it is under attack or if the behavior iscorrect and for proper continued operation must be ignored.

This approach breaks any implementation that validates traffic, which isclearly correct behavior given the existence of the version handshake.Your comments make it clear that this is a *change* in network behavior- essentially abandoning the version handshake. Whether is is harder tomaintain is irrelevant to the question of whether it is a break withexisting protocol.

If you intend for the network to abandon the version handshake and/orpromote changes that break it I propose that you write up this newbehavior as a BIP and solicit community feedback. There are a lot ofdevices connected to the network and it would be irresponsible to breaksomething as fundamental as the P2P protocol handshake because you havea feeling it's going to be hard to maintain.

Look at feefilter BIP 133(https://github.com/bitcoin/bips/blob/master/bip-0133.mediawiki#backward-compatibility)or sendheaders BIP130(https://github.com/bitcoin/bips/blob/master/bip-0130.mediawiki#backward-compatibility)Isn't it the same there?Once BIP151 is implemented, it would make sense to bump the protocolversion, but this needs to be done once this has beenimplemented/deployed. Or do I make a mistake somewhere?

Post by Eric Voskuil via bitcoin-devAs for DOS, waste of bandwidth is not something to be ignored. If a peeris flooding a node with addr message the node can manage it because itunderstands the semantics of addr messages. If a node is required toallow any message that it cannot understand it has no recourse. Itcannot determine whether it is under attack or if the behavior iscorrect and for proper continued operation must be ignored.

How do you threat any other not known message types? Any peer can sendyou any type of message anytime. Why would your implementation how youthreat unknown messages be different for messages specified in BIP151?

Post by Jonas Schnelli via bitcoin-devLook at feefilter BIP 133(https://github.com/bitcoin/bips/blob/master/bip-0133.mediawiki#backward-compatibility)or sendheaders BIP130(https://github.com/bitcoin/bips/blob/master/bip-0130.mediawiki#backward-compatibility)Isn't it the same there?

No. This is what I was referring to. These messages are enabled byprotocol version. If they are received by a node below the version atwhich they are activated, they are unknown messages, implying an invalidpeer. The above messages cannot be sent until *after* the version isnegotiated. BIP151 violates this rule by allowing the new controlmessage to be sent *before* the version handshake.

This indeed is not ideal for compatibility checks, but increases security.I could not find a protocol specification that said communication mustbe terminated when messages are transmitted before the version handshakehas been done. I mostly looked into Bitcoin-Cores implementation (whichmeans also into BitcoinXT/UT, where this is allowed).

Also. BIP151 clearly says that the requesting peer needs to initiate theencryption (encinit).In case of light clients not supporting BIP151 connecting to peerssupporting BIP151, there should never be transmission of new messagetypes specified in BIP151.

Yes, the ordering of the messages. New messages can only be added afterthe handshake negotiates the higher version. Otherwise the handshake isboth irrelevant (as Pieter is implying) and broken (for all existingprotocol versions).

I could not find evidence of the protocol specification that wouldforbid (=terminate connection) such messages and I think allowingunknown-messages before the version handshake makes the protocol flexible.

Are there any reasons we should drop peers if they send us unknown, butcorrectly formatted p2p packages (magic, checksum, etc.) before theversion handshake, ... but not drop them if we have received unknownmessages after the version handshake?

Post by Eric Voskuil via bitcoin-devAs for DOS, waste of bandwidth is not something to be ignored. If a peeris flooding a node with addr message the node can manage it because itunderstands the semantics of addr messages. If a node is required toallow any message that it cannot understand it has no recourse. Itcannot determine whether it is under attack or if the behavior iscorrect and for proper continued operation must be ignored.

How do you threat any other not known message types?

You may be more familiar with non-validating peers. If a message type isnot known it is an invalid message and the peer is immediately dropped.We started seeing early drops in handshakes with bcoin nodes because ofthis issue.

If this had happened, it's very likely because the responding peer triedto initiate a encryption session which is against BIP151 specs.

Sure, a peer can do what it wants. It can send photos. But I'm not surewhat makes you think it would be correct to maintain the connection whenan *invalid* message is received.

Check:https://github.com/bitcoin/bitcoin/blob/a06ede9a138d0fb86b0de17c42b936d9fe6e2158/src/net_processing.cpp#L2595I think it was a wise implementation decision to allow unknown (notinvalid) messages.This had allowed us to deploy stuff like compact blocks, feefilter, etc.without breaking backward compatibility.IMO, without a such flexibility, the deployment complexity would beirresponsible high without really solving the DOS problem.

different for messages specified in BIP151?Because it properly validates the protocol.

For feefilter or compact block or sendheaders?You can't link a (unimplemented) specification (improvement process) toa protocol version before deployment. Or can you?Once it has been widely deployed, we should set a protocol minversionfor BIP151, right.

Post by Jonas Schnelli via bitcoin-devLook at feefilter BIP 133(https://github.com/bitcoin/bips/blob/master/bip-0133.mediawiki#backward-compatibility)or sendheaders BIP130(https://github.com/bitcoin/bips/blob/master/bip-0130.mediawiki#backward-compatibility)Isn't it the same there?

No. This is what I was referring to. These messages are enabled byprotocol version. If they are received by a node below the version atwhich they are activated, they are unknown messages, implying an invalidpeer. The above messages cannot be sent until *after* the version isnegotiated. BIP151 violates this rule by allowing the new controlmessage to be sent *before* the version handshake.

This indeed is not ideal for compatibility checks, but increases security.

The issue I raised is that it is not backward compatible. It sounds likeyou agree but consider it a fair trade. My suggesting was that the BIPbe updated to reflect the lack of compatibility.

Post by Jonas Schnelli via bitcoin-devI could not find a protocol specification that said communication mustbe terminated when messages are transmitted before the version handshakehas been done.

It doesn't need to be specified, most of Bitcoin is unspecified. Theversion handshake establishes the negotiated version. It is not possibleto determine if a message is of the negotiated version before theversion is negotiated. All messages apart from this one have followedthat rule.

An incoming connection will be dropped due to invalid protocol andpotentially banned depending on the implementation.

Post by Jonas Schnelli via bitcoin-devIn case of light clients not supporting BIP151 connecting to peerssupporting BIP151, there should never be transmission of new messagetypes specified in BIP151.

Not working with peers not supporting BIP151 is the compatibility issue.But it sort of seems the intent in this case is to rely on thatincompatibility (expecting connections to nonsupporting peers to fail asopposed to negotiating).

This implies the requesting peer is the peer that sends the message. Youseem to be saying that the requesting peer is the one that initiatedthe connection and the responding peer is the connection receiver. Ifthis is the case it should be more clearly documented. But in thecase I experienced the "requester" of an encrypted session was alsothe "receiver" of the connection.

Yes, the ordering of the messages. New messages can only be added afterthe handshake negotiates the higher version. Otherwise the handshake isboth irrelevant (as Pieter is implying) and broken (for all existingprotocol versions).

I could not find evidence of the protocol specification that wouldforbid (=terminate connection) such messages and I think allowingunknown-messages before the version handshake makes the protocol flexible.

Flexible is certainly one word for it. Another way to describe it isdirty. Allowing invalid messages in a protocol encourages protocolincompatibility. You end up with various implementations and eventuallyhave no way of knowing how they are impacted by changes. There could bea range of peers inter-operating with the full network while runningtheir own sub-protocols. Given the network is public and strongidentification of peers is undesirable, the invalid messages wouldreasonably just get sent to everyone. So over time, what is theprotocol? Due to certain "flexibility" it is already a hassle toproperly implement.

Post by Jonas Schnelli via bitcoin-devAre there any reasons we should drop peers if they send us unknown, butcorrectly formatted p2p packages (magic, checksum, etc.) before theversion handshake, ... but not drop them if we have received unknownmessages after the version handshake?

There is no reason to treat invalid messages differently based on wherethey occur in the communication. After the handshake the agreed versionis known to both peers. As a result there is never a reason for aninvalid message to be sent. Therefore it is always proper to drop a peerthat sends an invalid message.

Post by Eric Voskuil via bitcoin-devAs for DOS, waste of bandwidth is not something to be ignored. If a peeris flooding a node with addr message the node can manage it because itunderstands the semantics of addr messages. If a node is required toallow any message that it cannot understand it has no recourse. Itcannot determine whether it is under attack or if the behavior iscorrect and for proper continued operation must be ignored.

How do you threat any other not known message types?

You may be more familiar with non-validating peers. If a message type isnot known it is an invalid message and the peer is immediately dropped.We started seeing early drops in handshakes with bcoin nodes because ofthis issue.

If this had happened, it's very likely because the responding peer triedto initiate a encryption session which is against BIP151 specs.

Sure, a peer can do what it wants. It can send photos. But I'm not surewhat makes you think it would be correct to maintain the connection whenan *invalid* message is received.

https://github.com/bitcoin/bitcoin/blob/a06ede9a138d0fb86b0de17c42b936d9fe6e2158/src/net_processing.cpp#L2595I think it was a wise implementation decision to allow unknown (notinvalid) messages.This had allowed us to deploy stuff like compact blocks, feefilter, etc.without breaking backward compatibility.IMO, without a such flexibility, the deployment complexity would beirresponsible high without really solving the DOS problem.

This is a misinterpretation. The failure to validate did not enableanything except possibly some broken peers not getting dropped. None ofthe protocol changes previously deployed require the older version peerto allow invalid messages. While it may appear otherwise, due to aparticular implementation, it is never necessary to send a message to apeer that the peer does not understand. The handshake gives each peerthe other peer's version. That obligates the newer peer to conform tothe older (or disconnect if the old is insufficient). That's the natureof backward compatibility.

different for messages specified in BIP151?Because it properly validates the protocol.

For feefilter or compact block or sendheaders?

Yes, this is the purpose of version negotiation, which is why there areversion and verack messages. And this is also why, in the satoshiclient, two of the above messages are sent from the verack handler. Thefeefilter message is sent dynamically but only if the peer's versionallows it.

In general you should set a version before it's ever live on thenetwork. But if it precedes the protocol version negotiation theprotocol version number is moot.

I've been asked to throttle the discussion in the interest of reducinglist volume. I think the issue is pretty clearly addressed at thispoint, but feel free to follow up directly and/or via the libbitcoindevelopment list (copied).

Post by Jonas Schnelli via bitcoin-devI could not find a protocol specification that said communication mustbe terminated when messages are transmitted before the version handshakehas been done.

It doesn't need to be specified, most of Bitcoin is unspecified. Theversion handshake establishes the negotiated version. It is not possibleto determine if a message is of the negotiated version before theversion is negotiated. All messages apart from this one have followedthat rule.

Yes. But encryption negotiation must be done before the versionhandshake (security).

Post by Jonas Schnelli via bitcoin-devIt could very likely be possible that the initial responding peer triesto initiate a encryption session which would mean that BIP151 was notimplemented correctly.Correct me if I'm wrong please.

"Encryption initialization must happen before sending any other messagesto the responding peer (encinit message after a version message must beignored)."https://github.com/bitcoin/bips/blob/master/bip-0151.mediawiki#specification"A peer that supports encryption must accept encryption requests fromall peers... The responding peer accepts the encryption request bysending a encack message."This implies the requesting peer is the peer that sends the message. Youseem to be saying that the requesting peer is the one that initiatedthe connection and the responding peer is the connection receiver. Ifthis is the case it should be more clearly documented. But in thecase I experienced the "requester" of an encrypted session was alsothe "receiver" of the connection.

I think the BIP makes this very clear. IMO you are trying to hide yourstandpoint behind a wired interpretations of the BIP.

From the BIP:Â«To request encrypted communication, the requesting peer generates an ECephemeral-session-keypair and sends an encinit message to the respondingpeer and waits for a encack message. The responding node must do thesame encinit/encack interaction for the opposite communication direction.Â»

This seems to be pretty clear to me. You can interpret the "requestingpeer" and "responding peer" per message interaction. But then the wholeBIP would make no sense.

I'm happy if you can do a PR on the BIP that makes the wording better.This would actually be a productive step.

Yes, the ordering of the messages. New messages can only be added afterthe handshake negotiates the higher version. Otherwise the handshake isboth irrelevant (as Pieter is implying) and broken (for all existingprotocol versions).

I could not find evidence of the protocol specification that wouldforbid (=terminate connection) such messages and I think allowingunknown-messages before the version handshake makes the protocol flexible.

Flexible is certainly one word for it. Another way to describe it isdirty. Allowing invalid messages in a protocol encourages protocolincompatibility. You end up with various implementations and eventuallyhave no way of knowing how they are impacted by changes. There could bea range of peers inter-operating with the full network while runningtheir own sub-protocols. Given the network is public and strongidentification of peers is undesirable, the invalid messages wouldreasonably just get sent to everyone. So over time, what is theprotocol? Due to certain "flexibility" it is already a hassle toproperly implement.

Then you would have to go after all BIPs deployed this way. Thisargument has nothing to do with BIP151 it questions the whole newprotocol features deployment.Again, check this code part:

Post by Jonas Schnelli via bitcoin-devAre there any reasons we should drop peers if they send us unknown, butcorrectly formatted p2p packages (magic, checksum, etc.) before theversion handshake, ... but not drop them if we have received unknownmessages after the version handshake?

There is no reason to treat invalid messages differently based on wherethey occur in the communication. After the handshake the agreed versionis known to both peers. As a result there is never a reason for aninvalid message to be sent. Therefore it is always proper to drop a peerthat sends an invalid message.

That's up to the implementation. But the current flexibility existsbecause we not drop.Again, see above.

Post by Eric Voskuil via bitcoin-devAs for DOS, waste of bandwidth is not something to be ignored. If a peeris flooding a node with addr message the node can manage it because itunderstands the semantics of addr messages. If a node is required toallow any message that it cannot understand it has no recourse. Itcannot determine whether it is under attack or if the behavior iscorrect and for proper continued operation must be ignored.

How do you threat any other not known message types?

You may be more familiar with non-validating peers. If a message type isnot known it is an invalid message and the peer is immediately dropped.We started seeing early drops in handshakes with bcoin nodes because ofthis issue.

Yes, this is the purpose of version negotiation, which is why there areversion and verack messages. And this is also why, in the satoshiclient, two of the above messages are sent from the verack handler. Thefeefilter message is sent dynamically but only if the peer's versionallows it.

Again. Encryption â for the sake of security â must be the firstinteraction.This is exceptional for BIP151 and I'd like to hear the real downsidesof doing that.

I'm not sure I follow your question. The BIP should presumably declare aversion number if one is necessary.

What? You want to define protocol version number in draft improvementspecifications?How should that be possible?It's like defining a new HTML version number if you propose/draft a newvideo streaming format.

For the reasons Pieter listed, an explicit part of our version handshake and protocol negotiation is the exchange of otherwise-ignored messages to set up optional features.

Peers that do not support this ignore such messages, just as if they had indicated they wouldn't support it, see, eg BIP 152's handshake. Not sure why you consider this backwards incompatible, as I would say it's pretty clearly allowing old nodes to communicate just fine.

Post by Pieter Wuille via bitcoin-devWhile it may be desirable for this change to precede the versionhandshake it cannot be described as backward compatible.The worst possible effect of ignoring unknown messages is a waste ofdownstream bandwidth. The same is already possible by being sent addrmessages.Using the protocol level requires a strict linear progression of(allowed) network protocol features, which I expect to become harder

and

Post by Pieter Wuille via bitcoin-devharder to maintain.Using otherwise ignored messages for determining optional features iselegant, simple and opens no new attack vectors. I think it's very

As I said, it *may* be desirable, but it is *not* backward compatible,and you do not actually dispute that above.There are other control messages that qualify as "optional messages" butthese are only sent if the peer is at a version to expect them -explicit in their BIPs. All adopted BIPs to date have followed thispattern. This is not the same and it is not helpful to imply that it isjust following that pattern.As for DOS, waste of bandwidth is not something to be ignored. If a peeris flooding a node with addr message the node can manage it because itunderstands the semantics of addr messages. If a node is required toallow any message that it cannot understand it has no recourse. Itcannot determine whether it is under attack or if the behavior iscorrect and for proper continued operation must be ignored.This approach breaks any implementation that validates traffic, which isclearly correct behavior given the existence of the version handshake.Your comments make it clear that this is a *change* in network behavior- essentially abandoning the version handshake. Whether is is harder tomaintain is irrelevant to the question of whether it is a break withexisting protocol.If you intend for the network to abandon the version handshake and/orpromote changes that break it I propose that you write up this newbehavior as a BIP and solicit community feedback. There are a lot ofdevices connected to the network and it would be irresponsible to breaksomething as fundamental as the P2P protocol handshake because you havea feeling it's going to be hard to maintain.e

handshake and protocol negotiation is the exchange of otherwise-ignoredmessages to set up optional features.https://github.com/bitcoin/bitcoin/blob/master/src/protocol.h#L217-L242https://github.com/bitcoin/bitcoin/blob/master/src/protocol.h#L211-L216https://github.com/bitcoin/bitcoin/blob/master/src/protocol.h#L204-L210https://github.com/bitcoin/bitcoin/blob/master/src/protocol.h#L170-L196

had indicated they wouldn't support it, see, eg BIP 152's handshake.Notsure why you consider this backwards incompatible, as I would say it'spretty clearly allowing old nodes to communicate just fine.No, it is not the same as BIP152. Control messages apart from BIP151arenot sent until *after* the version is negotiated.I assume that BIP151 is different in this manner because it has adesireto negotiate encryption before any other communications, includingversion.e

handshake and protocol negotiation is the exchange of otherwise-ignoredmessages to set up optional features.https://github.com/bitcoin/bitcoin/blob/master/src/protocol.h#L217-L242https://github.com/bitcoin/bitcoin/blob/master/src/protocol.h#L211-L216https://github.com/bitcoin/bitcoin/blob/master/src/protocol.h#L204-L210https://github.com/bitcoin/bitcoin/blob/master/src/protocol.h#L170-L196

had indicated they wouldn't support it, see, eg BIP 152's handshake.Notsure why you consider this backwards incompatible, as I would say it'spretty clearly allowing old nodes to communicate just fine.No, it is not the same as BIP152. Control messages apart from BIP151arenot sent until *after* the version is negotiated.I assume that BIP151 is different in this manner because it has adesireto negotiate encryption before any other communications, includingversion.e

Sorry, I'm still missing it...So your claim is that a) ignoring incoming messages of a type you do not recognize is bad, and thus b) we should be disconnecting/banning peers which send us messages we do not recognize (can you spell out why? Anyone is free to send your host address messages/transactions they are generating/etc/etc, we don't ban nodes for such messages, as that would be crazy - why should we ban a peer for sending us an extra 50 bytes which we ignore?), and thus c) this would be backwards incompatible with software which does not currently exist?

Usually "backwards incompatible" refers to breaking existing software, not breaking theoretical software. Note that, last I heard, BIP 151 is still a draft, if such software actually exists we can discuss changing it, but there are real wins in sending these messages before VERSION.

of version number.https://github.com/bitcoin/bitcoin/blob/master/src/net_processing.cpp#L1372-L1403Inside of the VERACK handler (i.e. after the handshake) there is a peerversion test before sending SENDCMPCT (and SENDHEADERS).I have no idea where the fee filter message is sent, if it is sent atall. But I have *never* seen any control messages arrive before thehandshake is complete.

Post by Matt Corallo via bitcoin-devpretty clearly allowing old nodes to communicate just fine.No, it is not the same as BIP152. Control messages apart from BIP151arenot sent until *after* the version is negotiated.I assume that BIP151 is different in this manner because it has adesireto negotiate encryption before any other communications, includingversion.e