Sanjay Radia
added a comment - 01/Jun/11 05:20 The attached document proposes a design to achieve wire compatibility.
The design is complementary to using AVRO/PB like serialization technologies.
Further some of the steps such a separating data types are necessary to use such serialization technologies.
The design, can be used to provide compatibility from non-avro/pb (say 23) to avro/pb release

I see in the table on page 5 you list a 20 client will need a client adapter to communicate with a 22/23 server, but 20 does not support client adapters. Is this intended just to show what would be needed, is it a typo, or is the intention to backport client adapter support to 0.20?

The document does not talk about port numbers or protocol detection. If the intention is to support adapters and the base protocol offers enough compatibility to detect protocol version which we currently have then it seems straight forward. But what about when we switch to Avro/PB. Is the intention to continue to use the same port but with a different protocol, and if so how is protocol detection going to happen, or is the intention to switch to a new port number?

The design also does not explicitly state anything about sunsetting support for older adapters. I would like to see it called out explicitly that we will not support all adapters on all versions and also a statement about what minimum support is guaranteed. It would be a QA nightmare to try and support all versions.

Robert Joseph Evans
added a comment - 15/Jul/11 15:20 I like the design. It seems to be well though out.
I see in the table on page 5 you list a 20 client will need a client adapter to communicate with a 22/23 server, but 20 does not support client adapters. Is this intended just to show what would be needed, is it a typo, or is the intention to backport client adapter support to 0.20?
The document does not talk about port numbers or protocol detection. If the intention is to support adapters and the base protocol offers enough compatibility to detect protocol version which we currently have then it seems straight forward. But what about when we switch to Avro/PB. Is the intention to continue to use the same port but with a different protocol, and if so how is protocol detection going to happen, or is the intention to switch to a new port number?
The design also does not explicitly state anything about sunsetting support for older adapters. I would like to see it called out explicitly that we will not support all adapters on all versions and also a statement about what minimum support is guaranteed. It would be a QA nightmare to try and support all versions.

Adapters are a powerful concept, but difficult to maintain. I wonder how soon we require them.

Adapters would be required to support interoperability between the pre-evolvable parameter world and the evolvable parameter world since parameter serialization will change fundamentally in that transition. But we only expect to make that transition just once, so we perhaps needn't build a general mechanism capable of handling such transitions?

Once we transition I think we'll be able to handle interoperability between adjacent major versions using your rules 1-3. Prior to the transition we would not provide wire compatibility between major versions. Do we expect protocol changes that we cannot handle with rules 1-3?

Perhaps it comes down to whether we're willing to have another non-interoperable major release. These are unfortunate, but a non-interoperable release is not a regression, so I don't see it as a hard requirement that we don't have another before we add interoperability as a feature.

If interoperability is a requirement for future releases, then perhaps we should push HDFS-2060 into 0.23. If we did that and in some subsequent release decide we wish to incompatibly alter wire formats in some way that cannot be handled with rules 1-3 then we could implement adapters. But we might have a few interoperable releases in the meantime.

Doug Cutting
added a comment - 08/Sep/11 18:05 Adapters are a powerful concept, but difficult to maintain. I wonder how soon we require them.
Adapters would be required to support interoperability between the pre-evolvable parameter world and the evolvable parameter world since parameter serialization will change fundamentally in that transition. But we only expect to make that transition just once, so we perhaps needn't build a general mechanism capable of handling such transitions?
Once we transition I think we'll be able to handle interoperability between adjacent major versions using your rules 1-3. Prior to the transition we would not provide wire compatibility between major versions. Do we expect protocol changes that we cannot handle with rules 1-3?
Perhaps it comes down to whether we're willing to have another non-interoperable major release. These are unfortunate, but a non-interoperable release is not a regression, so I don't see it as a hard requirement that we don't have another before we add interoperability as a feature.
If interoperability is a requirement for future releases, then perhaps we should push HDFS-2060 into 0.23. If we did that and in some subsequent release decide we wish to incompatibly alter wire formats in some way that cannot be handled with rules 1-3 then we could implement adapters. But we might have a few interoperable releases in the meantime.