We’d like to have authentication between Flume Nodes and Clients.I believe that currently there’s no easy way to set up any kind of authentication between Avro Sources, Sinks, and RpcClients.

I would like to ask what would be the best way to extend Flume to support authentication?Would it be enough to extend or rewrite Avro Source, Sink and RpcClient?Does Avro or Netty support authentication?

We’d prefer using Kerberos or maybe SPNEGO, but I’m not sure what options do we have.

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.

> Hi Flume Users,****>> ** **>> We’d like to have authentication between Flume Nodes and Clients.****>> I believe that currently there’s no easy way to set up any kind of> authentication between Avro Sources, Sinks, and RpcClients.****>> ** **>> I would like to ask what would be the best way to extend Flume to support> authentication?****>> Would it be enough to extend or rewrite Avro Source, Sink and RpcClient?**> **>> Does Avro or Netty support authentication?****>> ** **>> We’d prefer using Kerberos or maybe SPNEGO, but I’m not sure what options> do we have.****>> ** **>> Thanks and regards,****>> Rudolf****>> ** **>> Rudolf Rakos> *Morgan Stanley | ISG Technology> *Lechner Odon fasor 8 | Floor 06> Budapest, 1095> Phone: +36 1 881-4011> [EMAIL PROTECTED]>>> Be carbon conscious. Please consider our environment before printing this> email.> ****>>> ------------------------------>> NOTICE: Morgan Stanley is not acting as a municipal advisor and the> opinions or views contained herein are not intended to be, and do not> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall> Street Reform and Consumer Protection Act. If you have received this> communication in error, please destroy all electronic and paper copies and> notify the sender immediately. Mistransmission is not intended to waive> confidentiality or privilege. Morgan Stanley reserves the right, to the> extent permitted under applicable law, to monitor electronic> communications. This message is subject to terms available at the following> link: http://www.morganstanley.com/disclaimers If you cannot access these> links, please notify us by reply message and we will send the contents to> you. By messaging with Morgan Stanley you consent to the foregoing.>>

We’d like to have authentication between Flume Nodes and Clients.I believe that currently there’s no easy way to set up any kind of authentication between Avro Sources, Sinks, and RpcClients.

I would like to ask what would be the best way to extend Flume to support authentication?Would it be enough to extend or rewrite Avro Source, Sink and RpcClient?Does Avro or Netty support authentication?

We’d prefer using Kerberos or maybe SPNEGO, but I’m not sure what options do we have.

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.

________________________________

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.

> I think that maybe it’s possible to use Avro RPCPlugins to inject> authentication data during the Avro handshake.****>> ** **>> Thanks for your help Mike.****>> ** **>> Regards,****>> Rudolf****>> ** **>> *From:* Mike Percy [mailto:[EMAIL PROTECTED] <javascript:_e({}, 'cvml',> '[EMAIL PROTECTED]');>]> *Sent:* Wednesday, January 23, 2013 9:16 PM> *To:* [EMAIL PROTECTED] <javascript:_e({}, 'cvml',> '[EMAIL PROTECTED]');>> *Subject:* Re: Authentication - Avro Source, Sink, RpcClient****>> ** **>> I agree that AvroSource/Sink SASL and Kerberos auth would be really> useful. It would need some work at the Avro level, though.****>> ** **>> There is also the possibility of doing the same thing on top of Thrift, in> which case it would require a brand new source/sink/client implementation> but it wouldn't require any protocol work AFAICT.****>> ** **>> Regards****>> Mike****>> ** **>> On Wed, Jan 23, 2013 at 3:45 AM, Rakos, Rudolf <> [EMAIL PROTECTED] <javascript:_e({}, 'cvml',> '[EMAIL PROTECTED]');>> wrote:****>> Hi Flume Users,****>> ****>> We’d like to have authentication between Flume Nodes and Clients.****>> I believe that currently there’s no easy way to set up any kind of> authentication between Avro Sources, Sinks, and RpcClients.****>> ****>> I would like to ask what would be the best way to extend Flume to support> authentication?****>> Would it be enough to extend or rewrite Avro Source, Sink and RpcClient?**> **>> Does Avro or Netty support authentication?****>> ****>> We’d prefer using Kerberos or maybe SPNEGO, but I’m not sure what options> do we have.****>> ****>> Thanks and regards,****>> Rudolf****>> ****>> Rudolf Rakos> *Morgan Stanley | ISG Technology> *Lechner Odon fasor 8 | Floor 06> Budapest, 1095> Phone: +36 1 881-4011> [EMAIL PROTECTED] <javascript:_e({}, 'cvml',> '[EMAIL PROTECTED]');>>>> Be carbon conscious. Please consider our environment before printing this> email.> ****>> ** **> ------------------------------>>> NOTICE: Morgan Stanley is not acting as a municipal advisor and the> opinions or views contained herein are not intended to be, and do not> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall> Street Reform and Consumer Protection Act. If you have received this> communication in error, please destroy all electronic and paper copies and> notify the sender immediately. Mistransmission is not intended to waive> confidentiality or privilege. Morgan Stanley reserves the right, to the> extent permitted under applicable law, to monitor electronic> communications. This message is subject to terms available at the following> link: http://www.morganstanley.com/disclaimers If you cannot access these> links, please notify us by reply message and we will send the contents to> you. By messaging with Morgan Stanley you consent to the foregoing.****>> ** **>>> ------------------------------>> NOTICE: Morgan Stanley is not acting as a municipal advisor and the> opinions or views contained herein are not intended to be, and do not> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall> Street Reform and Consumer Protection Act. If you have received this> communication in error, please destroy all electronic and paper copies and> notify the sender immediately. Mistransmission is not intended to waive> confidentiality or privilege. Morgan Stanley reserves the right, to the> extent permitted under applicable law, to monitor electronic> communications. This message is subject to terms available at the following> link: http://www.morganstanley.com/disclaimers If you cannot access these> links, please notify us by reply message and we will send the contents to> you. By messaging with Morgan Stanley you consent to the foregoing.

I had to make 4 small changes.Unfortunately I couldn’t subclass these classes, but I could reuse most of the code.· First I added an “auth” configuration option to AvroSource and AvroSink, to be able to enable / disable authentication.

Instead of getting a client using the Transceiver:avroClient = SpecificRequestor.getClient(AvroSourceProtocol.Callback.class, transceiver);

I get a client using a Requestor:SpecificRequestor requestor = new SpecificRequestor(AvroSourceProtocol.Callback.class, transceiver);if (auth) { requestor.addRPCPlugin(new AuthenticatorRPCPlugin());}avroClient = SpecificRequestor.getClient(AvroSourceProtocol.Callback.class, requestor);

The AuthenticatorRPCPlugin is an org.apache.avro.ipc.RPCPlugin with 3 methods overriden to exchange the authentication data during the Avro handshake:

1. clientStartConnect(org.apache.avro.ipc.RPCContext)This is called on the client, before the handshake.

2. serverConnecting(org.apache.avro.ipc.RPCContext)This is called on the server. The connection will be aborted, if an exception is thrown here.

3. clientFinishConnect(org.apache.avro.ipc.RPCContext)This is called on the client, after the handshake.The org.apache.avro.ipc.RPCContext#getHandshakeRequest() returns an org.apache.avro.ipc.HandshakeRequest which contains a java.util.Map<java.lang.String, java.nio.ByteBuffer> where I could store the authentication data (org.apache.avro.ipc.HandshakeRequest#getMeta and org.apache.avro.ipc.HandshakeRequest#setMeta methods).

This might not be the nicest way to implement authentication, but this way it’s pretty much transparent to Flume.I think it would be pretty easy to implement some kind of encryption too using the other org.apache.avro.ipc.RPCPlugin methods.

We’d like to have authentication between Flume Nodes and Clients.I believe that currently there’s no easy way to set up any kind of authentication between Avro Sources, Sinks, and RpcClients.

I would like to ask what would be the best way to extend Flume to support authentication?Would it be enough to extend or rewrite Avro Source, Sink and RpcClient?Does Avro or Netty support authentication?

We’d prefer using Kerberos or maybe SPNEGO, but I’m not sure what options do we have.

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.________________________________

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.co

Rudolf, wow interesting! So it's working, huh? Can you submit a patch?

Regards,Mike

On Tuesday, February 5, 2013, Rakos, Rudolf wrote:

> Yes, I think so.****>> ** **>> I had to make 4 small changes. ****>> Unfortunately I couldn’t subclass these classes, but I could reuse most of> the code.****>> ** **>> **· **First I added an “auth” configuration option to AvroSource> and AvroSink, to be able to enable / disable authentication.****>> **· **AvroSource#start()>> Responder responder = *new* SpecificResponder(AvroSourceProtocol.*class*,> *this*);> *if* (auth) {> responder.addRPCPlugin(*new* AuthenticatorRPCPlugin());> }>> ****>> **· **AvroSink#createConnection()>> Instead of using RpcClientFactory:> client = RpcClientFactory.getInstance(clientProps);>> I create my own RpcClient (I didn’t want to modify RpcClientFactory):> client = *new* AuthNettyAvroRpcClient(auth, ...);>> ****>> **· **AuthNettyAvroRpcClient#connect(long,> java.util.concurrent.TimeUnit)>> Instead of getting a client using the Transceiver:> avroClient = SpecificRequestor.getClient(AvroSourceProtocol.Callback.*> class*, transceiver);>> I get a client using a Requestor:> SpecificRequestor requestor = *new*SpecificRequestor(AvroSourceProtocol.Callback.> *class*, transceiver);> *if* (auth) {> requestor.addRPCPlugin(*new* AuthenticatorRPCPlugin());> }> avroClient = SpecificRequestor.*getClient*(AvroSourceProtocol.Callback.*> class*, requestor);****>> ** **>> The AuthenticatorRPCPlugin is an org.apache.avro.ipc.RPCPlugin with 3> methods overriden to exchange the authentication data during the Avro> handshake:****>> **1. **clientStartConnect(org.apache.avro.ipc.RPCContext)> This is called on the client, before the handshake.****>> **2. **serverConnecting(org.apache.avro.ipc.RPCContext)> This is called on the server. The connection will be aborted, if an> exception is thrown here.****>> **3. **clientFinishConnect(org.apache.avro.ipc.RPCContext)> This is called on the client, after the handshake.****>> The org.apache.avro.ipc.RPCContext#getHandshakeRequest() returns an> org.apache.avro.ipc.HandshakeRequest which contains a java.util.Map<java.lang.String,> java.nio.ByteBuffer> where I could store the authentication data (> org.apache.avro.ipc.HandshakeRequest#getMeta andorg.apache.avro.ipc.HandshakeRequest#> setMeta methods).****>> ** **>> This might not be the nicest way to implement authentication, but this way> it’s pretty much transparent to Flume.****>> I think it would be pretty easy to implement some kind of encryption too> using the other org.apache.avro.ipc.RPCPlugin methods.****>> ** **>> Regards,****>> Rudolf****>> ** **>> *From:* Mike Percy [mailto:[EMAIL PROTECTED] <javascript:_e({}, 'cvml',> '[EMAIL PROTECTED]');>]> *Sent:* Tuesday, February 05, 2013 9:33 AM> *To:* [EMAIL PROTECTED] <javascript:_e({}, 'cvml',> '[EMAIL PROTECTED]');>> *Subject:* Re: Authentication - Avro Source, Sink, RpcClient****>> ** **>> Rudolf did you try it? Any luck?****>> ** **>> Regards****>> Mike>> On Friday, January 25, 2013, Rakos, Rudolf wrote:****>> I think that maybe it’s possible to use Avro RPCPlugins to inject> authentication data during the Avro handshake.****>> ****>> Thanks for your help Mike.****>> ****>> Regards,****>> Rudolf****>> ****>> *From:* Mike Percy [mailto:[EMAIL PROTECTED]]> *Sent:* Wednesday, January 23, 2013 9:16 PM> *To:* [EMAIL PROTECTED]> *Subject:* Re: Authentication - Avro Source, Sink, RpcClient****>> ****>> I agree that AvroSource/Sink SASL and Kerberos auth would be really> useful. It would need some work at the Avro level, though.****>> ****>> There is also the possibility of doing the same thing on top of Thrift, in> which case it would require a brand new source/sink/client implementation> but it wouldn't require any protocol work AFAICT.****

Rudolf, wow interesting! So it's working, huh? Can you submit a patch?

Regards,Mike

On Tuesday, February 5, 2013, Rakos, Rudolf wrote:Yes, I think so.

I had to make 4 small changes.Unfortunately I couldn’t subclass these classes, but I could reuse most of the code.• First I added an “auth” configuration option to AvroSource and AvroSink, to be able to enable / disable authentication.

Instead of getting a client using the Transceiver:avroClient = SpecificRequestor.getClient(AvroSourceProtocol.Callback.class, transceiver);

I get a client using a Requestor:SpecificRequestor requestor = new SpecificRequestor(AvroSourceProtocol.Callback.class, transceiver);if (auth) { requestor.addRPCPlugin(new AuthenticatorRPCPlugin());}avroClient = SpecificRequestor.getClient(AvroSourceProtocol.Callback.class, requestor);

The AuthenticatorRPCPlugin is an org.apache.avro.ipc.RPCPlugin with 3 methods overriden to exchange the authentication data during the Avro handshake:

1. clientStartConnect(org.apache.avro.ipc.RPCContext)This is called on the client, before the handshake.

2. serverConnecting(org.apache.avro.ipc.RPCContext)This is called on the server. The connection will be aborted, if an exception is thrown here.

3. clientFinishConnect(org.apache.avro.ipc.RPCContext)This is called on the client, after the handshake.The org.apache.avro.ipc.RPCContext#getHandshakeRequest() returns an org.apache.avro.ipc.HandshakeRequest which contains a java.util.Map<java.lang.String, java.nio.ByteBuffer> where I could store the authentication data (org.apache.avro.ipc.HandshakeRequest#getMeta and org.apache.avro.ipc.HandshakeRequest#setMeta methods).

This might not be the nicest way to implement authentication, but this way it’s pretty much transparent to Flume.I think it would be pretty easy to implement some kind of encryption too using the other org.apache.avro.ipc.RPCPlugin methods.

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained her

________________________________

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.______

+

Rakos, Rudolf 2013-02-07, 09:29

NEW: Monitor These Apps!

All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by Sematext