Currently kafka broker config is all statically defined in a propertiesfile with the broker. This mostly works pretty well, but for per-topicconfiguration (the flush policy, partition count, etc) it is pretty painfulto have to bounce the broker every time you make a config change.

Currently each of our topic-level configs is paired with a default. So youwould have something like segment.size.byteswhich would be the default, and then you can override this for topics thatneed something different using a map: segment.size.bytes.per.topic

The proposal is to move the topic configuration into zookeeper so that fora topic "my-topic" we would have a znode /brokers/topics/my-topic/configand the contents of this znode would be the topic configuration either asjson or properties or whatever.

There are two ways this config could work:1. Defaults resolved at topic creation time: At the time a topic is createdthe user would specify some properties they wanted for that topic, anytopic they didn't specify would take the server default. ALL theseproperties would be stored in the znode.2. Defaults resolved at config read time: When a topic is created the userspecifies particularly properties they want and ONLY the properties theyparticularly specify would be stored. At runtime we would merge theseproperties with whatever the server defaults currently are.

This is a somewhat nuanced point, but perhaps important.

The advantage of the first proposal is that it is simple. If you want toknow the configuration for a particular topic you go to zookeeper and lookat that topics config. Mixing the combination of server config andzookeeper config dynamically makes it a little harder to figure out whatthe current state of anything is.

The disadvantage of the first proposal (and the advantage of the secondproposal) is that making global changes is easier. For example if you wantto globally lower the retention for all topics, in proposal one you wouldhave to iterate over all topics and update the config (this could be doneautomatically with tooling, but under the covers the tool would do this).In the second case you would just update the default value.

Also can we abstract the config call too? We have so much in chef, it's not that i don't want to call our zookeeper cluster for it but we don't have our topology yet mapped out in znodes they are in our own instances of code.

It should have both a pull and push for changes, one thing that's nice with zookeeper and having a watcher.

> Currently kafka broker config is all statically defined in a properties> file with the broker. This mostly works pretty well, but for per-topic> configuration (the flush policy, partition count, etc) it is pretty painful> to have to bounce the broker every time you make a config change.> > That lead to this proposal:> https://cwiki.apache.org/confluence/display/KAFKA/Dynamic+Topic+Config> > An open question is how topic-default configurations should work.> > Currently each of our topic-level configs is paired with a default. So you> would have something like> segment.size.bytes> which would be the default, and then you can override this for topics that> need something different using a map:> segment.size.bytes.per.topic> > The proposal is to move the topic configuration into zookeeper so that for> a topic "my-topic" we would have a znode> /brokers/topics/my-topic/config> and the contents of this znode would be the topic configuration either as> json or properties or whatever.> > There are two ways this config could work:> 1. Defaults resolved at topic creation time: At the time a topic is created> the user would specify some properties they wanted for that topic, any> topic they didn't specify would take the server default. ALL these> properties would be stored in the znode.> 2. Defaults resolved at config read time: When a topic is created the user> specifies particularly properties they want and ONLY the properties they> particularly specify would be stored. At runtime we would merge these> properties with whatever the server defaults currently are.> > This is a somewhat nuanced point, but perhaps important.> > The advantage of the first proposal is that it is simple. If you want to> know the configuration for a particular topic you go to zookeeper and look> at that topics config. Mixing the combination of server config and> zookeeper config dynamically makes it a little harder to figure out what> the current state of anything is.> > The disadvantage of the first proposal (and the advantage of the second> proposal) is that making global changes is easier. For example if you want> to globally lower the retention for all topics, in proposal one you would> have to iterate over all topics and update the config (this could be done> automatically with tooling, but under the covers the tool would do this).> In the second case you would just update the default value.> > Thoughts? If no one cares, I will just pick whatever seems best.> > -Jay

I am not sure if I understand what you are proposing, though. Are yousaying support both the config file and zk for topic-level configs? I hateto do things where the answer is "do both"...I guess I feel that althougheveryone walks away happy it ends up being a lot of code and combinatorialtesting. So if there is a different plan that hits all requirements I likethat better. I am very sensitive to the fact that zookeeper is an okaykey/value store but a really poor replacement for a config managementsystem. It might be worth while to try to work out a way that meets allneeds, if such a thing exists.

Is bouncing brokers for topic-overrides a problem for you in yourenvironment? If so how would you fix it?

-Jay

On Fri, Jan 18, 2013 at 7:53 AM, Joe Stein <[EMAIL PROTECTED]> wrote:

> Can I help out?>> Also can we abstract the config call too? We have so much in chef, it's> not that i don't want to call our zookeeper cluster for it but we don't> have our topology yet mapped out in znodes they are in our own instances of> code.>> It should have both a pull and push for changes, one thing that's nice> with zookeeper and having a watcher.>> /*> Joe Stein, Chief Architect> http://www.medialets.com> Twitter: @allthingshadoop> Mobile: 917-597-9771> */>> On Jan 18, 2013, at 12:09 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:>> > Currently kafka broker config is all statically defined in a properties> > file with the broker. This mostly works pretty well, but for per-topic> > configuration (the flush policy, partition count, etc) it is pretty> painful> > to have to bounce the broker every time you make a config change.> >> > That lead to this proposal:> > https://cwiki.apache.org/confluence/display/KAFKA/Dynamic+Topic+Config> >> > An open question is how topic-default configurations should work.> >> > Currently each of our topic-level configs is paired with a default. So> you> > would have something like> > segment.size.bytes> > which would be the default, and then you can override this for topics> that> > need something different using a map:> > segment.size.bytes.per.topic> >> > The proposal is to move the topic configuration into zookeeper so that> for> > a topic "my-topic" we would have a znode> > /brokers/topics/my-topic/config> > and the contents of this znode would be the topic configuration either as> > json or properties or whatever.> >> > There are two ways this config could work:> > 1. Defaults resolved at topic creation time: At the time a topic is> created> > the user would specify some properties they wanted for that topic, any> > topic they didn't specify would take the server default. ALL these> > properties would be stored in the znode.> > 2. Defaults resolved at config read time: When a topic is created the> user> > specifies particularly properties they want and ONLY the properties they> > particularly specify would be stored. At runtime we would merge these> > properties with whatever the server defaults currently are.> >> > This is a somewhat nuanced point, but perhaps important.> >> > The advantage of the first proposal is that it is simple. If you want to> > know the configuration for a particular topic you go to zookeeper and> look> > at that topics config. Mixing the combination of server config and> > zookeeper config dynamically makes it a little harder to figure out what> > the current state of anything is.> >> > The disadvantage of the first proposal (and the advantage of the second> > proposal) is that making global changes is easier. For example if you> want> > to globally lower the retention for all topics, in proposal one you would> > have to iterate over all topics and update the config (this could be done> > automatically with tooling, but under the covers the tool would do this).> > In the second case you would just update the default value.> >> > Thoughts? If no one cares, I will just pick whatever seems best.

how about a command line script (bin/kafka-config-init.sh) to load in afile for the configs to initialize the config values in zookeerper butkafka reads the configs from zookeeper

another script can also have options for doing updates(bin/kafka-config-update.sh)

if we provide a writing mechanism then the config management systems (weuse chef) can interact nice the zookeeper updates in a standard way that wedocument and support

win? win?

On Fri, Jan 18, 2013 at 4:19 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Yes please, any help very much appreciated.>> I am not sure if I understand what you are proposing, though. Are you> saying support both the config file and zk for topic-level configs? I hate> to do things where the answer is "do both"...I guess I feel that although> everyone walks away happy it ends up being a lot of code and combinatorial> testing. So if there is a different plan that hits all requirements I like> that better. I am very sensitive to the fact that zookeeper is an okay> key/value store but a really poor replacement for a config management> system. It might be worth while to try to work out a way that meets all> needs, if such a thing exists.>> Is bouncing brokers for topic-overrides a problem for you in your> environment? If so how would you fix it?>> -Jay>> On Fri, Jan 18, 2013 at 7:53 AM, Joe Stein <[EMAIL PROTECTED]>> wrote:>> > Can I help out?> >> > Also can we abstract the config call too? We have so much in chef, it's> > not that i don't want to call our zookeeper cluster for it but we don't> > have our topology yet mapped out in znodes they are in our own instances> of> > code.> >> > It should have both a pull and push for changes, one thing that's nice> > with zookeeper and having a watcher.> >> > /*> > Joe Stein, Chief Architect> > http://www.medialets.com> > Twitter: @allthingshadoop> > Mobile: 917-597-9771> > */> >> > On Jan 18, 2013, at 12:09 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:> >> > > Currently kafka broker config is all statically defined in a properties> > > file with the broker. This mostly works pretty well, but for per-topic> > > configuration (the flush policy, partition count, etc) it is pretty> > painful> > > to have to bounce the broker every time you make a config change.> > >> > > That lead to this proposal:> > > https://cwiki.apache.org/confluence/display/KAFKA/Dynamic+Topic+Config> > >> > > An open question is how topic-default configurations should work.> > >> > > Currently each of our topic-level configs is paired with a default. So> > you> > > would have something like> > > segment.size.bytes> > > which would be the default, and then you can override this for topics> > that> > > need something different using a map:> > > segment.size.bytes.per.topic> > >> > > The proposal is to move the topic configuration into zookeeper so that> > for> > > a topic "my-topic" we would have a znode> > > /brokers/topics/my-topic/config> > > and the contents of this znode would be the topic configuration either> as> > > json or properties or whatever.> > >> > > There are two ways this config could work:> > > 1. Defaults resolved at topic creation time: At the time a topic is> > created> > > the user would specify some properties they wanted for that topic, any> > > topic they didn't specify would take the server default. ALL these> > > properties would be stored in the znode.> > > 2. Defaults resolved at config read time: When a topic is created the> > user> > > specifies particularly properties they want and ONLY the properties> they> > > particularly specify would be stored. At runtime we would merge these> > > properties with whatever the server defaults currently are.> > >> > > This is a somewhat nuanced point, but perhaps important.> > >> > > The advantage of the first proposal is that it is simple. If you want> to> > > know the configuration for a particular topic you go to zookeeper and/*Joe Steinhttp://www.linkedin.com/in/charmallocTwitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>*/

> how about a command line script (bin/kafka-config-init.sh) to load in a> file for the configs to initialize the config values in zookeerper but> kafka reads the configs from zookeeper>> another script can also have options for doing updates> (bin/kafka-config-update.sh)>> if we provide a writing mechanism then the config management systems (we> use chef) can interact nice the zookeeper updates in a standard way that we> document and support>> win? win?>> On Fri, Jan 18, 2013 at 4:19 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:>> > Yes please, any help very much appreciated.> >> > I am not sure if I understand what you are proposing, though. Are you> > saying support both the config file and zk for topic-level configs? I> hate> > to do things where the answer is "do both"...I guess I feel that although> > everyone walks away happy it ends up being a lot of code and> combinatorial> > testing. So if there is a different plan that hits all requirements I> like> > that better. I am very sensitive to the fact that zookeeper is an okay> > key/value store but a really poor replacement for a config management> > system. It might be worth while to try to work out a way that meets all> > needs, if such a thing exists.> >> > Is bouncing brokers for topic-overrides a problem for you in your> > environment? If so how would you fix it?> >> > -Jay> >> > On Fri, Jan 18, 2013 at 7:53 AM, Joe Stein <[EMAIL PROTECTED]>> > wrote:> >> > > Can I help out?> > >> > > Also can we abstract the config call too? We have so much in chef,> it's> > > not that i don't want to call our zookeeper cluster for it but we don't> > > have our topology yet mapped out in znodes they are in our own> instances> > of> > > code.> > >> > > It should have both a pull and push for changes, one thing that's nice> > > with zookeeper and having a watcher.> > >> > > /*> > > Joe Stein, Chief Architect> > > http://www.medialets.com> > > Twitter: @allthingshadoop> > > Mobile: 917-597-9771> > > */> > >> > > On Jan 18, 2013, at 12:09 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:> > >> > > > Currently kafka broker config is all statically defined in a> properties> > > > file with the broker. This mostly works pretty well, but for> per-topic> > > > configuration (the flush policy, partition count, etc) it is pretty> > > painful> > > > to have to bounce the broker every time you make a config change.> > > >> > > > That lead to this proposal:> > > >> https://cwiki.apache.org/confluence/display/KAFKA/Dynamic+Topic+Config> > > >> > > > An open question is how topic-default configurations should work.> > > >> > > > Currently each of our topic-level configs is paired with a default.> So> > > you> > > > would have something like> > > > segment.size.bytes> > > > which would be the default, and then you can override this for topics> > > that> > > > need something different using a map:> > > > segment.size.bytes.per.topic> > > >> > > > The proposal is to move the topic configuration into zookeeper so> that> > > for> > > > a topic "my-topic" we would have a znode> > > > /brokers/topics/my-topic/config> > > > and the contents of this znode would be the topic configuration> either> > as> > > > json or properties or whatever.> > > >> > > > There are two ways this config could work:> > > > 1. Defaults resolved at topic creation time: At the time a topic is> > > created> > > > the user would specify some properties they wanted for that topic,> any> > > > topic they didn't specify would take the server default. ALL these> > > > properties would be stored in the znode.> > > > 2. Defaults resolved at config read time: When a topic is created the> > > user> > > > specifies particularly properties they want and ONLY the properties

+

Jay Kreps 2013-01-18, 23:29

NEW: Monitor These Apps!

All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by Sematext