Wanted to check if there is any known limit on the # of topics in a Kafkacluster? I wanted to design a system which has say 5k topics andmulti-threaded consumers reading messages from these topics. Does anyonehave experience with such large topic size? I see in Kafka's page a testfor throughput w.r.t. to topic size, but the max topic size is 50.

> Hello,>> Wanted to check if there is any known limit on the # of topics in a Kafka> cluster? I wanted to design a system which has say 5k topics and> multi-threaded consumers reading messages from these topics. Does anyone> have experience with such large topic size? I see in Kafka's page a test> for throughput w.r.t. to topic size, but the max topic size is 50.>> Thanks>

Just curious - I don't see any limit in the # of topics that you canhave in a Kafka cluster.

So in principle, you could have as many topics as you want, so long asyour hardware can keep up, right?

Jason

On Wed, Mar 27, 2013 at 11:33 AM, Jun Rao <[EMAIL PROTECTED]> wrote:> At LinkedIn, our largest cluster has more than 2K topics. 5K topics should> be fine.>> Thanks,>> Jun>> On Tue, Mar 26, 2013 at 11:52 PM, Suyog Rao <[EMAIL PROTECTED]> wrote:>>> Hello,>>>> Wanted to check if there is any known limit on the # of topics in a Kafka>> cluster? I wanted to design a system which has say 5k topics and>> multi-threaded consumers reading messages from these topics. Does anyone>> have experience with such large topic size? I see in Kafka's page a test>> for throughput w.r.t. to topic size, but the max topic size is 50.>>>> Thanks>>

> Just curious - I don't see any limit in the # of topics that you can> have in a Kafka cluster.>> So in principle, you could have as many topics as you want, so long as> your hardware can keep up, right?>> Jason>> On Wed, Mar 27, 2013 at 11:33 AM, Jun Rao <[EMAIL PROTECTED]> wrote:> > At LinkedIn, our largest cluster has more than 2K topics. 5K topics> should> > be fine.> >> > Thanks,> >> > Jun> >> > On Tue, Mar 26, 2013 at 11:52 PM, Suyog Rao <[EMAIL PROTECTED]> wrote:> >> >> Hello,> >>> >> Wanted to check if there is any known limit on the # of topics in a> Kafka> >> cluster? I wanted to design a system which has say 5k topics and> >> multi-threaded consumers reading messages from these topics. Does anyone> >> have experience with such large topic size? I see in Kafka's page a test> >> for throughput w.r.t. to topic size, but the max topic size is 50.> >>> >> Thanks> >>>

We've seen big performace degradation when we tested 1024 topics, so we'veopted to go for a much smaller topic count (< 100).

On the read side, I think performance is largely driven by the ability ofthe operating system to effectively cache access to #partitions*topicfiles. Clearly if you divide your available memory by 1024 topics you'llhave less in the file cache per topic than you would with only 3 topics(for example).On Wed, Mar 27, 2013 at 11:14 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> With more topics, you may hit one of those limits: (1) # dirs allowed in a> FS; (2) open file handlers (we keep all log segments open in the broker);> (3) ZK nodes.>> Thanks,>> Jun>> On Wed, Mar 27, 2013 at 8:36 AM, Jason Huang <[EMAIL PROTECTED]>> wrote:>> > Just curious - I don't see any limit in the # of topics that you can> > have in a Kafka cluster.> >> > So in principle, you could have as many topics as you want, so long as> > your hardware can keep up, right?> >> > Jason> >> > On Wed, Mar 27, 2013 at 11:33 AM, Jun Rao <[EMAIL PROTECTED]> wrote:> > > At LinkedIn, our largest cluster has more than 2K topics. 5K topics> > should> > > be fine.> > >> > > Thanks,> > >> > > Jun> > >> > > On Tue, Mar 26, 2013 at 11:52 PM, Suyog Rao <[EMAIL PROTECTED]>> wrote:> > >> > >> Hello,> > >>> > >> Wanted to check if there is any known limit on the # of topics in a> > Kafka> > >> cluster? I wanted to design a system which has say 5k topics and> > >> multi-threaded consumers reading messages from these topics. Does> anyone> > >> have experience with such large topic size? I see in Kafka's page a> test> > >> for throughput w.r.t. to topic size, but the max topic size is 50.> > >>> > >> Thanks> > >>> >>