One reason for doing one-line logging for one action is that it would help some log debugging/trouble shooting using grep, for example. Admittedly sometimes this makes harder for human to read, but would be better for scripts to process them, and often we need one action to be shown in one line so we capture the correlations with simple grep other than more fancy parsing. What do you think?

Guozhang

> Kafka can log giant log lines> ----------------------------->> Key: KAFKA-1122> URL: https://issues.apache.org/jira/browse/KAFKA-1122> Project: Kafka> Issue Type: Bug> Affects Versions: 0.8> Reporter: Jason Rosenberg> Priority: Minor>> There are a number of log lines that the kafka server, and high-level consumer, can log, that can end up becoming a giant log line. This can be cumbersome to deal with in a log file.> This happens in my case as I have have a large number of topics (on the order of 500-700 topics). Typically, these giant log lines will say something separately about every topic on the broker. An example:> 2013-11-04 23:28:11,148 INFO [kafka-request-handler-0] server.ReplicaManager - [Replica Manager on Broker 10]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0) -> (LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2),.....> Imagine that line going on with a separate entry for 700 topics. There are many other examples of this phenomenon in the server, and high-level consumer.> I'd think these log lines could be separated into a single line per topic.

You are both right, is there some type of patch we could introduce to toggle for both use cases? Perhaps in Logging do some parsing that would output multi line when some default configuration is overridden?

Or better yet we have a tool or known command using awk in it to to use when tailing the log?

> Kafka can log giant log lines> ----------------------------->> Key: KAFKA-1122> URL: https://issues.apache.org/jira/browse/KAFKA-1122> Project: Kafka> Issue Type: Bug> Affects Versions: 0.8> Reporter: Jason Rosenberg> Priority: Minor>> There are a number of log lines that the kafka server, and high-level consumer, can log, that can end up becoming a giant log line. This can be cumbersome to deal with in a log file.> This happens in my case as I have have a large number of topics (on the order of 500-700 topics). Typically, these giant log lines will say something separately about every topic on the broker. An example:> 2013-11-04 23:28:11,148 INFO [kafka-request-handler-0] server.ReplicaManager - [Replica Manager on Broker 10]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0) -> (LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2),.....> Imagine that line going on with a separate entry for 700 topics. There are many other examples of this phenomenon in the server, and high-level consumer.> I'd think these log lines could be separated into a single line per topic.

Agree with Joe that a tool/command would help much here. [~jbrosenberg] What kind of information (keywords, patterns, etc) would you usually be looking for from the server logs and how do you usually do that? Just wanted to have a better feeling of the needs so we can see what would be the best approach here.

> Kafka can log giant log lines> ----------------------------->> Key: KAFKA-1122> URL: https://issues.apache.org/jira/browse/KAFKA-1122> Project: Kafka> Issue Type: Bug> Affects Versions: 0.8> Reporter: Jason Rosenberg> Priority: Minor>> There are a number of log lines that the kafka server, and high-level consumer, can log, that can end up becoming a giant log line. This can be cumbersome to deal with in a log file.> This happens in my case as I have have a large number of topics (on the order of 500-700 topics). Typically, these giant log lines will say something separately about every topic on the broker. An example:> 2013-11-04 23:28:11,148 INFO [kafka-request-handler-0] server.ReplicaManager - [Replica Manager on Broker 10]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0) -> (LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2),.....> Imagine that line going on with a separate entry for 700 topics. There are many other examples of this phenomenon in the server, and high-level consumer.> I'd think these log lines could be separated into a single line per topic.

Well, I just think log files should be something you can check on, to see things look normal, and then you can easily to searches over (e.g. in splunk, or something similar). So, having single lines that are 50K long, is simply not useful. You mention grep, how would grepping for a string in a giant log line be helpful?

If say I want to know everything that happened for a particular topic or partition, it's not helpful if it appears in a logline with also, 700 other topics.

I think breaking things up, so they are coherent for doing useful searches, and also, so that you can visually inspect a log file, and see that there are no unexpected exceptions, is useful. You can't do that easily when some lines are too long.

If I'm restarting the server, I will generally tail the log file for a while, to make sure it looks good.

> Kafka can log giant log lines> ----------------------------->> Key: KAFKA-1122> URL: https://issues.apache.org/jira/browse/KAFKA-1122> Project: Kafka> Issue Type: Bug> Affects Versions: 0.8> Reporter: Jason Rosenberg> Priority: Minor>> There are a number of log lines that the kafka server, and high-level consumer, can log, that can end up becoming a giant log line. This can be cumbersome to deal with in a log file.> This happens in my case as I have have a large number of topics (on the order of 500-700 topics). Typically, these giant log lines will say something separately about every topic on the broker. An example:> 2013-11-04 23:28:11,148 INFO [kafka-request-handler-0] server.ReplicaManager - [Replica Manager on Broker 10]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0) -> (LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2),.....> Imagine that line going on with a separate entry for 700 topics. There are many other examples of this phenomenon in the server, and high-level consumer.> I'd think these log lines could be separated into a single line per topic.

--This message was sent by Atlassian JIRA(v6.1#6144)

NEW: Monitor These Apps!

All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by Sematext