Android logger issues

This section describes some attributes of the Android logger code, which are
relevant for mainlining the code into Linux. Let's use a modified SWOT (Strengths,
Weaknesses, Opportunities, Threats) analysis for strategic planning to mainline
this code.

To find the logger's strengths and weaknesses, lets research and provide information
and hard numbers for how it compares with existing logging alternatives.

Alternatives

What are the alternatives to the Android logger?

logbuf (the kernel log buffer)

apparently, systemd uses

syslog

"the journal" - a new logging system for systemd (by Lennart Poettering)

One change, to add 'system' channel in Android, in last 3 years. System channel separates system log messages from application log messages to prevent buffer exhaustion in case of a chatty application.

Extension questions

Here are some questions about how the systems could be integrated:

What features could each log system have, if extended?

Would extension to an existing system interfere with it's current primary purpose?

For each change contemplated, can it be implemented to minimize the impact to existing code?

Barriers to entry as is (Weaknesses)

Does logger duplicate functionality that exists elsewhere?
Does it make sense to extend an existing system, rather than implement a new system?

What are barriers to entry:

use of ioctl?

hardcoded number of log channels

other style issues?

longterm maintainer?

Attractive features of logger as is (Strengths)

What are the differentiating features of logger, compared to alternatives?

minimal context switches for logging

use of ring buffer automatically limits log size

log is in memory (no cost to store unused log messages)

all messages are tagged

all messages have priority

all messages have timestamp

user-space program to filter messages by tag

channel selection by user-space policy

Discussions

This section has notes about discussions with the community kernel developers and with Google engineers,
with regards to mainlining this code

Google engineer requests/questions

Google doesn't want to change their class libraries or debug system

These have already shipped to developers and are integrated into other tools (eclipse log viewer)

the Android system and over 200,000 applications already use the existing classes

(So, can changes be made "under the hood" without changing the existing user interface?)

Google requests that any changes submitted to mainline also be placed in their kernel repository (or Gerrit review queue), so they can see them when they do merges and are not caught off guard.

I'm not sure the detailed steps required to perform this

Question: Why did Google write their own code, instead of using syslog?

Was it simply expedience, lack of familiarity with syslog, or are there specific reasons they wrote a new system (missing features, etc.)?

Logbuf and logger don't require writeable file system access which is mandatory for syslog. A no-writeable files-system (or a file-system which allow only few writes) is a common situation in embedded world.

adding warts to logbuf doesn't seem like a great idea, it's already got realtime issues that were never solved

but integration with user and kernel messages could be nice

integration also requires some oddities

do you convert the log message fields (pid and priority) to structured text, and parse them out again later

note that systemd added support for priorities and/or tags to logbuf (Kay Seivers) see

the ioctl for logger looks a lot like syslog(2), any way to consolidate?

you'd have to add channels to syslog(2), if you wanted to have separate channels

[2] hardcoded names and number of logs in user-space and the kernel (David Brown, Neil Brown, Andrew Morton)

no argument there (Brian Swetland)

[3] logbuf filesystem idea (Neil Brown)

If you created a 'logbuf' filesystem that used libfs to provide a single
directory in which privileged processes could create files then you wouldn't
need the kernel to "know" the allowed logs: radio, events, main, system.
The size could be set by ftruncate() (by privileged used again) rather than
being hardcoded.
You would define 'read' and 'write' much like you currently do to create a list of
datagrams in a circular buffer and replace the ioctls by more standard
interfaces:
LOGGER_GET_LOG_BUG_SIZE would use 'stat' and the st_blocks field
LOGGER_GET_LOG_LEN would use 'stat' and the st_size field
LOGGER_GET_NEXT_ENTRY_LEN could use the FIONREAD ioctl
LOGGER_FLUSH_LOG could use ftruncate

this fixes (2) by allowing arbitrary logs created by user space

unfortunately, the log channels don't exist until after user space is up a while

maybe could hardcode a mechanism for a default system log at startup, with other to follow

Greg KH also asks why we need a new interface for this rather than logbuf/syslog(2)

[4] features, requirements and wishlist (Brian Swetland)

avoid hardcoding the names and sizes of the logs

adjust permissions so that some apps can only read back their own log messages (security by pid or by channel?)

allow logs to survive reboot

allow an app to create it's own log, up to a specified size limit, but in a way that wouldn't

consume too much memory systemwide

The goals behind the logger driver have been:
- keep userland and kernel logging separate (so that spammy userland
logging doesn't make us lose critical kernel logs or the other way
round)
- make log writing very inexpensive -- avoid having to pass messages
between processes (more critical on ARM9 platforms where this implied
extra cache flushing), avoid having to make several syscalls to write
a log message (getting time of day, etc), and so on
- make log writing reliable -- don't trust userland to report its
timestamp, PID, or to correctly format the datagrams, etc
- allow a log watching process (logcat) to easily pull data from all
logs at once
- avoid committing a vast amount of memory to logging
- try to prevent clients from spamming each other out of log space
(only successful on a coarse granularity right now with the
main/system/radio/events logs)
- ensure logs are not lost at the moment an app crashes

[6] separate log channels are not needed or desirable (Kay Sievers)

for early boot and debug, you want interleaving

[7] write cost is not a problem (Kay Sievers, )

I have no numbers, but suspect there is no problem

[8] userspace printk tty driver (Greg KH, Kay Sievers mentioned it)

I never heard of it - need to do some research I guess

[9] Google doesn't see a need to invest in switching (Brian Swetland)

Having a bunch of people suggest how we can just rewrite our userspace
to get something similar but not really the same as the functionality
we already have registers as "big waste of our time."
Replacing a tiny, self-contained, and reliable driver that's worked
for us for five-ish years with a pile of userspace surgery is
uninteresting to me.