Hello,
Since I'm continuing to write software in D (no, nothing special...just
pet projects) I felt that I needed to create a library to handle logging
to a file. I have a good start and it seems to work pretty well. I
even have a couple ideas left to implement to make it better, however
I'm having trouble with one. One thing that I did implement, which I
think is pretty cool, is a set up the class to start a thread as a
daemon to run in the background and check for updates to the config
file, much like log4j. This way the user can change the log level at
runtime.
One thing I'm thinking that I want to do is have the writing to the
logfile handled by a separate class. This way I'm hoping to ensure that
there's not a lot of unnecessary synchronizing going on and the amount
of work writing to the logfile won't slow down the rest of the program.
My question is, what is the best approach?
1) I use the current core.thread library and put all my messages in a
buffer that the thread checks periodically, pulls off a new message, and
then writes it to a file. There will be some work to make sure nothing
collides with each other, but I think I can manage it.
2) I wait for the new threading library with message passing to come out
and just rely on the message passing to handle everything. It's a much
cleaner approach based on my experience with Erlang, but there are two
issues and the major one is I have no idea when it'll be ready.
Granted, I don't need this capability now, but I'm interested in getting
it to work.
3) Something else. I really don't have much experience with threading,
so I'm being very careful and really want to understand it. This
library looks to be a good way to learn, however if it's not the best
way to do things, then what would be?
Thanks in advance for any input.
Casey

1) I use the current core.thread library and put all my messages in a
buffer that the thread checks periodically, pulls off a new message, and
then writes it to a file. There will be some work to make sure nothing
collides with each other, but I think I can manage it.

Wouldn't this allow logging messages to be lost before a hard crash? To
me, that would greatly reduce the utility of the logging system.

3) Something else. I really don't have much experience with threading,
so I'm being very careful and really want to understand it. This
library looks to be a good way to learn, however if it's not the best
way to do things, then what would be?

Global mutex associated with the logging system. Lock, output, unlock.
There are scalability issues with that approach, but I don't see why it
wouldn't work in this situation. (Plus, if you have a message queue,
you probably need to protect that queue with a mutex anyway.)
--
Rainer Deyke - rainerd eldwood.com

1) I use the current core.thread library and put all my messages in a
buffer that the thread checks periodically, pulls off a new message, and
then writes it to a file. There will be some work to make sure nothing
collides with each other, but I think I can manage it.

Wouldn't this allow logging messages to be lost before a hard crash? To
me, that would greatly reduce the utility of the logging system.

If done improperly, yes, it would. My hope is that if I did go with
this method, I would try to find a way to ensure no data was lost.
Oddly enough, I'm currently having that problem with the single-threaded
version for some reason. The test program will stop and some of the
logging statements never make it to the file.

3) Something else. I really don't have much experience with threading,
so I'm being very careful and really want to understand it. This
library looks to be a good way to learn, however if it's not the best
way to do things, then what would be?

Global mutex associated with the logging system. Lock, output, unlock.
There are scalability issues with that approach, but I don't see why it
wouldn't work in this situation. (Plus, if you have a message queue,
you probably need to protect that queue with a mutex anyway.)

Understood. My goal is that if I do put the writing in another thread,
I do my best to ensure it will scale. I have a tendency to put a lot of
logging statements in code when I'm trying to debug something, so I
don't want to slow things down too much nor do I want to lose anything.
In short: I want the log writing to be as out of the way as possible.
Casey

1) I use the current core.thread library and put all my messages in a
buffer that the thread checks periodically, pulls off a new message, and
then writes it to a file. There will be some work to make sure nothing
collides with each other, but I think I can manage it.

Wouldn't this allow logging messages to be lost before a hard crash? To
me, that would greatly reduce the utility of the logging system.

If done improperly, yes, it would. My hope is that if I did go with
this method, I would try to find a way to ensure no data was lost.
Oddly enough, I'm currently having that problem with the single-threaded
version for some reason. The test program will stop and some of the
logging statements never make it to the file.

3) Something else. I really don't have much experience with threading,
so I'm being very careful and really want to understand it. This
library looks to be a good way to learn, however if it's not the best
way to do things, then what would be?

Global mutex associated with the logging system. Lock, output, unlock.
There are scalability issues with that approach, but I don't see why it
wouldn't work in this situation. (Plus, if you have a message queue,
you probably need to protect that queue with a mutex anyway.)

Understood. My goal is that if I do put the writing in another thread,
I do my best to ensure it will scale. I have a tendency to put a lot of
logging statements in code when I'm trying to debug something, so I
don't want to slow things down too much nor do I want to lose anything.
In short: I want the log writing to be as out of the way as possible.
Casey

I second Rainer. A logging system should commit (at least) error messages
immediately, particularly if the application has multiple threads. Otherwise it
is going to make debugging a crashing system a nightmare. When I do it I just
stick 'synchronized' in front of the statement that does the write.

I second Rainer. A logging system should commit (at least) error messages
immediately, particularly if the application has multiple threads. Otherwise it
is going to make debugging a crashing system a nightmare. When I do it I just
stick 'synchronized' in front of the statement that does the write.

Yes, I fully understand that and in the current single-threaded version
I have, that's exactly what happens: all error and fatal messages cause
the buffer to flush.
What I'm looking for is the best way to handle having a daemon writer.
My biggest concern here is multi-threaded applications. Granted, it
would be nice to not have the writing in the same thread as the rest of
the code to try to keep file I/O from affecting performance, but that's
secondary.
Here's what I know: a variable of type OutputStream cannot be shared. I
did not try using __gshared, but regardless while this would work and I
could easily synchronize the writes, I can see a lot of contention if
multiple threads trying to write a lot of data to a log file. (E.g.
trace statements that capture variable states to enhance debugging)
Granted, this shouldn't be the case in production code, but if I can
find a better way to do this, I'd love to. This is why I thought that
the new threading model with message passing would be good, but I again
have concerns if there are a lot of messages trying to be written. I
know in Erlang you can fill up the message buffer if you're not careful.
Casey