The Linux Administration group is for the discussion of technical issues technical issues that arise during the administration of Linux systems, including maintaining the operating system and supporting end-user applications.

Logrotate and Data Loss

I'm having a very large log file which is written by Oracle Database and I want to use Linux logrotate to manage it.

As I know, logrotate moves the log file and touches a new one.
Something like:
mv ... ...
touch ...
with using cronjob with it.

What happens if Oracle wants to add a new line in the file at the time that is being moved? Does the kernel lock the file and keep the changes till the file is being created/cleared again and writes them to it? Or we might see an error from Oracle (or other process that is writing the logs)? Or we'll just lose the log data?
This log is very important to us and we don't wanna lose even one line of logs.

If we would lose data, using logrotate, are there any other ways to be able to rotate logs without losing any data and getting any errors?

ITPoOYaN via linuxadmin-l wrote:
>
>
> Thanks wanderedinn, What you said is very strange to me and I should
> beware of it in the future. One question: What you said only happens
> if I use ":>_file_name" and doesn't happen if i move the file?

It's been a while since I researched this, but I believe that if you
move a file, the inode remains, thus the process connected to that file
will continue to write to that file and not a new one.

From
> my searches I've found out that if I move the file, Oracle will
> create a new alert.log file if not being able to find it.

I don't know how Oracle works, but this would only be the case if it was
attempting to reopen the file. I guess it's possible that it would
check for the existence of the file before each write, but I doubt that
as that would be too costly.

So from
> what I said above and what Mr. Hajigholamali said, by moving the file
> to a new destination I'll be able to archive the alert.log files
> without losing any data. My strategy will be: - Create a script to
> move the alert.log file to an archive destination with a new name. -
> Use crone to run the script every week. Is it OK to do the strategy
> above? Or I'll lose data?

The bottom line is, you don't want to move or remove a file that is open
by a current process. Bad things will happen. The process needs to be
alerted to stop writing to the file and restart once you create the new
one. There may be a mechanism in Oracle that does this automatically, I
don't know.

>

Help the community by fixing grammatical or spelling errors, summarizing or clarifying the solution, and adding supporting information or resources. Always respect the original author.

Popular White Paper On This Topic

You may be a tad bit confused when it comes to logrotate. You don't
setup a crontab schedule. It manages the schedule based on 2 things:
what folder you place the logrotate in and what the size/age limit is
on the log file.

Now to answer your questions:
1. What happens if Oracle wants to add a new line in the file at the
time that is being moved?
This will not affect the permissions or lock the file. IHS or APache
has a tremendous amount of activity and I have never seen a locking
issue.
2. Does the kernel lock the file and keep the changes till the file is
being created/cleared again and writes them to it? Or we might see an
error from Oracle (or other process that is writing the logs)? Or
we'll just lose the log data?
I have heard of some people having issues. I have never experienced
them. However there are alternatives to logrotate, like auditd,
skulker, or your own custom bash script.
This log is very important to us and we don't wanna lose even one line of logs.

There is a pre command (prerotate) and post command (postrotate) option
available within the logrotate.conf files -
Check the Oracle command line options and see if there is an option to tell
oracle to suspend logging - put that in the pre command, and then turn
logging back on in the post command.

As I understand, based on your answer it's not important to stop the log writer process while rotating files.
So why do people stop/suspend some processes like Apache/Oracle Listener while rotating files? Are they doing an unnecessary job?

What happens if for example i make my own logrotate script like: 1. Move the source file and create a new source file:
mv _source_file_ _destination_file_
touch _sourche_file
or 2. copies the source file and empties it:
cp _source_file_ _destination_file_
:>_sourche_file

I can't understand, which one is better and correct?- Using Linux logrotate without using prerotate and post rotate - Using a script like the 1st one with cronjob - Using a script like the 2nd one with cronjob
OR - I should use logrotate with prerotate and post rotate to suspend the Oracle process which writes the alert log that I want to rotate (and I donno exactly which process writes the alert log file and how to suspend it). I don't want to even lose one line of alert log file while rotating it.

If you zero out the log file, oracle will have a pointer into the file
where it left off and continue to attempt to write at that position.
What usually happens is, you end up with a lot of trash in the file
where the old data used to be.

Thanks wanderedinn, What you said is very strange to me and I should beware of it in the future.
One question: What you said only happens if I use ":>_file_name" and doesn't happen if i move the file?
From my searches I've found out that if I move the file, Oracle will create a new alert.log file if not being able to find it. So from what I said above and what Mr. Hajigholamali said, by moving the file to a new destination I'll be able to archive the alert.log files without losing any data.
My strategy will be:
- Create a script to move the alert.log file to an archive destination with a new name.
- Use crone to run the script every week.
Is it OK to do the strategy above? Or I'll lose data?

ITPoOYaN via linuxadmin-l wrote:
>
>
> Thanks wanderedinn, What you said is very strange to me and I should
> beware of it in the future. One question: What you said only happens
> if I use ":>_file_name" and doesn't happen if i move the file?

It's been a while since I researched this, but I believe that if you
move a file, the inode remains, thus the process connected to that file
will continue to write to that file and not a new one.

From
> my searches I've found out that if I move the file, Oracle will
> create a new alert.log file if not being able to find it.

I don't know how Oracle works, but this would only be the case if it was
attempting to reopen the file. I guess it's possible that it would
check for the existence of the file before each write, but I doubt that
as that would be too costly.

So from
> what I said above and what Mr. Hajigholamali said, by moving the file
> to a new destination I'll be able to archive the alert.log files
> without losing any data. My strategy will be: - Create a script to
> move the alert.log file to an archive destination with a new name. -
> Use crone to run the script every week. Is it OK to do the strategy
> above? Or I'll lose data?

The bottom line is, you don't want to move or remove a file that is open
by a current process. Bad things will happen. The process needs to be
alerted to stop writing to the file and restart once you create the new
one. There may be a mechanism in Oracle that does this automatically, I
don't know.

cpmams via linuxadmin-l wrote:
>
>
> Good strategy there although I'd copy the file rather than moving it then
> when the copying is confirmed pipe nothing into the log file by running { >
> /path/file_name }.

Again, I suggest that doing this while a process has the file open is
not a good thing. The process needs to know that the file has been moved.

Q: As long as inode doesn't change while moving a file and Oracle opens that file in append mode and it creates new file if it doesn't find it, why should I beware of finding a way to pause the process which writes the log files if I move it instead of copying and clearing it?

That's all fine and good if your premise is correct. That is, 'Oracle
opens that file in append mode and it creates new file if it doesn't
find it'. I find it highly unlikely that Oracle is going to be looking
for the file every time it tries to write a line to the log file. What
you're suggesting is, Oracle will attempt to locate the file and if it's
not there, will open it and write to it. Highly unlikely.

In most cases a process opens a log file once and keeps a pointer to the
end of the file for writing.

We use the following commands to prune log files without closing
them or losing any information:

cp logfile /newfs/logfile.date # creates a copy of log file for archival
purposes
>logfile # clears the file without closing it will appear to be
continuously open to Oracle
gzip /newfs/logfile.date # compresses the archived logfile to conserve space

Thanks for your complete answer, from now on I think I should consider more about the Oracle part in the writing process and become sure about how it opens the file and if checks for it or not and if not I should find a way to pause the process.

I'll do a test in VM and if everything is OK I'll use it on the primary DB.

I did not suggest you move (mv) the file. I said to copy (cp)
the file which does not change anything and then clear the
open file with the > command which does not affect its open
status with Oracle. Oracle will not perceive anything and
continue to write to the file as if nothing had happened.

I think the link I sent over previously - where you tell oracle to stop
logging is your best bet.

I think you said the log files get fairly large - how large are we talking?
If you do a "cp" and then truncate the original file, you have to take into
account the time involved to do the copy.
A "mv" appears almost instantaneous because all that changes is the file
name - the inode remains the same as
others have pointed out. It may take it several seconds to complete the
"cp" command and like you are saying,
by then there could be several more lines written to the log file.

I have seen this same thing happen with apache and its log files - which
is why the logrotate for httpd looks like :
/var/log/httpd/*log {
missingok
notifempty
sharedscripts
postrotate
/sbin/service httpd reload > /dev/null 2>/dev/null || true
endscript
}

The postrotate command closes and reopens the log files so it creates new
files and new inodes for the logs.

The file has not been managed for more than 5 years while the main DB where being used by a group of not very pro admins in a very bad situation, so the alert.log file is in Gig range size.

George, the link that you sent above didn't say anything about pausing Oracle from writing the ALERT log, it only said how to tell Oracle to stop from writing to listener log file and it only suggested to move the alert.log file.

For a while I'm not able to schedule a downtime of servers, so at least the first rotation should be a real consideration because of the size of alert.log and the time it takes if I want to copy it.

Till now everybody on the net have suggested to just MOVE the file and Oracle will create it the next time it wants to write into it. I hope this strategy won't come up with any problem.

Pooyan,
Perhaps you didn't see my post.
If you have free space to copy the file (should you need to archive it)
then you can use the > command to clear the file and Oracle will not
even notice as the file remains open to Oracle all the time. We have
used this technique for many such as the db2diag.log file for DB2
and the LDAP logs without issue for 10 years now.
Hope this helps.
Frank

I found some info on google that showed that the alert.log was explicitly
open, written to, and closed with each event.

Based on that, I believe the move and touch option should work without data
loss. If this catches oracle "in process" of writing to the file, oracle
should follow the inode and the output should go to the new file name, ie
alert.log.1

Upon the next event that triggers the alert.log it should go to the new
file name and the output goes to the new file.

You will probably want to try running this on a development server vs. your
production system, but if I understand it correctly, you should be good.

Copyright 1998-2015 Ziff Davis, LLC (Toolbox.com). All rights reserved. All product names are trademarks of their respective companies. Toolbox.com is not
affiliated with or endorsed by any company listed at this site.