SAS Enterprise Guide: Check the Log Macro

I created a LogCheck macro [code provided] to review the log during a batch process. The SAS log files can be rather large and it is so much easier if you can just extract the issues in an easy to understand format. This macro reads in each line of the log file and keeps the ones with Error, Warning, Note, or specific text you want to trap. Then it sends a colorful email to report what issues (if any) were found.

You can download the code and two sample programs to review or change for your situation. Let me know what you think in the Comments section. How can the code be improved? Any ideas you have?

Using the LogCheck Macro

Add the LogCheck macro to your batch job to report issues through email about the job. When the job has no errors, you will receive an email similar to the following. The subject line indicates no found issues – thus you have some assurance the job was successful.

When the job has errors – the macros sends an email similar to the following. Notice the subject line contains the number of Errors, Warnings, and Notes present in the log. The email lists each issue, where found in the log, and an excerpt from the log. This makes it easier to determine if an error early on in the job actually caused the other errors and you can quickly fix the root cause.

Adding the Macro to Your Code

CheckLog was designed to run with a batch job. To use this program, do the following:

1. Add the following code to the top of the SAS program you want to monitor. Update the LogName macro variable to the location and name of your log file. Refer to the PROC PRINTTO documentation if you need more guidance.

3. In the LogCheck_macro.sas – update the EMAIL macro variable to send to the email address of your choice. After making this change – run the two test programs (included in the download) to see how it works.

Change it Up for Your Environment

I have the code check for Notes that actually can produce errors but do not stop production. This can be annoying if it is something you do not care about – so you may want to prune this area of the code.

If there are some items specific to your environment that you want to trap – just add it here:

Some issues present as errors/warnings but may not be in reality. For instance, Teradata generates a Warning that it cleared the table – which is what you want to happen. It’s not an issue. So use this area of the code to change the type. A TYPE=4 is a NOTE and will show as gray in the email. Otherwise it will be a Type=2 that is bright yellow. Since a Type=4 generates an email that indicates an issue- you may want to delete this line.

If you want to email the error-ridden log to yourself – uncomment the Attach= line.

The two Program files demonstrate how the Checklog_macro.sas program works. So modify the checklog_macro.sas program to add your email address and then run the program. If you do not have email setup at your site – obviously this won’t be a great option for you. [Reference: SAS Global Forum Paper about Emailing from SAS]

Are there other Log Check Programs?

There are many other (probably superior) log check programs available from the SAS Global Forum papers and in other locations. I offer this code to the SAS community to use as you will.

Never miss a BI Notes post!

Click here for free subscription.
Once you subscribe you'll be asked to confirm your subscription through your email account.
You email address is kept private and you can unsubscribe anytime.

Tricia Aanderud is a SAS Business Intelligence and Visual Analytics consultant based in Raleigh, NC who works for Zencos Consulting. She has written several books about SAS, presented papers at many SAS conferences, and has been using SAS since 2001. Contact her for assistance with your next project.

20 Comments »

Hi Tricia et al,
A month later and I’m still thinking about log checking. But this time in the context of stored procedures.

It seems to me as SAS programmers, we know how important it is to scan the log before trusting any results.

But I feel like log scanning is frequently ignored in the context of developing stored processes that will be run by others. Arguably, log scanning is *more* important when developing web applications that will be used by non-programmers.

So as SAS developers, how do we best incorporate log scanning into our stored procedures / applications? I was thinking about playing around with some something like the following, for a stored procedure that makes a pdf report:

1. PROC PRINTTO to send log to a file on the server.
2. Usual reporting code to create pdf. Do not display the pdf to the user, but write it to a temp location .
3. PROC PRINTTO to close log file.
4. %LogCheck the log file.
5. If log is clean, open the pdf for the user to see/download.
6. If log is not clean, pop up a message (javascript window? pdf report?) with a report of the errors found in log. (even if not interpretable by user, it’s enough that they will know something bad happened).

Does that seem like reasonable approach? Have you seen papers/sites that talk about log checking in a server environment.

I like this method Chris – especially during batch. You’re right – there are few key times – if the Teradata/Oracle extract failed – the rest of the job is wrong.
I would prefer that execution stop and alert me over sending me an email that is 3 pages long.

I’ve been thinking about this a lot, and it’s kind of a trend with SAS users: We seem to think that the log has all of the indication we would need about whether something went wrong. However, it’s not really the case – there are other indicators.

A good one is the SYSERR macro variable, which could be checked after every step to see if anything failed. A macro that did this (e.g., see below) could halt the program and email you on the fly instead of running the whole program. In our log check reports, how many of those issues need to be fixed? Probably just the first 1 or 2. The rest were just reverberations from the initial issue(s) and didn’t even need to be mentioned or run.

The above could be called after every step or just after certain risky steps (e.g., you’re waiting on someone else’s data set once a month). Of course, this could fail, too, if SAS has a critical failure and crashes. Then you wouldn’t hear anything about it. Both our macros can suffer this fate, except if you schedule it as a separate task. Then your server could crash, but then you’d have other problems to worry about.

Anyway, just some thoughts about how we go about things. Perhaps a combination of both methods is best: check during AND after, and you can have very robust programs.

So in short – just know what is okay in your environment. In the example – I used my gmail account for simplicity. When used for work the emails were going to my work address so it was acceptable. Again – just know what is okay for your environment.

Another idea – you could have the program send you a text message that simply said Errors/Warnings existed.

There are some odd error messages that come out of Oracle like that, too. I haven’t seen any other phrases, like warning, that have an odd format like that, so changing “error:” to “error” would probably be sufficient.

Oh, just to be clear: Email is just plain text, and can be easily discovered by a black hat. You do not want to have an inadvertent disclosure of protected health information through email. The sector of education also has similar privacy laws. It’s best to be over-cautious in this area and simple avoid this kind of thing.

Well, you know about mine. My macro assumes you’ve already done something, and you forgot to check along the way. So there’s no need to set up your code with PROC PRINTTO. Now of course in EG, that’s the only way to do it, as far as I can figure out. It also emails, but I like the style of your email by bringing more attention to what was identified. I have an update coming out soon that will work a bit better in EG. I’ve been reviewing it to get it ready for the SAS Global Forum later this month.

One very serious note of caution: Sometimes these messages can contain information in the data set, which could include patient information like name, date of birth, address, and other protected information. This would be a violation (if not literally than in spirit) of HIPAA. I would remove the specific messages from your emails to avoid this very serious issue. That’s why in my macro only a summary is provided.

Wow … Quentin … a ton of great suggestions! Where have you been hiding?

-I had not seen that SUGI paper before so I put it on my list. They make a great point about excluding Notes – that is a better way to think about it.
-Hmm … Error without the colon … every tricky!
-Oh love the idea about resetting the counter. So would you place that code before the %logcheck? Can it be part of the %logcheck?

A question- Do you think there is a way to do this in EG without using PROC PRINTTO at the beginning to direct the log to a file? Is there a default location for the log file in EG?

In “regular” SAS, when you batch submit a program, the log file is written to the same directory as the program, with the same name, so it is possible to scan that file (actuallly you need to copy the file to a temp directory and scan the copied version). In interactive, you can use DM commands to copy the current log to a separate file, and scan it. I’m just about to start using EG, so don’t know where the log file is written by default. Just a thought.

They argue that it is safer to specify a list of notes to exclude, rather than than specify the notes to include. You end up with a longer list, but it’s a neat approach.

A couple other small points.

Searching for “ERROR:” will not catch a few error messages where SAS puts an error number before the colon, e.g.:

ERROR 48-59: The format MNDDYY was not found or could not be loaded.

I think we need to consider a situation in which the main code encounters an error and SAS enters syntax check mode (set obs to 0). Might be helpful to check for this and reset obs to MAX, to make sure the steps in %logcheck() run, e.g.:

2 Pingbacks »

[…] log file. If you don’t have one, search conference proceedings at lexjansen.com, or just use Tricia’s. This macro should read in the log file, and return results that indicate whether the log had any […]

[…] Guide: Identifying Errors in the Log April 4, 2012By Tricia AanderudIn an earlier post, I provided a check log program that you could use with your batch jobs. There was some really interesting discussions in the Comments about using SAS Enterprise Guide […]