It seems like most IT professionals today are so busy keeping up with day-to-day tasks, not to mention myriad unexpected issues that arise; that project planning and time management often take a back seat., Besides, “planning” sounds like such an “open-ended” thing that is frequently deemed a waste of time and lost in the shuffle.

In fact, a little proactive planning could save a lot of time and stress in the future. And it does not have to take up a lot of time, e.g. like planning how to better deal with production issues. Continue reading →

Does this sound familiar? When your NonStop gets very busy, your EMS also
gets very busy. In fact, sometimes you may find that EMS processing consumes a
lot of your CPU resources just for processing the flood of error messages. Why
are there so many messages in EMS?

Does this remind you of your EMS?

Many users dump EVERYTHING into EMS. The original intention of EMS is to allow
all the different errors to be analyzed and filtered in one place. But when everything
goes into this one pipe, the result is an overloaded, clogged pipe. When you dump
too much stuff into EMS:

It becomes difficult to find the error messages

It consumes a lot of CPU resource for EMS to file the messages

Operation tends to start ignoring messages in EMS console because they are too overwhelming

There is a better way –LogWatch

Instead of clogging up EMS, use LogWatch to monitor the different log files and work in conjunction with EMS.

LogWatch can monitor different files including:

Guardian files

OSS logs

VHS logs

Pathway logs

Third party logs, etc.

Lighten up the EMS load

Here is quick way to reduce EMS load: instead of routing your application errors to EMS, write them to disk logs.

Use LogWatch to monitor these application log files for errors.

LogWatch is scalable – you can have different instances of LogWatch monitoring different things.

LogWatch is easy to set up – you can set one up in minutes, and it won’t interfere with other instances.

Have LogWatch route only the errors to EMS.

Perfect companion to Prognosis or MOMI
If you are using a performance monitoring tool like Prognosis or MOMI, you will find LogWatch will work with it very effectively.

Use LogWatch to monitor disk log files for errors.

Configure LogWatch to route a message to EMS with specific Message ID or text pattern.

Enable Prognosis or MOMI to pick up these specific messages from EMS to take corrective actions.

Take Away – “Prevention is better than cure”More than many other IT folks, NonStop users understand and appreciate the importance of availability, the cornerstone of the platform. But applications do encounter errors, which could lead to stoppage. When that happens, it is important to recover from the failure as quickly as possible. Any extended down time due to an unavailable application translates to loss of revenue and users’ confidence. With some advanced planning and a good implementation plan for log monitoring, problems can be detected early and remedied promptly.

Analyze your logs – Where are the logs? What is written to the application logs? Take a look at some of the old logs and see what is going on in the environment.

Plan ahead – What are some of the log messages that require specific actions? What actions? Who should be responsible for actions?

Execute the plan – Start implementing a plan to monitor the key log files, and automate the log monitoring process with a tool like LogWatch.

Feedback please

Do you find this tutorial blog helpful? Let us know what you think, and how we can make it even better. Don’t forget, you can subscribe to our blogs (top right-hand corner of the home page) to get automatic email notification when a new blog is available.

Phil Ly is the president and founder of TIC Software, a New York-based company specializing in software and services that integrate NonStop with the latest technologies, including Web Services, .NET and Java. Prior to founding TIC in 1983, Phil worked for Tandem Computer in technical support and software development.

In my previous article (Know your OSS Logs Part 1), I discussed the importance of monitoring iTP Web Server logs. If you are running Java servlet or JSP applications with iTP Web Server on NonStop, it becomes even more imperative that you monitor their logs. Why? Because they are your only conduit into what’s happening in the execution environment. Is the application running correctly? Was there an environment issue? Did the application abend? All these and other useful information are kept in the logs.

In this article, I want to share with you some basic information on Java, Servlet and JSP logs. Again, all this information is already available in the HP documentation, and so I am going to give you the “Cliff Notes ” version here:

What to monitor?

servlet.out

stdout. This is the default location for Servlet to write out APPLICATION messages. So, if there is any issue encountered by the applications, , e.g. Pathsend failures, or datafile security errors, etc., the messages are usually reported in this file.

servlet.err

stderr. This is the location for Servlet to report errors encountered by the servlet.

JSP rollover logs

Logs related to the Servlet/JSP processing, and may contain many, many entries, which include the servlet container’s activities and status. This is a particularly difficult log to sift through, as there can be so much information in there. These logs are configured to “rollover” (create a new one) based on certain criteria, such as date or size.

servlet.out sample entry

$PM:inquiry-class failed with a server exception.An error has occurred with the link to the server.; TS/MP error 904; File system error 201; serverclass name: $PM.inquiry-class

Note: The above entry shows that the application has failed on a Pathsend and logged this message

Note: The above entry shows that the application has encountered a run time environment error due to missing code.

Why monitor?

Monitoring these logs allows you to check the health of the Java Servlet or NS/JSP applications and to detect errors as soon as they occur.

Did your Servlet just abort?

Did your NS/JSP just encountered a Pathsend error?

What NS/JSP pages are being accessed?

How can you quickly find the ERROR in your logs, among all those INFO and WARNING entires?

Which line of code in your Java Servlet has a problem?

Where are the logs?

The locations of these log files are specified in the server configuration file, and they usually reside in <NSJSP_HOME>/logs where <NSJSP_HOME> is /usr/tandem/webserver/servlet_jsp/ or /usr/tandem/webserver/servlets/

Who should look at them?

The Operation team members are the people that monitor these log files as it allows them to check the health of the system. However,the log entries are very important to Developers during development or QA phases as well, as the logs will help them quickly pinpoint the locations of code issues.

If there are errors, then Development may be contacted to look at them.

servlet.out

Operation, Development

servlet.err

Development

JSP logs

Administrator, Development

How to review the logs?

If you do it manually, it can be quite a daunting task to look through all the different entries to identify what you are looking for. While you could use cat, tail and vi to review them, realistically, you would be better off downloading your files to your desktop computer and using a desktop tool to sift through the file. But the best way is to automate with LogWatch.

Automate with LogWatch!

Instead of having to manually view these files in OSS directly, you can automate by using TIC LogWatch to:

Look for ERROR entries in the servlet logs

Look for any “thrown exception” messages

Extract the key information from the error messages and raise an alert email or EMS message to notify Operations or Development

Clone a copy of the entries to a Guardian file

All these and more can be done automatically with LogWatch

Feedback please

Do you find this tutorial blog helpful? Let us know what you think, and how we can make it even better. Don’t forget, you can subscribe to our blogs (top right-hand corner of this page) to get automatic email notification when a new blog is available.

Phil Ly is the president and founder of TIC Software, a New York-based company specializing in software and services that integrate NonStop with the latest technologies, including Web Services, .NET and Java.
Prior to founding TIC in 1983, Phil worked for Tandem Computer in technical support and software development.

The Challenge: Log information overload

Any Operations Support person can tell you, one of the challenges of the job is making use of logfiles to help keep abreast of potential problems.

However, if you have ever sat in front of a NonStop Viewpoint or EMS console display, you would know that the amount of messages that scroll by is impossible for one to read, let alone analyze.

Or open up any application disk log file, and I can bet you that there are tons of messages in there that no one knows what they mean. Except maybe the person who wrote the program. See if this scenario sounds familiar:

Operation: “I just saw this message in the log. What should I do with it?

Developer: “You can ignore that one too. It’s just an informational statement.”

(Wrong) Conclusion
Log messages are not important!

Making Log Messages Meaningful

The major difficulty that arises is that logfiles serve purposes that demand two somewhat conflicting sets of requirements:

Logs must be as detailed as possible, to help Development find very specific information about why and how something has happened.

Logs must be understandable and simple enough to facilitate the Operation’s and Technical Support’s job of actually making sense of them to respond in a timely manner

There is a better way – LogWatch

TIC’s Logwatch utility is designed to make a Support person’s job easier by doing some basic analyzing and display formatting for a wide range of logfile types. Logwatch can be easily configured to parse through your system’s OSS and Guardian logs to detect errors and raise alerts.Logwatch is easy to use and will work right out of the package to help you monitor OSS log files such as iTP Web Server logs and Java logs, or Guardian applications logs or EMS and VHS.

LogWatch allows you define what messages you are interested in, and filters out the rest for you.

Below is an example of how LogWatch can look for specific entries, e.g. [ERROR] in a very busy log file.

LogWatch can further extract relevant parts from the message, and add text to make it more meaningful.

Other Blogs you may be interested in

Do you find this tutorial blog helpful? Let us know what you think, and how we can make it even better. Don’t forget, you can subscribe to our blogs (top right-hand corner of this page) to get automatic email notification when a new blog is available.

Do you find this tutorial blog helpful? Let us know what you think, and how we can make it even better. Don’t forget, you can subscribe to our blogs (top right-hand corner of this page) to get automatic email notification when a new blog is available.

Phil Ly is the president and founder of TIC Software, a New York-based company specializing in software and services that integrate NonStop with the latest technologies, including Web Services, .NET and Java. Prior to founding TIC in 1983, Phil worked for Tandem Computer in technical support and software development.

I have always said that you can tell who the Nonstop Users are in the audience by the questions they ask, such as:

Is this scalable?

What is the performance?

How do I monitor it?

The last question highlights the importance that NonStop system managers place on providing responsive support for their end users.
That’s why system performance monitoring tools like Prognosis, MOMI and others are very popular at Nonstop installations. On the other hand, monitoring logs on the NonStop doesn’t get a lot of attention beyond the use of EMS or VHS. Yet it is an important topic that is worth a closer look. How would you answer the questions below?

When do you look at your logs?

“When there is a problem” seems to be a common response. Surprised? Not really. Given how overloaded everyone is, there is very little time to monitor logs. Unfortunately, “When there is a problem” usually translates to “User calls up and reports a problem.” Through routine monitoring of logs, problems can be detected and fixed before they affect the users.

Do you know where your logs are?

“Well… not all of them.”

When I speak of logs, most users think of system logs like VHS and EMS logs. Think again. Realistically, on a typical system, there are many other logs,
such as:

Application logs – entries written out by COBOL, TAL, C, C++ and other applications.

OSS logs – Are your running iTP Web server, Java, NonStop SOAP, etc? There are lots of good information in the OSS logs.

Subsystem logs – Some subsystems have their own logs.

3rd-party application logs – If you are running a 3rd-party application, e.g. Websphere MQ, chances are it has its own logs

So if you don’t know where your logs are, trying to hunt them down to resolve a problem can be time-consuming and detrimental to your service level agreement (SLA) with your end-users.

Are you overloading your logs?

Some users try to alleviate this problem by routing EVERYTHING to EMS or VHS. Unfortunately, that creates another problem: log overload.
When that happens, basically Operations staff stop looking at the hundred of log messages that fly off the screen. Yes, EMS has a very nice filtering capability that can be configured to filter out different types of messages. But the reality is that not everyone is up to the task of configuring filters, and certainly not for tens if not hundreds of different message types that flood the EMS log.

“What’s in your wallet?”

I meant.. logs. If my guess is correct, chances are your logs contain more than just error messages. They probably have warnings, statistics and “stuff programmers write to log for their own use. ” That’s another key reason a lot of people do not bother looking at logs: the content overwhelms them. See if this conversation sounds familiar:

Operator: “I just saw this message in the log. What should I do with it?”Answer: “Oh, don’t worry about it. It’s a debugging statement.”
Operator: “What about that one?”
Answer: “You can ignore that one, too.”

The result is a misinterpretation: no need to look at log messages unless someone reports a problem.

Monitor your logs

It can help you improve the quality of your end-user service.

Know your logsKnow what is in them and where they are. There’s usually a lot of useful information in those logs besides error messages.

Have a log monitoring strategyTake control and plan how you want to use the log information.

Automate as much as possibleHave a procedure in place and automate it whenever possible.

As more users are starting to use OSS in one form or another, such as iTP Web Server, Java or SQL/MX, it becomes more important to pay attention to some of the logs that reside in the OSS space. In my next article, I will focus on these OSS logs.

Feedback please

Do you find this tutorial blog helpful? Let us know what you think, and how we can make it even better. Don’t forget, you can subscribe to our blogs (top right-hand corner of this page) to get automatic email notification when a new blog is available.

Phil Ly is the president and founder of TIC Software, a New York-based company specializing in software and services that integrate NonStop with the latest technologies, including Web Services, .NET and Java. Prior to founding TIC in 1983, Phil worked for Tandem Computer in technical support and software development.