Self Diagnosis from Common CTSTATUS Messages

Self Diagnosis from Common CTSTATUS Messages

Posted By: David Faust

The c‑treeACE database engine is designed at every level to require as little attendance and maintenance as possible. Even for those times when the server detects potential problems, every attempt is made to output the cause, and possible solution. The CTSTATUS.FCS log file retains a record of any notable information and should be inspected from time to time to prevent and diagnose most common issues that can arise.

Check your CTSTATUS.FCS file any time you have cause to believe there is a problem with the database engine. This status log file can be found in the current working directory of the c‑treeACE process. Typically, this is the directory of the server process binary, or an area pointed to by the LOCAL_DIRECTORY configuration keyword.

Here is a reference list of the most common messages your FairCom support team will ask about or look for in your status logs should you have problems. In many cases, simply searching for these is enough to handle most situations.

Up and Running

Upon server startup, a handful of messages are written to the status log. Usually, this is a quick activity. Sometimes it may take a while if auto-recovery from a previous abnormal shutdown is taking place. Be sure the “Server is Operational” message is output if you are unable to connect to the server. Once this message is output, you are ready to go.

Note: The server serial number and TCP/IP connection port is output in this final information for your reference.

Build Info

When requesting server support from your FairCom team, one of the first questions you will be asked is “What is your server build?”. This information is output in the status log output on server startup. Be sure to identify your version and build information and provide this when you make a support request.

Fri Aug 13 10:30:34 2017

- User# 00001 Startup c‑tree(R) Server - V11.3.27711(Build-170526)

Server Name and Ports

Clients connect to the server via a communications port. Usually TCP/IP is used across networks. (Windows clients and servers on the same machine will use a shared memory protocol by default.)

Fri Aug 13 10:30:34 2017

- User# 00001 Server Name: FAIRCOMS

Fri Aug 13 10:30:36 2017

- User# 00001 Server using TCP/IP SQL Socket Port Number: 6597

Your ISAM port is identified by the server name. There are two connection ports that are possible with c‑treeACE SQL. One is the ISAM port, and the other is the SQL port; the SQL port is numeric only. These ports are configured in your ctsrvr.cfgconfiguration file.

Note: The ISAM TCP/IP port number is determined by the following formula. Start at base 5000 (configurable) and add the ASCII sum of the server name characters. As a result, keep in mind two different server names could potentially result in the same port number calculation! (For example, PROD_SERVER1 = SERVER_PROD1)

In most cases, this error is an informational message that a connection to the server, usually a dropped client, has occurred. When a client establishes a connection to the server, that connection remains through the life of the client until the client issues a StopUser() or CloseISAM() call. FairCom Client/Server Communication

Consider the case where a client application abnormally terminates though. A proper disconnection does not take place. The connection from the server’s view appears valid, however, until the server issues a read or write operation on the connection. Only at this time will the server note that the connection is no longer available, and then outputs a message toCTSTATUS.FCS indicating so. A ARQS_ERR (127) indicates that the server was attempting to write information to the connection and the ARSP_ERR (128) indicates the server was attempting to read information from the connection.

Normally, a few of these messages is not an issue and can be safely ignored. An easy example to generate such a message is to connect with a c‑treeACE command line client utility and use control-C to break out of the application. You will frequently find a message logged after this action.

Sometimes a very large number of these messages appear in the status log concerning administrators. While these errors can be masked from CTSTATUS.FCS output with the CTSTATUS_MASK VDP_ERR configuration keyword, it is recommended that you make every attempt to identify the root cause of the disconnections. A status log with a large number of these messages, in nearly every case, is indicative of an application that failed to properly disconnect from the server upon closing. Many times, it is simply that the application was closed improperly, and these can be safely ignored. In some multithreaded applications, tracking of connections to the server is miscalculated, and this is a symptom of a larger problem that should be inspected. This message has also indicated a faulty networking environment in several instances due to dropped connections. Faulty network interface cards, cables, switches and even routers have been identified after the appearance of these messages.

FYI — VDP originally designated “Virtual Device Protocol”.

Note: You may see many of these errors appear all at once if the server is stopped while clients are connected. This is normal server end processing.

License Notice

c‑treeACE V9 introduced the ability to limit the number of CPUs the server was licensed, or configured to run on. This message will be output every five (5) minutes when the activation key specifies fewer CPUs than detected on the system. Contact your FairCom business representative to arrange for a licensing change, or include the CPU_AFFINITY keyword on selected platforms to limit the number of CPUs the c‑treeACE process can access.

c‑treeACE is licensed for 2 CPU's, but 4 CPU's have been detected in the

host machine. Either upgrade the c‑treeACE license to support a greater

number of CPU's or bind c‑treeACE to specific CPU's using the

CPU_AFFINITY configuration keyword.

Note: On Windows machines, when run as a tool tray process, a “balloon tip” will appear every five minutes, along with this message in the status log.

Routine Diagnostic Messages

This following message can occur during normal server startup. I000000x.FCS files are server housekeeping files created at server startup, and deleted upon a clean server shutdown. If the file is missing during startup (normal) the check for the file fails, and this message is output. During startup, low level I/O errors are detected and output as part of normal startup operations. These can be safely ignored

Dynamic Dump

The dynamic dump is the preferred backup method for online c‑tree files. When processing a dynamic dump, a series of messages reporting progress through the dump is output. It is important to inspect for errors within these messages to ensure you have a pristine backup of your data.

Sat Jul 15 04:45:00 2017

- User# 00011 DD: processing script file...

Sat Jul 15 04:45:00 2017

- User# 00011 My_DynDump.scr

Sat Jul 15 04:45:00 2017

- User# 00011 DD: Begin dynamic dump...

Sat Jul 15 04:45:00 2017

- User# 00011 My_DynDump.scr

Sat Jul 15 04:45:02 2017

- User# 00011 DD: Effective Time

Sat Jul 15 04:45:02 2017

- User# 00011 DD: Dynamic Dump begins with log #66

Sat Jul 15 04:45:02 2017

- User# 00011 DD: could not access...: 12

Sat Jul 15 04:45:02 2017

- User# 00011 /u/My_DumpNow.scr: 13

The most frequent errors are files that could not be found. For example, the dump script was not updated with file name changes. A missing or deleted dump script file will also cause errors.

Tip: Configure your dump script files as read only to prevent them from accidental deletion.

Automatic Recovery

During startup, the server examines its transaction logs to determine if there was a previous clean shutdown. The transaction logs maintain changes to transaction controlled data and index files and are used to recover information to bring these files back to a consistent state. Various messages appear during the multiple stages of recovery.

Mon Aug 21 10:46:26 2017

- User# 00001 Index repair time: 0 second(s) for 1 repair(s).

Mon Aug 21 10:46:26 2017

- User# 00001 tranrcv: Recomposing index file FAIRCOM.FCS DI:

Mon Aug 21 10:46:26 2017

- User# 00001 tranrcv: Processing abort node list entries.

Mon Aug 21 10:46:26 2017

- User# 00001 tranrcv: Recomposing index file mark.idx:

Mon Aug 21 10:46:26 2017

- User# 00001 tranrcv: Processing LOGIDX node entries.

Mon Aug 21 10:46:26 2017

- User# 00001 tranrcv: Checking index delete stack.

Mon Aug 21 10:46:26 2017

- User# 00001 tranrcv: Recomposing index file mark.idx M#01:

Mon Aug 21 10:46:26 2017

- User# 00001 tranrcv: Processing LOGIDX node entries.

Mon Aug 21 10:46:26 2017

- User# 00001 tranrcv: Recomposing index file mark.idx M#02:

Mon Aug 21 10:46:26 2017

- User# 00001 tranrcv: Processing LOGIDX node entries.

Mon Aug 21 10:46:26 2017

- User# 00001 Index composition time: 0 second(s).

Some files, however, may no longer be present. The SKIP_MISSING_FILES configuration option is used to skip files that may no longer be present, yet referenced in the server’s transaction logs.

Tue Aug 22 16:15:38 2017

- User# 00002 RCVchk: skipped .\ctreeSQL.dbs\customer.dat

Tue Aug 22 16:15:38 2017

- User# 00002 .\ctreeSQL.dbs\inventory.dat

Tip: Add SKIP_MISSING_FILES to your ctsrvr.cfg configuration file, however, comment it by default. This way it is immediately available should you really need to use it during a server recovery process. Only use this option when you are certain files should actually be missing, otherwise they will be lost after recovery. In that case, you will only be able to restore them from a prior backup.

Clean Shutdown

Once the Server shutdown completed message appears, the server is safely down, and any maintenance or update activities can take place.

Fri Aug 11 10:51:02 2017

- User# 00013 Server shutdown initiated

Fri Aug 11 10:51:04 2017

- User# 00013 Communications terminated

Fri Aug 11 10:51:04 2017

- User# 00013 Perform system checkpoint

Fri Aug 11 10:51:04 2017

- User# 00013 Server shutdown completed

Fri Aug 11 10:51:04 2017

- User# 00013 Maximum memory used was 21584854 bytes

Fri Aug 11 10:51:04 2017

- User# 00013 Maximum number of lock hash bins for a file was 128

Fri Aug 11 10:51:04 2017

- User# 00013 Maximum number of lock hash bins for a user was 256

Fri Aug 11 10:51:04 2017

- User# 00013 Maximum number of preimg hash bins for a user was 512

Failed Server Process

Oops. Your c‑treeACE server fails for some reason. While it is not possible to detect every situation which may require the server to shut down, a best attempt is made. By far, the most common cause is lack of disk space for either files, or the server’s transaction logs. Without disk space, the server cannot continue operation, and will immediately come down as clean as possible to protect your data. This is noted with a READ_ERR (36) or WRITE_ERR (37) in the final uerr_cod.

Be sure to never start a server that is undergoing auto-recovery. It is always recommended to check the status log BEFORE you attempt to restart a server you think has not started. Look for the “Server shutdown completed” message, or output such as the previous before restarting any server.

On many platforms, a core or dump file will be generated. Be sure to save off this file should your FairCom support team request this information. The stack trace and core file information will in many cases determine an immediate cause and solution to the problem.

Concluding Remarks

This certainly doesn’t cover every message that can occur, however, with an occasional routine inspection of yourCTSTATUS.FCS log, you should be able to identify potential problems before they happen. Even in those cases where a failure occurs, there may be enough information to quickly diagnose the situation and get back up and running quickly. And always remember, your FairCom support team is here ready to help.

Archives

Resources

SIGN UP FOR OUR NEWSLETTER

FairCom needs the contact information you provide to us to contact you about our products and services.
You may unsubscribe from these communications at anytime. For information on how to unsubscribe,
as well as our privacy practices and commitment to protecting your privacy, check out our Privacy Policy.