Received: (qmail 7732 invoked by uid 2012); 17 May 1999 17:13:08 -0000
Message-Id: <19990517171308.7731.qmail@hyperreal.org>
Date: 17 May 1999 17:13:08 -0000
From: Eero Pikat
Reply-To: eero@barchart.com
To: apbugs@hyperreal.org
Subject: Random crash
X-Send-Pr-Version: 3.2
>Number: 4430
>Category: os-windows
>Synopsis: Apache randomly loops running CGI scripts (Apache loops, not the CGI)
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: apache
>State: closed
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: apache
>Arrival-Date: Mon May 17 10:20:00 PDT 1999
>Closed-Date: Sun Oct 01 13:25:15 PDT 2000
>Last-Modified: Sun Oct 01 13:25:15 PDT 2000
>Originator: eero@barchart.com
>Release: 1.3.6
>Organization:
>Environment:
Windows NT Server 4.0
>Description:
I have three machines, all equally configured, running different programs, but all running either Apache 1.3.3, 1.3.4, and 1.3.6. (For the following, by crash I mean that Apache goes into some loop, taking up 10% of CPU. It will accept connections, but never returns the document.) The machine that runs almost no cgi's rarely crashes, even though it serves more documents than any other. The second machine, that serves almost as many documents as the first, but all of which are cgi's, crashes at random about twice a day. This is independent of server load (meaning that it can crash when the server isn't busy, or that it crashes when the server is less busy than at other times.)
It doesn't seem to be any one cgi, since they all run evenly, and if it were, it owuld crash much more often. Also, if the cgi blows up, then usually you have the exe that takes up the CPU time. (Example: an infinite perl fork program would cause perl.exe to take up 100% cpu, not Apache.)
If anything, it seems to be an error in STDIN on a form post. But not always. Any independent try returns success. I can't get it to fail, but it does so randomly. And by random, I mean that at times it will go hours, under a heavy load, no problem. Then it'll crash twice or thrice in 10 minutes, and then no problems for hours.
The server logs offer no help either. I've trapped the server logs around a crash. The error log will have very few entries, and nothing suspicious. The access log also doesn't seem to have anything suspicious. At least, I'll see the same things at other times when it doesn't crash.
Also, it seems that Apache adds the log entry after the command completes. (It needs to know the status code to be able to log it.) This thing seems to crash Apache, so that Apache no longer is able to write to the log.
Has anyone else had this problem. (I have roughly 45000 executions of various cgi's on a daily basis on this server.)
The other servers are much less affected (fewer cgi's executing, but at least once a week, the other server will also crash. Sometimes when absolutely nothing is happening.)
EAP.
>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
Synopsis-Changed-From: Random crash
Synopsis-Changed-To: Apache randomly loops running CGI scripts (Apache loops, not the CGI)
Synopsis-Changed-By: coar
Synopsis-Changed-When: Wed May 26 08:54:28 PDT 1999
Release-Changed-From-To: 1.3.3, 1.3.4, 1.3.6-1.3.6
Release-Changed-By: coar
Release-Changed-When: Wed May 26 08:54:28 PDT 1999
Category-Changed-From-To: general-os-windows
Category-Changed-By: coar
Category-Changed-When: Wed May 26 08:54:28 PDT 1999
Comment-Added-By: stoddard
Comment-Added-When: Thu May 27 08:13:35 PDT 1999
Comment-Added:
What is MaxRequestsPerChild set to? Do you notice the
problem after doing a restart (apache -k restart)?
Comment-Added-By: stoddard
Comment-Added-When: Thu May 27 09:04:02 PDT 1999
Comment-Added:
A few more questions...
How much data do your CGI's typically serve? Are they invoked
in response to a POST (FORM submit)? If so, how much data is
POST'ed?
When the problem is encountered, does Apache quite serving
ALL requests, or just the request that is hanging?
When you hit the problem, could you go into performance monitor, then display the number of active threads in
both of the Apache.exe processes? You should see two threads in the parent process, and ThreadsPerChild+1 threads in the
child process.
State-Changed-From-To: open-closed
State-Changed-By: wrowe
State-Changed-When: Sun Oct 1 13:25:15 PDT 2000
State-Changed-Why:
[This is a standard response.]
No response from submitter, assuming issue has been resolved.
>Unformatted:
[In order for any reply to be added to the PR database, ]
[you need to include in the Cc line ]
[and leave the subject line UNCHANGED. This is not done]
[automatically because of the potential for mail loops. ]
[If you do not include this Cc, your reply may be ig- ]
[nored unless you are responding to an explicit request ]
[from a developer. ]
[Reply only with text; DO NOT SEND ATTACHMENTS! ]