Turns out that my python process had taken over listening to http in the instant that httpd was restarting. So, I killed python and tried starting httpd again - but ran into the same error. Netstat again:

Now my java process had taken over listening to http. I killed that too and could then successfully restart httpd.

But this is a terrible workaround. Why will these python and java processes start listening to port 80 as soon as httpd is restarted? How to solve?

Two other comments. 1) Both java and python processes are started by apache from a php script. But when apache is restarted, they should not be affected. And 2) I have the same setup on two other machines running Ubuntu and there's no problem there.

Any ideas?

Edit:

The Java process listens to port 7203 and the python process supposedly doesn't listen to any port. For some reason, they start listening to port 80 when apache is restarted. This hasn't happened before. On Ubuntu it runs fine. For some reason, on my current CentOS 5.5 machine, this problem arises.

4 Answers
4

The problem might be the way how Apache starts the sub-processes. They might get spawned by forking and letting the cloned Apache processes become other processes. Every clone inherits all open file-handles of the parent process, including the open handle of TCP port 80.

Netstat only shows one application associated with the open file handle, while three processes keep the handle open.

Possible solutions to your problem:

If it is important to keep the sub-processes running during a Apache restart, the simplest solution would be to start the 3 Processes as separate system-services.

If they depend on a running Apache, a 'apache stop' command should also terminate them. This could be achieved by editing the /etc/init.d/apache script.

If you are forced to start them from Apache, you need to start them as real "daemon processes"!

In theory, only one process should be listening on a given IP/port. If you need more than one application to listen on the same port, you need a reverse proxy kind of setup that will determine based on the content received which process to get the message.

It would be helpful to know what your Python and Java apps are doing. If they are also servers that listen on port 80, they are getting stuck at the point of opening port 80 for listening while Apache is running and as soon as you kill Apache, the next one in the process queue gets through and opens the port. You will need to change your Python and Java codes to listen on different ports.

You are failing to clean shutdown your apache master process and it's children are left hanging on to sockets. Instead of running restart, run a service httpd stop. If this fails to exit cleanly, you need to investigate further the java and python apps you've enabled through apache.

If a service httpd stop doesn't work correctly, uninstall your java and python apps until you can cleanly restart. Then, investigate why those processes are not terminating correctly.