We have a build file that does a git checkout, compiles a war, deploys to tomcat and then tail -f 's the catalina.log. If the ssh session over which this is done somehow dies, even though we are only tail -f'ing the log, the entire tomcat instance will die. Obviously this is not ideal, but I'd like to know why this is happening and how to fix it.

If you comment out the tail -f command, does tomcat still die?
–
Keith ThompsonJan 17 '12 at 10:32

Other processes may be involved. So you should look at the pstree of the sshd process while tomcat runs.
–
artistoexJan 17 '12 at 11:10

@Keither : No, obviously if tail -f is not there, the script returns (0) and then the children go play in the park. I'd like the tail -f functionality; but without forked/background processes dying.
–
NoxvilleJan 17 '12 at 13:02

@artistoex: I'll have a poke at that, hopefully there is a hint there (although I believe I have poked at this before).
–
NoxvilleJan 17 '12 at 13:02

1 Answer
1

Don't run dæmons from an interactive shell connected to a terminal.

Configure dæmons to be run properly, managed by whatever dæmon supervisor(s) your system comes with, be that anything from s6-supervise to launchd. Use the management interface to bring the dæmons up and down.

When your SSH connection is lost, the SSH server process closes the master side of the pseudo-terminal that it opens. This causes a SIGHUP to be sent to the session leader process, the top level shell, and to all of the processes in the foreground process group. When bash receives a SIGHUP, it sends SIGHUP to all of the child processes that it has not been told to disown. That of course includes the tomcat process that you ran from that interactive shell.

Dæmons should have no controlling TTY, and should not belong to any process group or session that is known to any TTY, lest unprivileged users be able to send them random and possibly unexpected signals. Running them from your system's dæmon supervisor(s) will ensure that this, and indeed all other aspects of the dæmon's process state, are all set appropriately for a dæmon process. Many dæmon supervisors will handle logging automatically for the dæmon, too.