I'm connecting to a Linux machine through SSH, and I'm trying to run a heavy bash script that makes filesystem operations. It's expected to keep running for hours, but I cannot leave the SSH session open because of internet connections issues I have.

I doubt that running the script with the background operator, the ampersand (&), will do the trick, because I tried it and later found that process was not completed. How can I logout and keep the process running?

9 Answers
9

Screen is a "virtual" terminal which you can run from a "real" terminal (actually all terminals today are "virtual" but that is another topic for another day). Screen will keep running even if your ssh session gets disconnected. Any process which you start in a screen session will keep running with that screen session. When you reconnect to the server you can reconnect to the screen session and everything will be as if nothing happened, other than the time which passed.

nohup ignores the HUP signal. The HUP signal is sent to all child processes when the controlling terminal is closed (for example by a disconnected ssh session). Processes started with nohup will not receive the HUP signal. Although the processes will keep running you can no longer interact with them because they are no longer attached to any terminal.

nohup is useful mainly for long running batch processes which, once started, no longer need any attention.

As an alternative to nohup, the Bash built-in command disown can be used to tell bash to not send the program the HUP signal.

I like this answer as it provides a solution for both interactive and non-interactive situations. In the interactive case, screen gives you a lot more options, but if you are using authorized_keys to allow people to run a script remotely via ssh, the nohup option is a nice simple way for the script to start processes which last longer than the ssh session used to start them.
–
Mark BoothSep 15 '11 at 10:33

lesmana, and @Mark Booth I like your answers as it tells more useful tips.. 1+
–
rahmanisbackSep 15 '11 at 17:17

1

@rahmanisback - Remember that you can change your accepted answer at any time. Just because so far EricA's answer is voted most highly doesn't mean it's the best answer for you, indeed making it your accepted answer may well encourage more people to vote it up as a good answer.
–
Mark BoothSep 15 '11 at 23:02

In bash, the disown keyword is perfectly suited to this. First, run your process in the background (either use &, or ^Z then type bg):

$ wget --quiet http://server/some_big_file.zip &
[1] 1156

By typing jobs you can see that the process is still owned by the shell:

$ jobs
[1]+ Running wget

If you were to log out at this point, the background task would also get killed. However, if you run disown, bash detaches the job and allows it to continue running:

$ disown

You can confirm this:

$ jobs
$ logout

You can even combine the & and disown on the same line, like:

$ wget --quiet http://server/some_big_file.zip & disown
$ logout

This is better than running nohup in my opinion because it doesn't leave nohup.out files littered all over your filesystem. Also, nohup must be run before you run the command — disown can be used if you only decide later on that you want to background and detach the task.

That's a very good answer, 1+. The only preference for nohup or Screen would be the independence of bash, and might be used with other shell. But I will stick to your approach whenever I'm using bash.
–
rahmanisbackSep 15 '11 at 17:34

Yes – this is bash–specific, as bash is the only shell I've ever used. I wonder if other shells support anything similar (i.e. launching in background without nohup) – it would be fantastic if someone could post other answers for other shells.
–
Jeremy VisserSep 16 '11 at 2:46

1

see my answer showing how to do it it with any sh-lookalike
–
w00tSep 16 '11 at 16:49

1

+1 because of being able to decide later. Just this moment i had a need for this. worked as advertised
–
sean9999Nov 2 '14 at 3:45

dtach is a program written in C that emulates the detach feature of
screen, which allows a program to be executed in an environment that
is protected from the controlling terminal. For instance, the program
under the control of dtach would not be affected by the terminal being
disconnected for some reason.

dtach was written because screen did not adequately meet my needs; I
did not need screen's extra features, such as support for multiple
terminals or terminal emulation support. screen was also too big,
bulky, and had source code that was difficult to understand.

screen also interfered with my use of full-screen applications such as
emacs and ircII, due to its excessive interpretation of the stream
between the program and the attached terminals. dtach does not have a
terminal emulation layer, and passes the raw output stream of the
program to the attached terminals. The only input processing that
dtach does perform is scanning for the detach character (which signals
dtach to detach from the program) and processing the suspend key
(which tells dtach to temporarily suspend itself without affecting the
running program), and both of these can both be disabled if desired.

Contrary to screen, dtach has minimal features, and is extremely tiny.
This allows dtach to be more easily audited for bugs and security
holes, and makes it accessible in environments where space is limited,
such as on rescue disks.

Thank you. I came here just to post about dtach too. It's my go-to thing for terminal detachment now; screen does WAY too much, and interferes with input to a ridiculous degree. The fact screen needs its own termcap is pretty unsettling.
–
fluffySep 15 '11 at 18:12

Thanks. I will install it and give it a try.
–
rahmanisbackSep 15 '11 at 22:32

byobu on Ubuntu is a nice front-end to screen. By pressing Ctrl-? you get a list of all the keyboard shortcuts. It adds a status bar that can be useful for watching CPU load, disk space, etc. Overall it provides an experience that I would describe as a terminal based VNC connection.

nohup allows starting a job in the background with its output routed to a log file, which can always be redirected to /dev/null if not required.

Here's a way to daemonize any shell process, no external programs needed:

( while sleep 5; do date; done ) <&- >output.txt &

When you then close your session, the job will continue to run as evidenced by the output.txt file (which has buffering so it takes a while to show non-zero). Don't forget to kill your job after testing.

So all you need to do is close stdin and background the job. To be really good, first cd / so you don't hold on to a mount.

Interesting. That emerged some notable questions. I can see that you set the STDIN to nothing, the - operator? What is the difference between < /dev/null and &- ? I guess that STDIN (and others STDOUT and STDERR) could be assigned either a file by < file, or a stream by <& stream in case of STDIN. Would it be the same using < /dev/null in your example above? And Does the operator - above refer to a null as the stream?
–
rahmanisbackSep 16 '11 at 22:29

When you do x<&-, that closes file descriptor x. In this case there is no x, which makes bash default to 1, i.e. standard input. If you use </dev/null you're not closing stdin, you're just giving an empty file to the program as input.
–
w00tSep 19 '11 at 17:15

1

And to be honest I don't really know why this daemonizes the thing you're running :-) It does work though, we use it in production. I discovered it while messing around hoping I could daemonize a process in shell without needing anything special - so I started by closing stdin and that was enough. I should go read some shell sources but I presume that if you close stdin and background the process, it also detaches the process.
–
w00tSep 19 '11 at 17:19

2

I think the reason is that a SIGHUP (the actual signal that causes the child to quit when the shell dies) is triggered when a parent process closes its child's stdin handle. However, if stdin starts out as null, rather than being closed after the fact, there is no way for the parent to trigger the SIGHUP. Nice find, though — never would have thought of that.
–
Jeremy VisserOct 15 '11 at 14:06