“Fire and Forget” with nohup: An essential remote administration command for managing long tasks remotely.

Here’s a nifty Linux command that you normally won’t encounter in most Linux training courses, but will be absolutely essential once you get into remote administration: the nohup command. Paired with & operator, this will let you queue long-running tasks for your server (even just using a smartphone), and then immediately disconnect.

That’s the problem when ssh-ing to your server – when you ask your server to do something that takes some time (say, a simple “dnf update” or “apt-get upgrade”), you can’t disconnect until your task finishes. If you initiate that update command, for example, and then break your SSH session (say, you want to use your phone for something else now; or maybe your mobile data conks out long enough to cut the session without you really meaning to), then the update command stops because it’s parent shell quits.

That’s usually not what you want. In my case, I have to remotely administer 16 R&D servers, and as part of their normal operation – crunching numbers / performing all sorts of data processing and simulations – I have to trigger really long tasks, in the range of several hours to days. Whether I’m on my phone or laptop, I can’t have the SSH session last that long. If only there was a way to tell the command not to quit even when the shell quits, and just happily run in the background…

Oh, yeah: nohup.

Usage is simple:

nohup [your command] &

nohup means “No hang up” - this tells the command to specifically not quit even when its parent shell quits or exits. This makes it so you can safely disconnect your SSH session without inadvertently making your queued, long-runnning task prematurely end.

The & operator is what makes your command run in the background, so that you can immediately take control of your terminal again, without waiting for the command to finish.

So this:

nohup dnf -y update &

...makes your package update command run in the background (& operator) so you can proceed to do something else while it’s doing the update. The nohup command makes it persist even after you exit your SSH session – so that after you do what other thing you need to do, you can then exit your SSH session, and the DNF command will continue until it is completely done.

Since this is running in the background and will run even after you exit, how about any output that normally goes to the terminal/screen? By default, nohup dumps all of them into a file called “nohup.out”. If you find yourself using nohup regularly and frequently (as I do), you might want to specify the output file, to make sure you don’t overwrite older ones that may be useful to review later. In this case, a simple redirection can be added to our basic command:

nohup [your command] > my_nohup_outputfile.txt &

You can customize the name of your output file so that it is descriptive of the actual task you are nohup-ing, so it’s easy for you to collect these files for later review.

That’s also how you monitor the progress of your background task. I’ll often SSH into my servers to see how they’re doing, and all I’ll have to do is check the contents of the nohup output file for their current experiments. The experiments normally echo progress information specifically to make progress tracking easier, so it naturally fills the nohup output file as the experiments continue.