Long-Running Commands Over SSH: Keep Jobs Running After Disconnect

When you start a database backup or a long compile over SSH, closing the laptop lid or losing Wi‑Fi should not decide whether the job finishes. Backgrounding with & and truly detaching from the session are not the same thing: a process must stop depending on the SSH session's terminal and I/O, or use a mechanism such as nohup, setsid, or a terminal multiplexer built to survive disconnects.

The examples below use the lab-server SSH alias on vm2.lab.example (pointing at vm1.lab.example). Replace that name with your own host alias or user@remote.

Why & alone is not enough over SSH

A common mistake is wrapping a background job in a one-line ssh command and assuming the ampersand fully detaches it:

time ssh lab-server "bash -c 'sleep 8 &'"

Sample output:

real    0m8.424s
user    0m0.015s
sys     0m0.015s

The real time matches the sleep length, so your SSH client stayed connected for the whole job. That is the opposite of “walk away.”

& backgrounds the command in the remote shell, but it does not fully detach the process from the SSH connection. The child can inherit stdin, stdout, and stderr connected to the SSH channel. Even after the remote shell exits, those descriptors may stay open, so the local ssh process can keep waiting for EOF. Hangup behavior also depends on the shell, whether a pseudo-terminal is allocated, and session setup — not on & alone.

Use one of the methods below instead of a bare & inside ssh '…'.

Run a detached command with nohup

nohup (no hang up) arranges for the command to ignore SIGHUP. Over SSH you still need explicit I/O redirection — do not rely on automatic nohup.out behavior:

ssh lab-server 'nohup long-command > ~/job.log 2>&1 < /dev/null & echo $! > ~/job.pid'

GNU nohup redirects stdout to nohup.out only when stdout is attached to a terminal. In a typical ssh host '…' one-liner, stdout is a pipe, not a tty, so nohup may leave that stream connected to the SSH channel instead of creating nohup.out. Redirecting stdin, stdout, and stderr avoids that ambiguity.

Replace long-command with your backup script, build, or installer. The remote shell should return immediately while the job keeps running.

Confirm the process survived after SSH exits using the PID file:

ssh lab-server 'ps -p "$(cat ~/job.pid)" -o pid=,cmd='

Sample output (while the job is still running):

399417 sleep 15

That ps line checks the exact PID you recorded, which is more reliable than grepping the process list. Tail the log while you wait:

ssh lab-server 'tail -f ~/job.log'

Press Ctrl+c on the client to stop following the log; that does not stop the remote job.

Run a command in a new session with setsid

setsid runs the command in a separate session, which helps detach it from the SSH terminal/session relationship. In practice it is often paired with the same redirection pattern as nohup:

ssh lab-server 'setsid long-command > ~/job.log 2>&1 < /dev/null &'

Prefer setsid when you want a clean session boundary. Prefer nohup when you explicitly want hangup immunity for a single command. Either way, keep > log 2>&1 < /dev/null on non-interactive SSH jobs.

Keep an interactive job with tmux

Terminal multiplexers fit longer work where you want to reconnect and see live output. Install tmux on the remote host once if it is not already present (dnf install tmux on RHEL-family systems, apt install tmux on Debian or Ubuntu).

Start an interactive session:

ssh -t lab-server 'tmux new-session -s deploy'

Inside the tmux pane, run your long command normally:

long-running-command

Output stays visible in the pane while the command runs. Detach without stopping the session: press Ctrl+b, then d.

Your SSH connection can close; the tmux server on the remote host keeps the session alive. Reconnect later:

ssh -t lab-server 'tmux attach-session -t deploy'

List sessions when you forget the name:

ssh lab-server 'tmux ls'

For day-to-day key bindings and pane splits, see the tmux cheat sheet.

Use disown for a job already running

disown is a Bash builtin for jobs you started in an interactive SSH shell you already opened:

ssh -t lab-server

On the remote shell, start the job in the background:

long-running-command > ~/job.log 2>&1 &

List background jobs so you know which number Bash assigned:

jobs

Sample output:

[1]+  Running                 long-running-command > ~/job.log 2>&1 &

Mark the job so Bash does not send it SIGHUP when the shell exits (replace %1 with the job number shown by jobs):

disown -h %1

disown -h marks the job so Bash does not send that job SIGHUP when Bash would otherwise do so. It does not change the process's global SIGHUP disposition. disown %1 (without -h) removes the job from Bash's job table entirely.

Log out with exit. The job should keep running because you disowned it before the interactive shell closed.

Do not rely on disown inside a non-interactive ssh host '…' one-liner. In testing, bash -c 'sleep 8 & disown' still blocked the SSH client for eight seconds — the same wait as a plain background job. For automation, use nohup or setsid with explicit redirection instead.

GNU Screen alternative

GNU Screen solves the same problem as tmux: a persistent terminal you detach and reattach. Install the screen package on the remote host when you prefer Screen's interface or legacy scripts.

Start an interactive session:

ssh -t lab-server 'screen -S backup'

Run your command inside Screen, detach with Ctrl+a then d, and reattach later:

ssh -t lab-server 'screen -r backup'

Most new deployments standardize on tmux, but Screen remains common on older servers and jump hosts.

Check the job and its output

After you detach, verify once that the work is still running and that output is going where you expect:

ssh lab-server 'ps -p "$(cat ~/job.pid)" -o pid=,cmd='

Read the log:

ssh lab-server 'tail -20 ~/job.log'

For tmux or Screen, list or attach to the session instead of hunting a PID:

ssh lab-server 'tmux ls'

If the log stays empty but the process is gone, the command failed early — read the last lines of the log and rerun with a shorter test command before you launch a multi-hour job.

Troubleshooting

References

• nohup(1) — run a command immune to hangups
• setsid(1) — run a program in a new session
• tmux(1) — terminal multiplexer
• screen(1) — GNU Screen multiplexer
• ssh(1) — OpenSSH client

Summary

Long-running work over SSH fails quietly when the remote command stays bound to the session's I/O. A plain ssh host 'cmd &' one-liner often blocks the client until the job finishes because the background process can inherit the SSH channel's file descriptors — not because & alone is a complete detachment mechanism.

For scripts and automation, use explicit redirection with nohup or setsid:

ssh host 'nohup command > ~/job.log 2>&1 < /dev/null &'

Do not assume nohup.out appears on a non-interactive SSH job; GNU nohup only performs that automatic redirection when the stream is terminal-connected.

For interactive work — watching a build, adjusting flags, running follow-up commands — start tmux or GNU Screen with ssh -t, run the command inside the session, detach with the multiplexer key sequence, and attach later. disown -h is the rescue move in an interactive Bash shell: it stops Bash from sending SIGHUP to that job when you exit, but it is a poor fit for non-interactive one-liners.

After you detach, check once with ps -p "$(cat ~/job.pid)" or tmux ls and read the log path you chose. Start long work with one of these patterns before you close the laptop; use SSH keepalive settings separately when idle timeouts drop sessions you still need open.