nohup

Here is a scenario

I ssh into a server
I execute a long running command. The command is currently running.
There is a network interruption, and the ssh session ends
On the server, since the ssh session ended, the process is killed. As this ssh process is being killed, the command's process is also killed.

Now, I want the command to continue executing even if the ssh session ends.

Solution? Use nohup.

nohup stands for "no hang up".

To use, just prefix the command with nohup.

nohup <command to run>

screen

Here is a scenario

I ssh into a server
I execute a long running command. The command is currently running.
There is a network interruption, and the ssh session ends
On the server, since the ssh session ended, the process is killed. As this ssh process is being killed, the command's process is also killed.

There are 2 solutions 1. use nohup to make the command execute 2. use something like screen to spawn the terminal. It will keep the session alive on server even if the ssh session gets interrupted.

ssh - debug connection issues

Usual issues:

The key doesn't have restricted permissions, so openssh refuses to use it.
The specified key in ssh config does not exist

Fix ssh key permission

# permission of the ssh dir itself
chmod 700 ~/.ssh/
# baseline permissions for all files in ssh dir
chmod 600 ~/.ssh/*
# set more open permissions for public keys in the ssh dir
chmod 644 ~/.ssh/*.pub

Debug ssh connection with verbose output

the -v flag increases verbosity (LogLevel) of output

ssh -v <host> – issues on your end (client)
ssh -vv – good enough logs of both client and server side
ssh -vvv – detailed logs from both client and server

ssh -vT <hostname>
# v: verbose
# T: Disable pseudo-terminal allocation

Eg: Debug why git via ssh isn't working:

ssh -vT [email protected]

Explanation of the -T flag - Stackoverflow - What is Pseudo TTY-Allocation? (SSH and Github)

tunnel traffic from local to server

Eg - forward local traffic on localhost's 8080, 5432, 6379 and 9092 to the server's ports

Using ssh

This method breaks down very easily and very often.

This is because if there is no traffic for a while over the ssh connection, the ssh session is ended.

So, for long term sessions with periods of inactivity, will need something that keeps the connection alive, and heals it if broken (autossh)

ssh -L 8080:localhost:8080 \
-L 5432:localhost:5432 \
-L 6379:localhost:6379 \
-L 9092:localhost:9092 \
-i ssh-key.pem [email protected]

-L - local port forward

Using autossh

autossh -M 3000 -f -N 54.183.1.38 -R 30808:localhost:30808

autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -L 5000:localhost:3306 [email protected]

-M - monitoring port
-o "ServerAliveInterval 30" - Seconds after which client will send a null packet to keep the connection alive
-o "ServerAliveCountMax 3" - Number of unacknowledged server alive messages, after which client will terminate the ssh session (indicates the remove server going down)

2>&1

https://stackoverflow.com/questions/818255/what-does-21-mean

Redirect errors and logs to a file

The numbers signify:

0 is stdin
1 is stdout
2 is stderr

Cli