Does processes end after ssh session disconnect?

Technical Help
1

I did start a machine learning training process yesterday which was suppose to end in ~12 hours. My computer was connected to through ssh and monitored the process but it was disconnected a couple of hours later after the start. I can’t see that it has saved the model so obviously it didn’t finish. But now I can’t tell if it is still running or if something wrong has happened. So do anyone know if processes ends automatically after a disconnect from ssh session? I’ve read that this depends on setups and I cannot see anywhere from docs how lambdalabs are setup.

2

Hi,

I’d try checking any running python processes with:
ps aux | grep python
And see if you recognize any of the commands being run is the training you started yesterday.

Something I’d recommend doing in the future, is performing your long-running training in a screen session which will hold open a session, regardless if you lose SSH connectivity. So you’d open a screen session, run your training process, then detach from the screen before performing any other tasks, and then reattach to the screen session later to check if it’s complete.

Here are a few example commands and keyboard shortcuts for managing your screen sessions:

 - Start a new screen session:
   screen

 - Start a new named screen session:
   screen -S {{session_name}}

 - Start a new daemon and log the output to screenlog.x:
   screen -dmLS {{session_name}} {{command}}

 - Show open screen sessions:
   screen -ls

 - Reattach to an open screen:
   screen -r {{session_name}}

 - Detach from inside a screen:
   Ctrl + A, D

 - Kill the current screen session:
   Ctrl + A, K

 - Kill a detached screen:
   screen -X -S {{session_name}} quit

Be sure to detach from a screen before attempting to logout as CTRL + D will log you out of that session and also kill the screen.

Hopefully this helps and please do reach out should you have any further questions or concerns!

Regards,
Calvin Wallace
Linux Support Engineer

2 Likes
3

I would add that nohup does just that (no termination on session end), simply instead of

$ command

use

$ nohup command
1 Like