Pythonic way to detach a process?

Question

I'm running an etcd process, which stays active until you kill it. (It doesn't provide a daemon mode option.) I want to detach it so I can keep running more python.

What I would do in the shell;

etcd & next_cmd

I'm using python's sh library, at the enthusiastic recommendation of the whole internet. I'd rather not dip into subprocess or Popen, but I haven't found solutions using those either.

What I want;

sh.etcd(detach=True)
sh.next_cmd()

or

sh.etcd("&")
sh.next_cmd()

Unfortunately detach is not a kwarg and sh treats "&" as a flag to etcd.

Am I missing anything here? What's the good way to do this?

Answer 1

To implement sh's &, avoid cargo cult programming and use subprocess module directly:

import subprocess
etcd = subprocess.Popen('etcd') # continue immediately
next_cmd_returncode = subprocess.call('next_cmd') # wait for it
# ... run more python here ...
etcd.terminate() 
etcd.wait()

This ignores exception handling and your talk about "daemon mode" (if you want to implement a daemon in Python; use python-daemon. To run a process as a system service, use whatever your OS provides or a supervisor program such as supervisord).

• Dr_Zaszuś: Using subprocess.Popen('python file.py') on Windows returns FileNotFoundError: [Errno 2] No such file or directory: 'python file.py'. What could be the problem?
• jfs: It looks like a good separate question. Related: Calling a python script with input within a python script using subprocess
• Taylor D. Edmiston: @Dr_Zaszuś Try adding shell=True as a kwarg to your Popen call since the command has multiple words. Or you can pass the words as a list of strings. docs.python.org/3.7/library/subprocess.html#subprocess.Popen

Answer 2

Author of sh here. I believe you want to use the _bg special keyword parameter http://amoffat.github.io/sh/#background-processes

This will fork your command and return immediately. The process will continue to run even after your script exits.

Answer 3

subprocess is easy enough to do this too:

This approach works (python3). The key is using "start_new_session=True"

UPDATE: despite Popen docs saying this works, it does not. I found by forking the child and then doing os.setsid() it works as I want

client.py:

#!/usr/bin/env python3
import time
import subprocess
subprocess.Popen("python3 child.py", shell=True, start_new_session=True)
i = 0
while True:
 i += 1
 print("demon: %d" % i)
 time.sleep(1)

child.py:

#!/usr/bin/env python3
import time
import subprocess
import os
pid = os.fork()
if (pid == 0):
 os.setsid()
 i = 0
 while True:
 i += 1
 print("child: %d" % i)
 time.sleep(1)
 if i == 10:
 print("child exiting")
 break

output:

./client.py
demon: 1
child: 1
demon: 2
child: 2
^CTraceback (most recent call last):
 File "./client.py", line 9, in <module>
 time.sleep(1)
KeyboardInterrupt
$ child: 3
child: 4
child: 5
child: 6
child: 7
child: 8
child: 9
child: 10
child exiting

• Roy Shilkrot: looks like you copy-pasted this answer in all similar stackoverflow questions, but it's not correct - it does not detach the subprocess from it's parent, it simply creates a subprocess (on linux, python 3.8)
• Goblinhack: Bit confused as the Popen docs say "If start_new_session is true the setsid() system call will be made in the child process prior to the execution of the subprocess. (POSIX only)" And "The setsid() library function allows a process to disassociate itself from its parent:". So it "should" be working. Admittedly I tested on mac. Ok I need to check...
• Goblinhack: You're correct... Ok I found a workaround by forking in the child again and doing os.setsid. That seems to work - can you confirm ? This does sound like a Popen bug as I've seen in many places "As of Python 3.2, you can use subprocess.Popen() and pass start_new_session=True to accomplish fully detach the child process from the parent."

Answer 4

Note in the following two examples there is a call to time.sleep(...) to give etcd time to finish starting up before we send it a request. A real solution would probably involving probing the API endpoint to see if it was available and looping if not.

Option 1 (abusing the multiprocessing module):

import sh
import requests
import time
from multiprocessing import Process
etcd = Process(target=sh.etcd)
try:
 # start etcd
 etcd.start()
 time.sleep(3)
 # do other stuff
 r = requests.get('http://localhost:4001/v2/keys/')
 print r.text
finally:
 etcd.terminate()

This uses the multiprocessing module to handle the mechanics of spawning a background tasks. Using this model, you won't see the output from etcd.

Option 2 (tried and true):

import os
import signal
import time
import requests
pid = os.fork()
if pid == 0:
 # start etcd
 os.execvp('etcd', ['etcd'])
try:
 # do other stuff
 time.sleep(3)
 r = requests.get('http://localhost:4001/v2/keys/')
 print r.text
finally:
 os.kill(pid, signal.SIGTERM)

This uses the traditional fork and exec model, which works just as well in Python as it does in C. In this model, the output of etcd will show up on your console, which may or may not be what you want. You can control this by redirecting stdout and stderr in the child process.

• Trevor Boyd Smith: similar to this solution which uses multiprocessing and os.fork()

Answer 5

Posting this if for no other reason than finding it next time I google the same question:

 if os.fork() == 0:
 os.close(0)
 os.close(1)
 os.close(2)
 subprocess.Popen(('etcd'),close_fds=True)
 sys.exit(0)

Popen close_fds closes the file descriptors other than 0,1,2, so the code closes them explicitly.