python subprocess communicate freezes

Question

I have the following python code that hangs :

cmd = ["ssh", "-tt", "-vvv"] + self.common_args
cmd += [self.host]
cmd += ["cat > %s" % (out_path)]
p = subprocess.Popen(cmd, stdin=subprocess.PIPE,
 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate(in_string)

It is supposed to save a string (in_string) into a remote file over ssh.

The file is correctly saved but then the process hangs. If I use

cmd += ["echo"] instead of
cmd += ["cat > %s" % (out_path)]

the process does not hang so I am pretty sure that I misunderstand something about the way communicate considers that the process has exited.

do you know how I should write the command so the the "cat> file" does not make communicate hang ?

• Kobi K: I think this post can help, it's an example of how to write to a remote file using SSH
• Andrew Walker: Slight tangent, but rather than using an SSH process to do this, have you considered something like SSHFS ? This would mean that you'd only need to worry about writing to a file, rather than maintaining all of this

Answer 1

-tt option allocates tty that prevents the child process to exit when .communicate() closes p.stdin (EOF is ignored). This works:

import pipes
from subprocess import Popen, PIPE
cmd = ["ssh", self.host, "cat > " + pipes.quote(out_path)] # no '-tt'
p = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate(in_string)

You could use paramiko -- pure Python ssh library, to write data to a remote file via ssh:

#!/usr/bin/env python
import os
import posixpath
import sys
from contextlib import closing
from paramiko import SSHConfig, SSHClient
hostname, out_path, in_string = sys.argv[1:] # get from command-line 
# load parameters to setup ssh connection
config = SSHConfig()
with open(os.path.expanduser('~/.ssh/config')) as config_file:
 config.parse(config_file)
d = config.lookup(hostname)
# connect
with closing(SSHClient()) as ssh:
 ssh.load_system_host_keys()
 ssh.connect(d['hostname'], username=d.get('user'))
 with closing(ssh.open_sftp()) as sftp:
 makedirs_exists_ok(sftp, posixpath.dirname(out_path))
 with sftp.open(out_path, 'wb') as remote_file:
 remote_file.write(in_string)

where makedirs_exists_ok() function mimics os.makedirs():

from functools import partial
from stat import S_ISDIR
def isdir(ftp, path):
 try:
 return S_ISDIR(ftp.stat(path).st_mode)
 except EnvironmentError:
 return None
def makedirs_exists_ok(ftp, path):
 def exists_ok(mkdir, name):
 """Don't raise an error if name is already a directory."""
 try:
 mkdir(name)
 except EnvironmentError:
 if not isdir(ftp, name):
 raise
 # from os.makedirs()
 head, tail = posixpath.split(path)
 if not tail:
 assert path.endswith(posixpath.sep)
 head, tail = posixpath.split(head)
 if head and tail and not isdir(ftp, head):
 exists_ok(partial(makedirs_exists_ok, ftp), head) # recursive call
 # do create directory
 assert isdir(ftp, head)
 exists_ok(ftp.mkdir, path)

• Jerome WAGNER: but why does it work with -tt + echo ? your explanation seems to imply that it should not work with echo either !?
• jfs: @JeromeWAGNER: echo doesn't expect input on stdin. Try any program that doesn't exit until all input (stdin) is consumed. Though I do not understand it fully either. It might be a bug somewhere in the chain with closing pseudo-tty file descriptors. It is platform dependant e.g., here's code that uses pty (it is how I imagine ssh launches remote child processes (I can be completely wrong about it))

Answer 2

It makes sense that the cat command hangs. It is waiting for an EOF. I tried sending an EOF in the string but couldn't get it to work. Upon researching this question, I found a great module for streamlining the use of SSH for command line tasks like your cat example. It might not be exactly what you need for your usecase, but it does do what your question asks.

Install fabric with

pip install fabric

Inside a file called fabfile.py put

from fabric.api import run
def write_file(in_string, path):
 run('echo {} > {}'.format(in_string,path))

And then run this from the command prompt with,

fab -H username@host write_file:in_string=test,path=/path/to/file

• Jerome WAGNER: thanks for the idea but I am trying to patch an existing project (ansible) and the dependency on fabric would not be accepted.
• William Denman: welcome. but yes, I figured it might not be suited for your needs. Did Kobi K's answer work for you?
• Jerome WAGNER: yes Kobi K's answer seems to work but I do not understand the issue with communicate. I found that removing the "-tt" makes the thing work ; this is a mistery for me.
• Snidhi Sofpro: For cases where ssh needs to be run with the -tt option from python, like for interactive sessions that need input from the stdin, even for remote scripts that need sudo pw, executing the ssh script via os.system() would overcome the issue. Related post: stackoverflow.com/questions/3692387/…