python subprocess call with pipe, shell=True not working

Question

I'm trying to do a subprocess call from my python script that replaces the carriage return and newline characters in a file with a space, and then saves it back to file itself. I have verified that this works:

cat file.txt | tr '\r\n' ' ' > file.txt

and so tried to do the same thing in python. My call looks like this:

formatCommand = "cat " + fileName + " | tr '\\r\\n' ' ' > " + fileName
print(formatCommand) #this showed me that the command above is being passed
subprocess.call(formatCommand, shell=True)

Rather than successfully delete the newlines like I expect it to, the file ends up being empty.

I consulted this post about a similar problem, but the solution was to use shell=True which I already employ, and the redirect makes the Popen more complicated. Furthermore, I don't see why it doesn't work with the shell=True.

• abarnert: Why are you using this in the first place? What you're trying to do in a shell pipeline is just as easy in Python...
• dano: This actually results in an empty file for me when I do it directly from a bash shell.
• taronish4: You're right... I could totally just do a replace call. However, now I'm just curious as to why it doesn't work, so I'll keep the question up.
• abarnert: @taronish4: dano is right. The problem is that your tr invocation is wrong, both from the shell and from Python; it has nothing to do with your use of subprocess.
• abarnert: Also, as a side note: You should look into using either str.format or % rather than concatenation, especially when you have quotes like this all of the place. And raw strings might help too. For example, isn't this more readable (and more obviously right)? r"cat {} | tr '\r\n' ' ' > {}".format(fileName, fileName)

Answer 1

There's a race condition in your shell command. The first command in your pipeline is cat file.txt, the second command is tr '\r\n' ' ' > file.txt. Both commands are run in parallel at the same time. The first command reads from file.txt, the second trunctates file.txt and then writes to it. If the truncation happens before the first command reads from the file then the file will be empty.

• jfs: note: the redirection > file.txt is done by the shell before tr command is even run. I don't know whether cat may ever see non-empty file.txt here.
• Ross Ridge: The whole tr command, including redirections, is done in parallel to to the cat command. So it's possible for the cat command to output something, even finish if the file is smaller than the pipe buffer, before the tr redirections to occur. For example echo test > foo.txt; cat foo.txt | tr t T $(sleep 1) > foo.txt; cat foo.txt will probably print TesT.