how to simulate tar filename substitution across piped subprocess.Popen() calls?

Mon Nov 12 13:30:05 EST 2012

On 12/11/12 18:22:44, jkn wrote:
> Hi Hans
>> On Nov 12, 4:36 pm, Hans Mulder <han... at xs4all.nl> wrote:
>> On 12/11/12 16:36:58, jkn wrote:
>>>>>>>>>>>>>>>>>>>>> slight followup ...
>>>>> I have made some progress; for now I'm using subprocess.communicate to
>>> read the output from the first subprocess, then writing it into the
>>> secodn subprocess. This way I at least get to see what is
>>> happening ...
>>>>> The reason 'we' weren't seeing any output from the second call (the
>>> 'xargs') is that as mentioned I had simplified this. The actual shell
>>> command was more like (in python-speak):
>>>>> "xargs -I {} sh -c \"test -f %s/{} && md5sum %s/{}\"" % (mydir, mydir)
>>>>> ie. I am running md5sum on each tar-file entry which passes the 'is
>>> this a file' test.
>>>>> My next problem; how to translate the command-string clause
>>>>> "test -f %s/{} && md5sum %s/{}" # ...
>>>>> into s parameter to subprocss.Popen(). I think it's the command
>>> chaining '&&' which is tripping me up...
>>>> It is not really necessary to translate the '&&': you can
>> just write:
>>>> "test -f '%s/{}' && md5sum '%s/{}'" % (mydir, mydir)
>>>> , and xargs will pass that to the shell, and then the shell
>> will interpret the '&&' for you: you have shell=False in your
>> subprocess.Popen call, but the arguments to xargs are -I {}
>> sh -c "....", and this means that xargs ends up invoking the
>> shell (after replacing the {} with the name of a file).
>>>> Alternatively, you could translate it as:
>>>> "if [ -f '%s/{}' ]; then md5sum '%s/{}'; fi" % (mydir, mydir)
>>>> ; that might make the intent clearer to whoever gets to
>> maintain your code.
>> Yes to both points; turns out that my problem was in building up the
> command sequence to subprocess.Popen() - when to use, and not use,
> quotes etc. It has ended up as (spelled out in longhand...)
>>> xargsproc = ['xargs']
>> xargsproc.append('-I')
> xargsproc.append("{}")
>> xargsproc.append('sh')
> xargsproc.append('-c')
>> xargsproc.append("test -f %s/{} && md5sum %s/{}" % (mydir,
> mydir))

This will break if there are spaces in the file name, or other
characters meaningful to the shell. If you change if to
 xargsproc.append("test -f '%s/{}' && md5sum '%s/{}'"
 % (mydir, mydir))
, then it will only break if there are single quotes in the file name.
As I understand, your plan is to rewrite this bit in pure Python, to
get rid of any and all such problems.
> As usual, breaking it all down for the purposes of clarification has
> helpd a lot, as has your input. Thanks a lot.

You're welcome.
-- HansM