homepage

I have a program that runs something like the following:
$ hack run -- :target --arg1 --arg2 arg3 arg4
This basically runs a program identified by :target with the args. However, I cannot pass "--" to the program. For example, if I type:
$ hack run -- :hack run -- :target clean --help
the second "--" is swallowed by the parser, and I get an the help for "hack run" instead of instead of "hack clean". The run subcommand just does the following:
all_args = [target.bin_path] + args.args
os.execv(target.bin_path, all_args)
However, the first hack run has the following list for args:
args = Namespace(args=['run', ':hack', 'clean', '--help'], func=<function do_run at 0x19c3e60>, target=':hack')
Where is the second "--"? I would have expected the args list to be:
args=['run', '--', ':hack', 'clean', '--help']
About the python version, I am using python 2.6. However, I am using the latest release of argparse from [1] and am assuming that it's very similar code.
[1]http://code.google.com/p/argparse/downloads/list 

I wanted to include a minimal example script, so here it is. If you run this like so:
$ ./test.py -- blah -- blah2
you will get the the following output:
Namespace(args=['blah2'], target=['blah'])
I would have expected the following instead:
Namespace(args=['--', 'blah2'], target=['blah'])

It doesn't look like that file got included last time, so here's a second try.

Hah.. was just about to report this. I'm in the midst of converting a bunch of scripts from optparse to argparse, and this is one of the problems I'm hitting.

See nargs=argparse.REMAINDER for an approach that doesn't require that first '--':
http://docs.python.org/library/argparse.html#nargs
But yeah, removing more than one '--' is probably a bug. Fixing it would be a little backwards incompatible though. Is it possible that anyone actually depends on the behavior of consuming multiple '--' arguments?

Using argparse.REMAINDER will not help for my problem in my real program as the first arg needs to be handled by my program and mapped into the real binary name.
If I recall correctly from memory, the following is what happened when I tried using argparse.REMAINDER. If call my program like so:
$ hack run :target --help
The subcommand will be "run" in this case. Because :target is handled by argparse, the "--help" will not be seen as part of the remainder of the arguments, and I will get the help for the "hack run" subcommand instead of the target binary getting the --help argument. I have pushed most of the program to [1] if you want to take a look. Specifically, see cli/commands/run.py:do_run for that bit of code that handles the run subcommand.
[1]https://github.com/wt/repo-digg-dev-hackbuilder 

I think it is unlikely that anyone depends on argparse consuming multiple -- strings. If you are worried about it we could restrict the change to 3.3. But personally I think this would be OK for a bug fix.

Ok, I agree - I'm fine with it as a bugfix. Depending on the removal of extra -- strings would be pretty crazy anyway. ;-)

I don't know that this is a bug. Rather, the string '--' means different things to argparse and optparse. In argparse, '--' is a psuedo-argument taken to mean "everything after this is a postional argument" and not "stop processing arguments", which is the optparse meaning. In that context it doesn't seem like removing additional '--' is a bug in argparse, since additional '--' would merely be restating the same thing.

No, it is definitely a bug. It prevents implementing parsers that pass strings on to another sub-parser or command. Imagine, for example, implementing a script that takes some arguments, but takes the entire rest of the command string and passes it to some unix program that it calls...and that unix program will recognize '--' in that string as the end of processing *its* options, and the user of the python command may well want to do that. Unless this bug is fixed, it would be impossible to use argparse to implement such a reasonable scenario.

I agree with David. It's a bug. I have programs (not using argparse yet) that do exactly what he describes.

In that case, wouldn't you use 'parse_known_args' instead of 'parse_args'
and pass the remaining arguments to the next script? This case is
explicitly mentioned in the argparse documentation. Again it seems to me
that the meaning of '--' has changed slightly between optparse and
argparse. Whether or not that was correct or intended is perhaps another
isssue.
On Sun, Mar 18, 2012 at 1:32 PM, Eric V. Smith <report@bugs.python.org>wrote:
>
> Eric V. Smith <eric@trueblade.com> added the comment:
>
> I agree with David. It's a bug. I have programs (not using argparse yet)
> that do exactly what he describes.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue13922>
> _______________________________________
>

No. parse_known_args assumes you have known and unknown args intermixed. 
Going back to the original example, what if "hack" and ":target" had overlapping parameter names (say, they both supported "--arg1")? I think parse_known_args would pick up all instances of "--arg1". I really want "--" to mean "treat everything else as non-optional arguments".
Whether that's the original intent, I don't know. But as David says, unless it is, it's impossible to implement such a scheme with argparse.

> It prevents implementing parsers that pass strings on to another 
> sub-parser or command.
...
> wouldn't you use 'parse_known_args' instead of 'parse_args'
> and pass the remaining arguments to the next script
I'll just say again that the recommended way of passing arguments to another command is via nargs=argparse.REMAINDER. (Though you may still need "--" if the first argument in the remainder is a flag shared by your parser, as Warren Turka pointed out.)
> I really want "--" to mean "treat everything else as non-optional 
> arguments"
Yep, and that's what it's intended to mean, and what the documentation says:
"you can insert the pseudo-argument '--' which tells parse_args() that everything after that is a positional argument"
http://docs.python.org/library/argparse.html#arguments-containing
It's therefore a bug if there's a '--' after the first '--', but it's not being treated as a positional argument.

Added patch so that only the first '--' is removed by an argparse.PARSE or argparse.REMAINDER argument. Note that, as Steven said, argparse.REMAINDER should be used in the OP's issue (and the added test makes sure all remaining arguments are preserved even if they appear in the parent parser).

This patch looks like the right fix to me. The tests look good too.

New changeset 18b114be013e by R David Murray in branch '3.2':
#13922: argparse no longer incorrectly strips '--' after the first one.
http://hg.python.org/cpython/rev/18b114be013e
New changeset bd2c167dfabc by R David Murray in branch '2.7':
#13922: argparse no longer incorrectly strips '--' after the first one.
http://hg.python.org/cpython/rev/bd2c167dfabc
New changeset a636f365d815 by R David Murray in branch 'default':
Merge #13922: argparse no longer incorrectly strips '--' after the first one.
http://hg.python.org/cpython/rev/a636f365d815 

Committed. Thanks, Jeff. By the way, although this patch isn't big enough to require it, it would be great if you would submit a contributor agreement: http://www.python.org/psf/contrib.

Thanks for fixing this issue. You guys are great!
wt
On Sat, Jul 21, 2012 at 8:00 PM, R. David Murray <report@bugs.python.org>wrote:
>
> R. David Murray <rdmurray@bitdance.com> added the comment:
>
> Committed. Thanks, Jeff. By the way, although this patch isn't big
> enough to require it, it would be great if you would submit a contributor
> agreement: http://www.python.org/psf/contrib.
>
> ----------
> resolution: -> fixed
> stage: -> committed/rejected
> status: open -> closed
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue13922>
> _______________________________________
>

This patch introduced a regression. Before, parse_args would take a tuple as an argument, and in _get_values it was converted to a list via list comprehension, which meant it was working with tuples too. In the current version, that raises an AttributeError, since tuples do not have .remove().
The simplest solution would be to convert arg_strings to a list before calling .remove(). The downside is that it will break silently when you pass it a string, whereas now you get a confusing error message (but, this is the same behavior as before this fix)

@gcbirzan: Could you please open up a new issue? The current issue is fixed - it's just that the fix caused a new issue.
I would say that the `args` parameter was never intended to be anything but a list, so currently there's a documentation bug since that isn't stated explicitly anywhere. (But certainly the tests only test for lists, not any other sequences.)
Adding support for stuff other than lists feels more like a feature request, but given that it apparently worked by accident before, I guess we need to treat it as a bug. Probably the fix for that bug needs to have a ton of tests that check whether tuples work in all the various ways that parse_args is used.

New changeset 5d8454fcc629 by R David Murray in branch '3.2':
#15847: allow args to be a tuple in parse_args
http://hg.python.org/cpython/rev/5d8454fcc629
New changeset 76655483c5c8 by R David Murray in branch 'default':
merge #15847: allow args to be a tuple in parse_args
http://hg.python.org/cpython/rev/76655483c5c8
New changeset a2147bbf7868 by R David Murray in branch '2.7':
#15847: allow args to be a tuple in parse_args
http://hg.python.org/cpython/rev/a2147bbf7868 

There are several problems with the patch provided in msg156315
This description:
"Added patch so that only the first '--' is removed by an argparse.PARSE or argparse.REMAINDER argument."
should read
"Added patch so that only the first '--' is removed by arguments that are not argparse.PARSER or argparse.REMAINDER ."
The test that adds a third subparser with a nargs='...' argument does not test this change. It exercises both nargs types that are not affected by the change in argparse.py. As such older versions of argparse pass this test.
I thinking argparse.py change is correct (delete only the 1st '--'), though I'm not sure it addresses the concern of the original poster.

There's another 'feature' to the patch proposed here. It only deletes the first '--' in the list of strings passed to '_get_values' for a particular action.
 parser = argparse.ArgumentParser()
 parser.add_argument('foo')
 parser.add_argument('bar', nargs='*')
 print(parser.parse_args('-- 1 -- 2 3 4'.split(' ')))
 # Namespace(bar=['2', '3', '4'], foo='1')
'_get_values' first gets ('foo',['--','1']), then ('bar',['--','2','3','4'])
 print(parser.parse_args('-- -- 1 -- 2 -- 3 4'.split(' ')))
 # with this '1st only' change
 # Namespace(bar=['1', '2', '--', '3', '4'], foo='--')
 # without it, deleting all '--'; note foo is empty
 # Namespace(bar=['1', '2', '3', '4'], foo=[])
And to confuse things a bit more:
 print(parser.parse_args('1 -- 2 3 4'.split(' ')))
 # Namespace(bar=['2', '3', '4'], foo='1')
passes ['1','--'] with 'foo'
If 'bar' nargs='...', bar gets all of the '--' (with or without this patch).
The handling of '--' is complex because it is used in one place to mean, 'everything else is an argument', effectively adding '-AA...' to the arg_strings_pattern. It also matches with the nargs_pattern (e.g. '(-*A-*)...'). And then it may or may not be removed in _get_values().

I am working on an alternative solution that moves the '--' removal to the consume_positionals() method, and only does it if there is a corresponding '-' in the arg_strings_pattern.

This patch removes only one '--', the one that put a '-' in the 'arg_strings_pattern'. It does this in 'consume_positionals' right before calling 'take_action'. As before it does not do this if nargs is PARSER or REMAINDER.
test_argparse.py has two DoubleDashRemoval cases, that attempt to highlight the changes from production (delete all --) and development (delete first -- in each positional group) versions.
I have not made any changes to the documentation. All it says now is:
"If you have positional arguments that must begin with - and don’t look like negative numbers, you can insert the pseudo-argument '--' which tells parse_args() that everything after that is a positional argument:"

http://bugs.python.org/issue9571 proposes a refinement to this patch - drop the '--' from PARSER and REMAINDER nargs if it is the 1st string.