Skip to content

Navigation Menu

Sign in
Sign up

Specify how to request an OpenCLI specification from a tool #46

Unanswered
cmcgee1024 asked this question in Ideas and suggestions
Discussion options

OpenCLI can become much more useful as a tool integration if there is a standard way to request from a tool its command structure. For example, a shell autocompletion system might not yet have knowledge of the tool, but if the user invokes it from the system path then it can request the OpenCLI specification from the tool and then provide assistance to the user. In order for workflows like this to work there needs to be a standard method to get the OpenCLI specification from the tool by invoking it.

There is an enormous number of command-line tools and scripts out there, and so the invocation mechanism will need to be unique enough to avoid most collisions with the universe of existing options. Also, the method of invoking the tool to query for an OpenCLI specification will need to be able to cause most tools that do not have OpenCLI specification support to exit with a non-zero exit code. Since there is a possibility of new versions of the specification in the future, this approach should take future versioning into account as well.

One potential approach is to use a globally unique option, specific to OpenCLI, such as --help-opencli, that is also intuitive to use. Future versions can become their own option --help-opencli<version> to permit tools to over different versions of the specification in the future.
You must be logged in to vote

Replies: 5 comments 15 replies

Comment options

Seconded. Completion generation (my main use case as a user) has a couple of quasi-standards, but in general it's a mess. Just in the half dozen or so of tools I install with mise, I got

  • $tool completion
  • $tool completions
  • $tool --generate-completion
  • $tool --gen-completions
  • $tool generate-shell-completion

Of course, OpenCLI would just add another quasi-standard, but I would at least expect to cover all adopting tools by one downstream script.

I'd propose to reserve $tool --[generate-]opencli and $tool [generate-]opencli, depending on whether it's a command-ey or option-ey CLI. That should fit in nicely with --version/version and --help/help which are basically standard.
You must be logged in to vote
0 replies
Comment options

The Swift Argument Parser project is currently discussing the future of an analogous feature to produce a JSON representation of a tool, called tool info, here: https://forums.swift.org/t/dropping-the-experimental-from-dump-help/82099/5

One suggestion in there is to logically extend the --help pseudo-standard using additional suffixes. For example, --help-hidden can show a variation of the help revealing hidden arguments. Dumping a format and version-specific representation of the help could be something like --help-dump-tool-info-v1.

Following this line of reasoning for OpenCLI might yield something like this:

<tool> --help-dump-opencli-v0.1
You must be logged in to vote
11 replies
Comment options

Proposal:

Tools MUST implement --help-opencli and MAY implement --help-opencli-v0.1 to respond to different versions of the spec.

The latter would be discoverable via the spec itself, wouldn't it? You'd be able to invoke --help-opencli and iterate through Document's options to discover --help-opencli-XYZ.
Comment options

Yes, I'm definitely thinking of clients that make runtime use of OpenCLI, and also those that are using it for discovery. The loose coupling of clients and tools is of particular interest to me.

I think we should also consider the possibility of low-fidelity clients, ones that are scripted using jq, and a bit of shell glue as an example. Perhaps they are querying a set of tools identifying some bit of criteria about their argument structure. At a point in time someone finds --help-opencli and scripts against what it returns. It appears to work fine at that version, maybe v0.1, and the script is committed somewhere. Then at some point in the future that same script runs with a different tool, or a newer version of the same one. It breaks inexplicably because the data no longer has the same shape that it once did. A key that was required is now optional, or has been renamed to something else. Other things appear to be in the same place. What's going on? The same flag isn't producing the same content anymore. There's no obvious way for me to put an assertion in the script to check and report that the tool isn't compatible with the script. Do I check for the presence/absence of this field that is no longer there? What if this happens again with something else? This seems brittle.

For federated tools, I expect there to be a related problem. I probably want to be able to support as many tools as possible, so I will support older and newer versions of the OpenCLI specification, and maybe also ToolInfo V0 and V1 (coming soon) from swift argument parser. There will probably be an order of preferred versions from newest supported to older legacy ones and a loop to find the first one that works. It would be a nightmare for me to not be certain what version that I'm getting back. Sure, I could probably store snapshots of the JSON schemas in my source code, and schema validate the payload to find the likely version. But, what if it validates against multiple schemas? Through which version should I interpret this data?

Originally, I was thinking of repeatedly invoking the tool with an asserted version for content negotiation. With tools written in C/Swift/Rust the time to get a response is probably going to be much faster and less resource intensive than making a REST/HTTP request over a network. But, I can see an argument that there is an overhead to doing repeated fork/exec calls, and managed runtimes can require more resources to get started. It makes sense to provide the client with a way to minimize this cost and still be able to get content that they are compatible.

Based on your proposal, and with a RESTful design trick:

Tools MUST implement --help-opencli with this top-level structure:

{
 "v0_1": ... complete version 0.1 content ...
 "vX_Y": ... complete version X.Y content ...
}

For each version X.Y of OpenCLI that a tool supports it MUST also implement --help-opencli-vX_Y that returns content based on that version of the OpenCLI schema.

I think that for low-fidelity clients, they either add the additional .vX_Y to their jq query based on the --help-opencli output that matches their expected content, or they use --help-opencli-vX_Y to simplify. A new version of OpenCLI comes out, but the script still works until tools (or argument parsing frameworks) start dropping legacy versions. In either case the error will be relating to a version-like identifier, and so any errors and problematic scripting code should be fairly self-explanatory.

Federated clients can make a single request of a tool using --help-opencli stream and unmarshal the content to the most suitable version. This minimizes overhead of potential multiple invocations of the same tool.

RESTful API's can sometimes have a top-level collection resource with details of the child resources, and allow direct access to the child resource using a child path. This approach might feel familiar to anyone who has worked with REST. It's don't think we will need a schema for the top-level object because of the simplicity of it, and it doesn't require that the OpenCLI Document maintain stability across versions for its top-level list of flags.
Comment options

@cmcgee1024 I like the idea of a version in the payload. For inspiration, OpenAPI uses this:

{
 "openapi": "3.0.2",

So using:

{
 "opencli": "0.0.1",

Seems like it would be a pretty decent step.
Comment options

@cmcgee1024, this particular proposal feels a bit awkward to me ... probably mainly b/c I struggle to think of prior art for a server responding to all the protocols it supports in one go:

{
 "v0_1": ... complete version 0.1 content ...
 "vX_Y": ... complete version X.Y content ...
}

I think I grasp your position, though, and am 100% aligned with you:

In order for workflows like this to work there needs to be a standard method to get the OpenCLI specification from the tool by invoking it.

Asserting the version helps to communicate clearly what the client is expecting ... It would be a nightmare for me to not be certain what version that I'm getting back.

(For my primary use-case, it would be a nightmare to not know which flag to invoke! 😰 — but that just restates your first point)

It seems to me like we've got a pretty clear path to supporting both:

  1. Tools must respond to --help-opencli with the latest version of the spec they support
    • as @alecthomas pointed out, this response already includes "opencli": "<VERSION>" which enables clients to interpret the response correctly
    • We could even add to the Document object a field adjacent to "opencli" that lists all the versions the tool supports ("opencli-variants" or "opencli-versions") to make it clear to clients which flags they may invoke.
  2. Tools must respond to --help-opencli-vX.Y for each version of the spec they support

Together, these rules allow the spec to evolve, clients to pin themselves to supported versions, and clients that understand multiple versions of the spec to gracefully degrade.

Questions for you:

  • Do you see low-fidelity (jq) clients invoking --help-opencli-vX.Y rather than --help-opencli?

  • Do you think we should specify the response a CLI gives if /--help-opencli-v(\d+).(\d+)/ is passed but the version isn't supported by the client?

    It might be helpful to differentiate "The flag was unrecognized" from "The version was unsupported". I have some work in this vein.

    We could also make version a value of the flag instead of a part of the flag name — e.g. --help-opencli and --help-opencli=v0.1 are both legal (this might be hard for some frameworks to support) or else we tools respond to --help-opencli as well as --help-opencli-version=0.1)
Comment options

I'm not quite seeing how --help-opencli-vX.Y is significantly better than reading the version from --help-opencli, to warrant the extra engineering overhead, which seems not insignificant to me. Having to generate N slightly different versions of the same spec just in case some client wants it seems like it would up being quite a burden on maintainers. I'd also say that it would be advantageous for the spec to start as simple as possible, and evolve from there if demand requires it.

IMO the initial language should be super simple:

Tools MUST implement --help-opencli

PS. I'm the creator and maintainer of the kong and kingpin CLI libraries from the Go ecosystem. I'd love to integrate OpenCLI support, it seems like it will be very useful.
Comment options

First pass at implementing this: #72

You must be logged in to vote
0 replies
Comment options

@boblail @alecthomas this has been a really engaging discussion. Thank you.

I have been playing around with jq and have changed my position a bit. I think that what I had in mind might actually be more difficult with what I was proposing than by putting the version attribute in the top-level object as you are proposing. Low-fidelity clients or scripts may or may not assert the version. If they break, then they'll quickly discover the version and can assert it with a more actionable message.

if .version != "0.1" then error("Expected OpenCLI version 0.1, found \(.version)") else <my query goes here> end

Next level might be to have the client negotiate what version that they want in the request. How will they discover how to negotiate version 0.1? Again, the payload of the flag that they do know can provide the hints.

{
 "version": "0.2",
 "variants": ["--help-opencli-v0.2", "--help-opencli-v0.1", "--help-opencli-all"],
 ... Remainder of the v0.2 content goes here ...
}

This can help a client to discover that they can negotiate a particular version of OpenCLI from this client with these flags, such as the v0.1 that they currently require.

For richer clients that support multiple OpenCLI versions, they might discover that there is an all variant that captures all of the available versions. This is done so that a client doesn't need to repeatedly invoke the tool in search of a common dialect. Languages like Go/Rust/Swift can offer fast startup times, but that's not always the case in all environments. Process overhead can be significant. The all variant would produce an array of OpenCLI metadata like this in one shot:

[
 {
 "version": "0.2",
 ... v0.2 content goes here ...
 },
 {
 "version": "0.1",
 ... v0.1 content goes here ...
 }
]

This would permit an average user to have knowledge of only one flag --help-opencli and discover all available variants and discover the versioning system.

I would limit the available variants to help and prevent proliferation of variants that aren't broadly available or expected. So, the variants would match the regex from @boblail (/--help-opencli-v(\d+).(\d+)/) and also the --help-opencli-all.

Do you see low-fidelity (jq) clients invoking --help-opencli-vX.Y rather than --help-opencli?

Yes, I hope that a low-fidelity client would discover this and be able to simplify their scripting logic, and enable content negotiation at the same time.

Do you think we should specify the response a CLI gives if /--help-opencli-v(\d+).(\d+)/ is passed but the version isn't supported by the client?

I think that from a low-fidelity client this distinction isn't all that helpful. Whether the version is not supported, or the tool doesn't even have OpenCLI support at all would probably be treated in the same way. Rich clients are either version knocking (not recommended), or using the --help-opencli-all, and in either case exit code non-zero is probably sufficient to indicate whether the response has something meaningful.

We could also make version a value of the flag instead of a part of the flag name — e.g. --help-opencli and --help-opencli=v0.1 are both legal (this might be hard for some frameworks to support) or else we tools respond to --help-opencli as well as --help-opencli-version=0.1)

This is a good question. We've been discussing this in context of the swift-argument-parser ToolInfo capability, which closely mirrors OpenCLI in a number of ways. An argument for longer flags sharing a prefix is that it might be simpler for a trivial shell script (ie. low-fidelity tools) to be able to detect the cases it supports and print a JSON heredoc. The argument for making these into options is that the human readable --help text for the entire OpenCLI description can be placed under one option making it less cluttered and also help to alleviate the need for the discoverability above. I wonder how various argument parsers handle options that may or may not require a value (--help-opencli and --help-opencli=vx.y). There are some tricks in SAP to be able to do something like this, but there are some caveats. We might be able to do something special if OpenCLI is integrated directly in the argument parser. Maybe with other frameworks it's not possible to distinguish options that aren't present, and those that are present but take the default value.
You must be logged in to vote
3 replies
Comment options

Hey @cmcgee1024!

I think you're proposing:

  • everything MUST implement --help-opencli?
  • everything MAY go on to implement --help-opencli-vx.y to support requesting an alternate version and --help-opencli-all to support requesting all supported versions?

LGTM! 🚀

I'd be inclined to defer complexity until it's necessary, but you've described a path that would allow OpenCLI to make breaking changes (e.g. between v1 and v2) and then give tools a way to support new and old clients — I think that's forward thinking and valuable to have in our back pocket!
Comment options

What would --help-opencli-all return? Some structured data on what versions are supported?
Comment options

I think this is over complicating it.

Tools that want to be discoverable:

  1. MUST provide the --help-opencli-v1 flag at the top command.
  2. --help-opencli-v1 MUST be 1.x compliant (which ever version they are using)

This should be fine because all 1.x versions will have to be backwards compatible to 1.0. So if a client ask for --help-opencli-v1 and gets a spec that is version 1.999 and they only understand 1.001, they should still be fine. They will not understand new things that were added but they were not going to understand them anyway. They will understand the parts of 1.999 that is still compliant with 1.001.

If and when we need breaking changes, we can move to --help-opencli-v2. But I don't see the compelling use-case for it now. We are defining a spec and not software. We should be concerned with making the spec backwards comparable.

And --help-opencli-v1 can return a pre-1.0 spec until things stabilize. or we can have a --help-opencli-v0 for the pre-stable period.
Comment options

Not sure if this thread is still active, but I had some thoughts.

I'm building a tool that would very much want to take advantage of this, so I love the idea.

But I had a couple thoughts:

  • As far as the option name goes: I think the priority must be to avoid collision. A tool builder should never reasonably want to use our term. I think this priority should trump wording elegance, typing ergonomics, or spec complexity. We would want any tool builder to be able to add it without worrying about conflicts with their current definitions.

    With that said, what we’re considering is right there on the edge. Maybe --help-opencli-spec would be better, since it might be even further from anything a tool maker would naturally choose?

  • A more philosophical consideration: opencli seems to me to be intended as a specification that could theoretically document any command line interface (cli). By putting something like --help-opencli in the spec, we would be expressing a strong preference toward POSIX/GNU-style conventions. Do we want to do that? Or do we want to stay neutral and not center around POSIX/GNU conventions? I’m not sure we’ve decided that yet.

  • With regards to versioning, I have a couple thoughts:

    • If we strongly plan and require that all future versions are backwards compatible with 1.0 (giving ourselves some grace for a few pre-1.0 versions), then the tool returning whatever version it was programmed to via --help-opencli should work for any downstream consumer. The consumer can parse it using the version they support and consume what they can. So if a tool outputs 1.3 and a consumer parses with a 1.0 parser, they may miss newer information but should still be able to consume what their 1.0 parser understands. (Maybe we could consider --help-opencli-v1 and plan on --help-opencli-v2 if we ever need to make breaking changes down the road)
    • I wonder if requiring or even strongly encouraging multi-version support would make this less palatable to tool makers. I don’t think anything discussed in this thread should be strictly required. I’m leaning more toward documenting these as best practices rather than mandates.
You must be logged in to vote
1 reply
Comment options

  • I wonder if requiring or even strongly encouraging multi-version support would make this less palatable to tool makers. I don’t think anything discussed in this thread should be strictly required. I’m leaning more toward documenting these as best practices rather than mandates.

I think so.

So if a tool outputs 1.3 and a consumer parses with a 1.0 parser, they may miss newer information but should still be able to consume what their 1.0 parser understands.

👍

(Maybe we could consider --help-opencli-v1 and plan on --help-opencli-v2 if we ever need to make breaking changes down the road)

This makes sense to me too — introducing versioned versions of this only when there are nonbackward-compatible post-1.0 changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

AltStyle によって変換されたページ (->オリジナル) /