-
Notifications
You must be signed in to change notification settings - Fork 179
Cortex.cpp: CLI #1109
-
Goal
- Consistent syntax
- Command chains for easier DX inference, e.g.
cortex run
CLI Syntax
- Method names
- Follows Docker when ever possible:
cortex models pull - Otherwise follows OpenAI API & CRUD naming:
GETDELETELISTUPDATE - ✅
cortex models delete - ❌
cortex models remove
- The order of
methodvsrequired parameters
- ✅
cortex engines uninstall <engine_id> - ❌
cortex engines <engine_id> uninstall
- Required vs optional variables
- Required variables don't have a flag
- ✅
cortex engines uninstall <engine_id> - ❌
cortex engines uninstall -e <engine_id> - ✅
cortex engines uninstall <engine_id> -x <OPTIONAL_VAR>
Suggestion:
cortex METHOD CMD <REQUIRED_VARS> -F <OPTIONAL_VARS>
Command Chains
cortex run <model_id>: what sequence of calls does it make?- What other chaining do we have?
- Do we automatically start engines based on the model?
Implementation
- https://github.com/CLIUtils/CLI11 can be a good library: header only, no dependencies, c++11.
- feat:
cortex -v#1066 - feat: add back triton features #29
- docs: Add developer + installation documents #31
- feat: batch inference for nitro #41
- feat:
cortex models get <MODEL_ID>andcortex models list#1075 - feat:
cortex models delete#1064 - epic:
cortex models update <MODEL_ID>#1060 - feat: minimal deployment docker image #32
- API with llama.cpp for embedding #37
- feat:
cortex run model(:gguf)#1076 - feat:
cortex engines list#1074 - feat:
cortex engines get <engine_id>#1073 - epic:
cortex enginescommands #1072 - ps
- embeddings
- uninstall
- epic:
cortex updatecommand #1087
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 2 comments 6 replies
-
What is the syntax convention for our CLI?
CLI Syntax
- Method names
- Follows OpenAI API:
cortex models get- Follows Docker:
cortex models pull
- The order of
methodvsrequired parameters
cortex models set <param>cortex models <param> setCommand Chains
cortex run <model_id>: what sequence of calls does it make?- What other chaining do we have?
- Do we automatically start engines based on the model?
CLI Syntax
- Method names
- Follows Docker:
cortex models pull - We also support
cortex models getto get local model information that we have already pulled
- The order of method vs required parameters
cortex models set <param>
Command Chains
cortex run <model_id>: what sequence of calls does it make?
If model does not exist, pull model:cortex models pull <model_id>
If engine does not exist, pull engine:cortex engines <engine_name> install
Start model:cortex models start <model_id>
Start chat:cortex chat <model_id> -m <msg>- What other chaining do we have?
We only haveruncommand - Do we automatically start engines based on the model?
Yes, we do
Beta Was this translation helpful? Give feedback.
All reactions
-
- Can we make this syntax a bit more inconsistent:
- not:
cortex engines <engine_name> install - correct: cortex models start <model_id>
-
Can we decide on syntax around required vs optional flags, e.g. when do we expect users to use a flag, vs not?
-
I see this getting conflated as well
getvspullvsinstall. Was the decision:
- pull: just for models, docker like
- get: available for most methods, similar to a local api /get request
- install: for when user is installing something from remote/internet?
Beta Was this translation helpful? Give feedback.
All reactions
-
- Yes - I think we should follow the syntax that @dan-homebrew suggested
cortex <feature> <command> <subject>
cortex engines install <engine>
cc: @namchuai
2. All flags are optional for now (except cortex -v), I think
--verbosefor logging--versionspecify a version to download--messageto append message to chat command
- It is correct.
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 1
-
Maybe we can use a specific example:
Usage: cortex-nightly chat [OPTIONS] [model_id]
Positionals:
model_id TEXT
Options:
-h,--help Print this help message and exit
-m,--message TEXT Message to chat with model
Confusing
- model id is required
- message is not
- Request fails if message is not provided
OAI indicates both params are required.
image
So should the proper syntax actually be:
cortex chat <model_id> <message> [-options]ORcortex chat [-options] <model_id> <message>
cc @dan-homebrew
Beta Was this translation helpful? Give feedback.
All reactions
-
@vansangpfiev Quick check: does the CLI library we use auto-generate the "UI" (e.g. with helper flags, etc)?
@0xSage These are standard CLI library features, I wouldn't over-litigate this - there's best practices out there already
Beta Was this translation helpful? Give feedback.
All reactions
-
@dan-homebrew Yes - The "UI" is autogenerated by the CLI library
@0xSage The message is mandatory in the request body. But we are talking about the CLI, so if the message is not specified, can we just let user inputs it later? Something like:
.\cortex.exe models chat tinyllama
Inorder to exit, type `exit()`
> // input message here
The syntax should be:
cortex chat <model_id> -m <message>ORcortex chat -m <message> <model_id>
Beta Was this translation helpful? Give feedback.
All reactions
-
-
Can we make
cortex==cortex -h, instead of starting a server.
Runningdockeris equivalent todocker --help
Otherwise it's too jarring, and a bad onboarding experience for new devs.
@dan-homebrew -
This means most subcommands, e.g.
cortex models [no command]can also bring up the helper. Nice DX. -
Can I recommend we separate & group the command list when users run
cortex [-h]
Would really help readability.
Example:
> cortex Options: -h,--help Print this help message and exit --verbose Verbose logging -v, —version Cortex version Shortcuts: pull Download a model by URL (or HuggingFace Repo ID) run Start a model and interactive chat shell Commands: chat Sends a chat completion request models Subcommands for managing models embeddings Subcommands for generating embeddings engines Subcommands for managing inference engines API Server Commands: ps Show running models and their status start Start the API Server stop Stop the API server
Beta Was this translation helpful? Give feedback.
All reactions
-
Yes, this is straightforward and can be implemented in the final days as finishing touches
We can also put in the "Cortex" ASCII text
I am adding this in as a tail end task for Cortex v0.1
Beta Was this translation helpful? Give feedback.