Ambient "operator recommender" — predictive next-operator suggestions on the canvas #5240

@gupta-sahil01 Thanks for the diagram. A general principle we want to follow is to make each feature easy to develop and maintain. Please provide more details, such as: 1) any offline analysis needed; and 2) what information should be stored and where.

1. Offline analysis
   Three small text files we write once and update as the operator catalogue changes — no model training:

a short description of each operator (so the LLM knows what each one does)
a handful of example patterns ("after a Filter on text, suggest Sentiment") to guide ranking
a small test set of workflows to check ranking quality in CI

Adding a new operator just means adding one line to the descriptions file. CI catches it if you forget.
2) What's stored and where
Two places:

In the repo (versioned with code, reviewed in PRs): the three files above, plus the LLM prompt templates. That's the entire "knowledge base."
In memory at runtime (auto-evicting cache): recent suggestion results, and one-time analyses of any UDF source code we've seen. Caches a recent UDF summary so we don't pay the LLM to re-read the same Python twice.

No database, no model registry, no separate service. Four files in the repo + a cache.

I like a stateless backend api just takes in the current state and returns recommendations. Stateful things are harder to get right.

Back on to how this can be implemented, I'm mostly leaning towards LLM based recommendation. I'm not a huge fan of training a custom model for recommendations, if we use LLMs, we get a free ride as model capabilities increase.

My main concern is that if the speed of LLMs are fast enough, also if we bring LLMs to the table, we might need to maintain the conversation history somewhere for it to be more cache friendly.

I would really recommend looking into how cursor does code autocomplete. I know that cursor trains its own small mod (I think it's a small LLM model) and it's both fast and accurate. If we can get some more context on how cursor's autocomplete work today I'll be more comfortable. Also I would really like to see if there are any online articles discussing using LLMs to run code autocomplete.

Thanks for your valuable advice. I'll surely look into the auto-completion of code in cursor. That looks like something very similar to what I've proposed here

I'm still looking into the auto code completion of cursor. Right now, from what I've understood, is that cursor uses a small custom-trained model (not GPT or Claude) fine-tuned specifically for code completion, with a ~272k-token context window over your current file, open tabs, and recent edits.

There's a neat trick called "speculative edits", which means that they feed your existing code back to the model as draft tokens, so it only emits the bits that change, about a ×ばつ speedup, which is what lets full-file rewrites feel instant.

Hi @gupta-sahil01 thanks for looking into cursor, I think a custom-trained model is not very feasible for us, so for this one we should still stick to openly avaiable models. But cursor uses a smaller model for this task, so we should also consider using less powerful models, which are both cheaper and possibly faster. With this in mind I would suggest we proceed to next steps and do not stuck in researching for too long. Thanks. What would be a sensible next step?

Sensible next step: I'll open a small end-to-end PR that wires up a stateless /recommend endpoint in the existing agent-service (reusing the compile API for schemas, the operator catalog, and the same AI SDK + LiteLLM gateway the chatbot already uses), and a frontend hook that listens for "operator added" and feeds suggestions into the existing ghost-operator UI from drag-drop. First version returns hardcoded suggestions so we can confirm the wiring works; second version swaps in a real LLM call with a small/cheap model.

Would that be ok?

@gupta-sahil01 the plan sounds good, please go ahead with first steps so that we'll be able to see the API shape, thanks!

I have a question. How do I start building? Do I just show the UI part with the API calling, because for that I'll need credits, right? I have finished setting up all the microservices and frontend, and it's running successfully on my laptop. Should I set up the texera agent as well?

Hey, any help would be appreciated.

Hey, appreciate your thoughts. I've opened an issue to track this - #6293

@chenlica Professor it's already disabled by default. We need to enable it through the terminal at every layer.

So it's opt-in. Either I enable it in the terminal or I enable it in gui.conf. It's an env variable called GUI_WORKFLOW_WORKSPACE_OPERATOR_RECOMMENDATION_ENABLED

Thanks for the clarification.

I'm currently working on the PR reviewers' comments. I will start working on the second phase immediately after approval