[WIP] Python Notebook → Texera Workflow migration tool using LLMs #5959

As decided in our last monthly Apache meeting, we're using Discussions to be more transparent with the community about what we're working on. This is one of those posts: an early look at a feature still in progress, shared so anyone interested can follow along.

We're building a tool that turns an existing Jupyter notebook into a Texera workflow using an LLM. This post is an early, transparent look at the feature while it is still in progress, so the community knows what's coming.

Tracking issue: #4301

The problem

Users who already have working Python code outside of Texera have to rebuild it as a workflow from scratch. For non-trivial notebooks that is slow and tedious, and it discourages people from moving existing work onto Texera.

What the tool does

You upload a Jupyter notebook, pick a model, and the tool generates a Texera workflow for you:

• The notebook is sent to an LLM, which produces a workflow of UDF operators and the edges between them.
• The generated workflow opens in the workspace alongside the original notebook, embedded in a panel.
• Cells and operators are linked both ways. Click an operator and the source cell that produced it is highlighted; click a cell and the corresponding operator is highlighted. This makes it easy to see what came from where and to edit with confidence.
• The notebook stays associated with the workflow. Reopen the workflow later and the notebook comes back with it.

The generated workflow is a starting point, not a final answer. You edit it like any other workflow.

How it fits together (high level)

• A feature flag gates the whole tool, so it has no effect until it is turned on.
• A notebook-migration microservice mediates between Texera and a local JupyterLab instance embedded in the workspace.
• A frontend LLM session drives the conversion against the existing LiteLLM proxy, using a model you choose and an API key you provide.
• An orchestration service and a set of UI components (the import modal, the loading overlay, the embedded notebook panel, and the cell↔operator highlighting) tie it together.

Status: work in progress

This is being merged into main incrementally as a series of PRs rather than one large drop. The feature is behind a flag and off by default, so nothing changes for existing users while we build it out.

The tracking issue, #4301, is the best place to follow along. It has demo videos and architecture diagrams of the tool, along with links to every sub-issue and PR that makes up the feature.

Join the conversation

If you have any thoughts, questions, or feedback on the project, we'd love to hear them. Feel free to use this post as an open floor to talk about anything related to the feature!