Installing Custom Python Wheel Packages in a Databricks Serverless Notebook

Question

I have some custom functions and classes that I packaged as a Python wheel. I want to use them in my python notebook (with a .py extension) that runs on a serverless Databricks cluster.

I have read that it is not recommended to use %pip install directly on serverless cluster. Instead, dependencies should be managed through the environment configuration panel, which is located on the right-hand side of the notebook interface. However, this environment panel works when the notebook file has a .ipynb extension, not when it is a .py file.

Given this, is it recommended to use %pip install inside a .py file running on a serverless platform, or is there a better way to manage custom dependencies like Python wheels in this scenario?

Answer 1

You can manage Python dependencies for your notebook using the Environment side panel. There are two main ways to add them:

To add a dependency manually:

1. Open the Environment panel in your notebook.

2. Under Dependencies, click Add Dependency.

3. Enter the path to the dependency. You can use:

   • A standard package name (like in a requirements.txt file),

   • A .whl file,

   • Or a Python project folder (with pyproject.toml or setup.py).

   Path formats:

   • For workspace files: use an absolute path starting with /Workspace/.

   • For Unity Catalog volumes: use /Volumes/<catalog>/<schema>/<volume>/<path>.whl.

4. Click Apply to install the dependency and restart the Python environment.

enter image description here

For more details, refer to Configure the serverless environment

• Alexis Kan: Is this approach valid when running a job? I have hard time to define the databricks asset bundle using serverless with python wheel