[CI] CUDA 13.3 pip devcontainer builds fail during CMake configuration #8559

Summary

The CUDA 13.3 pip devcontainer jobs fail during CMake configuration when the build reaches an import cuda.tile check.

Failing test/component: CUDA 13.3 pip devcontainer build

Example failures:

• Update Dask RF to use the new distributed algo #8466
  • amd64: https://github.com/NVIDIA/cuml/actions/runs/33825593157/job/100877469580
  • arm64: https://github.com/NVIDIA/cuml/actions/runs/33825593157/job/100877469643
• Use latest intersphinx inventories on main #8558
  • amd64: https://github.com/NVIDIA/cuml/actions/runs/33858605010/job/100977618161
  • arm64: https://github.com/NVIDIA/cuml/actions/runs/33858605010/job/100977618041

Test Details

• Test file: build/pip/cuda-13.3/release/_deps/cuvs-src/cpp/cmake/modules/generate_cutile_kernels.cmake:61

• Test name: devcontainer / {amd64, arm64}, 13.3, pip

• Error message:

  -- CUML_CPP: Build and statically link CUVS libraries: OFF
  -- CPM: Adding package cuvs@26.10.00 (main)
  CMake Error at build/pip/cuda-13.3/release/_deps/cuvs-src/cpp/cmake/modules/generate_cutile_kernels.cmake:61 (message):
   cuda.tile (cuTile Python) is required to build cuTile embedded kernels.
   Install cutile-python and cuda-tileiras (conda), or cuda-tile[tileiras]
   (pip).
   Interpreter: /home/coder/.local/share/venvs/rapids/bin/python3.14
   Import error: Traceback (most recent call last):
   File "<string>", line 1, in <module>
   import cuda.tile
   ModuleNotFoundError: No module named 'cuda.tile'
  

The same error occurs in all four example jobs before compilation starts. CMake reports CUML_CPP: Build and statically link CUVS libraries: OFF before adding cuvs@26.10.00 (main).

Related: #8520 tracks pip devcontainer builds fetching and building cuVS source instead of using the installed package.

Root Cause Analysis