The index interaction happens in 2 places:

• The bazel downloader in the pip.parse extension evaluation is using it to query the index and download the packages. The first error is most likely because the value we are using is from the pip.parse attribute - should we use the one from the file?
• If you are building from sdist, in the whl_library. It seems that this is the second error that you are getting.

EDIT: If you are content with the answer feel free to convert the issue to a discussion (the best place for questions in general), otherwise we can groom this into a feature/bug ticket. :)

The first error is most likely because the value we are using is from the pip.parse attribute - should we use the one from the file?

Hard to say. My initial reaction was to say "no, we should use the value from pip.parse.experimental_*" but I don't have any strong reasoning for that.

If you are building from sdist, in the whl_library. It seems that this is the second error that you are getting.

The nurpc thing being downloaded from the private index is already a wheel, so it shouldn't be building from sdist.

Image

convert the issue to a discussion

Ah, thanks! I wasn't aware we used Discussions (or maybe I forgot...). Done!

Is this still a problem?

There's a TL;DR proposal summary at the end.

Well it doesn't impact us but that's because I've been operating in the "just make sure that the index url in requirements lock file matches that defined in experimental_index_url" mode, but I still don't fully understand where the canonical source is.

I think the ideal situation is that there is only a single place where index URLs are set. It seems like pip.parse is logical place for this, especially with the experimental Bazel downloader stuff. Any time rules_python downloads things from an index, including building a wheel from sdist in whl_library, we should use the bazel downloader (and thus gets to use the credential helper).

This brings up an issue though - some workflows use requirements.in which contains --(extra-)index-url args and generate the lock file with compile_pip_requirements. This happens before pip.parse sets index URLs and thus we have to use the index urls set in requirements.in.

Maybe we make the requirements lock file the source of index URLs for pip.parse instead of setting them manually with experimental_(extra_)index_url(s)? So an API like:

pip.parse(
 experimental_bazel_downloader = True,
 experimental_index_url_overrides = {"package": "https//baz.com/simple", ...},
 requirements_lock = "requirements.txt",
 ...

If requirements.txt has:

--index_url https://foo/simple
--extra-index-url https://bar/simple
--extra-index-url https://foobar/simple
package==1.2.3 \
 --hash=sha256:abcd1234

The experimental_bazel_downloader = True API would be functionally equivalent to:

pip.parse(
 experimental_index_url = "https://foo/simple",
 experimental_extra_index_urls = ["https://bar/simple", "https://foobar/simple"]
 experimental_index_url_overrides = {"package": "https//baz.com/simple", ...}, # note no "oauth2accesstoken"
 requirements_lock = "requirements.txt",
 ...

The net result of such a change is that the requirements.in file is now the canonical source for index URLs no matter how the lock file is made:

flowchart TD
 A[req.in] --> B[pip compile<br>uv pip compile]
 A --> C[compile_pip_requirements]
 B --> D[req.lock]
 C --> D
 D --> |effectively sets| E[pip.parse.experimental_index_url<br>pip.parse.experimental_extra_index_urls]

Proposal summary:

1. Remove experimental_index_url, experimental_extra_index_urls
2. Add boolean experimental_bazel_downloader (or similar wording)
3. Have pip.parse pull index URLs from the requirements lock file. If not present, default to public PyPI.

I think keeping experimental_index_url_overrides is prudent.

Alternative:

1. Keep experimental_index_url, experimental_extra_index_urls but support special options like DEFER or similar wording that causes pip.parse to pull index URLs from the requirements lock file.