Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Refinement of Providers into Providers and ExternProviders #2469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xizheyin wants to merge 3 commits into rust-lang:master
base: master
Choose a base branch
Loading
from xizheyin:rustc-query

Conversation

Copy link
Member

@xizheyin xizheyin commented Jun 15, 2025

Refined some vague concepts and pseudo-code in Query System

In the past, we only used vague concepts like local and extern to differentiate between different queries. however the code has more detailed Providers and ExternProviders to differentiate between them, so we should mention both of them in the documentation as well.

It is also important to indicate that rustc_middle::util::Providers is made up of Providers and ExternProviders in the query mod.

This PR also updates the example of registering a new Provider to consider ExternProviders.

r? @tshepang

tshepang reacted with heart emoji
@rustbot rustbot added the S-waiting-on-review Status: this PR is waiting for a reviewer to verify its content label Jun 15, 2025
@xizheyin xizheyin marked this pull request as draft June 15, 2025 12:47
@xizheyin xizheyin marked this pull request as ready for review June 15, 2025 12:52
xizheyin added 2 commits June 15, 2025 20:59 
Signed-off-by: xizheyin <xizheyin@smail.nju.edu.cn>
Signed-off-by: xizheyin <xizheyin@smail.nju.edu.cn>
Copy link
Member Author

first commit for ExternProviders, second commit for the nit in #2465.


#### Adding a new provider

Suppose you want to add a new query called `fubar`. You would:

1. Implement the provider function:
```rust,ignore
fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: DefId) -> Fubar<'tcx> { ... }
fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: LocalDefId) -> Fubar<'tcx> { ... }
Copy link
Member

@Noratrieb Noratrieb Jun 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and then make this a DefId

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tshepang suggest to make it LocalDefId. #2465 (comment)

I just went through the source code and realized that the key for local providers is pretty much LocalDefId. Which one are we going to choose?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only suggested that because I did not realize there is ExternProvider as well, which accepts DefId (and not LocalDefId)

either works for me... can always be adjusted later to mention both

 

Signed-off-by: xizheyin <xizheyin@smail.nju.edu.cn>
Comment on lines +177 to +217
### How queries interact with external crate metadata

When a query is made for an external crate (i.e., a dependency), the query system needs to load the information from that crate's metadata.
This is handled by the [`rustc_metadata` crate][rustc_metadata], which is responsible for decoding and providing the information stored in the `.rmeta` files.

The process works like this:

1. When a query is made, the query system first checks if the `DefId` refers to a local or external crate by checking if `def_id.krate == LOCAL_CRATE`.
This determines whether to use the local provider from [`Providers`][providers_struct] or the external provider from [`ExternProviders`][extern_providers_struct].

2. For external crates, the query system will look for a provider in the [`ExternProviders`][extern_providers_struct] struct.
The `rustc_metadata` crate registers these external providers through the `provide_extern` function in `rustc_metadata/src/rmeta/decoder/cstore_impl.rs`. Just like:
```rust
pub fn provide_extern(providers: &mut ExternProviders) {
providers.foo = |tcx, def_id| {
// Load and decode metadata for external crate
let cdata = CStore::from_tcx(tcx).get_crate_data(def_id.krate);
cdata.foo(def_id.index)
};
// Register other external providers...
}
```

3. The metadata is stored in a binary format in `.rmeta` files that contains pre-computed information about the external crate, such as types, function signatures, trait implementations, and other information needed by the compiler. When an external query is made, the `rustc_metadata` crate:
- Loads the `.rmeta` file for the external crate
- Decodes the metadata using the `Decodable` trait
- Returns the decoded information to the query system

This approach avoids recompiling external crates, allows for faster compilation of dependent crates, and enables incremental compilation to work across crate boundaries.
Here is a simplified example, when you call `tcx.type_of(def_id)` for a type defined in an external crate, the query system will:
1. Detect that the `def_id` refers to an external crate by checking `def_id.krate != LOCAL_CRATE`
2. Call the appropriate provider from `ExternProviders` which was registered by `rustc_metadata`
3. The provider will load and decode the type information from the external crate's metadata
4. Return the decoded type to the caller

This is why most `rustc_*` crates only need to provide local providers - the external providers are handled by the metadata system.
The only exception is when a crate needs to provide special handling for external queries, in which case it would implement both local and external providers.

[rustc_metadata]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/index.html
[providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.Providers.html
[extern_providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.ExternProviders.html
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read the code in rustc_metadata related to the extern providers and added it to the new commit, but I'm not sure there are parts I wrote correctly, if you have time to look at it. Thanks @Noratrieb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tshepang tshepang tshepang left review comments

@Noratrieb Noratrieb Noratrieb left review comments

Labels
S-waiting-on-review Status: this PR is waiting for a reviewer to verify its content
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /