-
Notifications
You must be signed in to change notification settings - Fork 419
feat(mito): Optimize async index building with priority-based batching #7034
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(mito): Optimize async index building with priority-based batching #7034
Conversation
Signed-off-by: SNC123 <sinhco@outlook.com>
Signed-off-by: SNC123 <sinhco@outlook.com>
Signed-off-by: SNC123 <sinhco@outlook.com>
Signed-off-by: SNC123 <sinhco@outlook.com>
Signed-off-by: SNC123 <sinhco@outlook.com>
Signed-off-by: SNC123 <sinhco@outlook.com>
552a39d to
3824c30
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Duplicate processing with the outer
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And why not call on_failure?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe mistake when rebase... 🐁
Will fix later.
Uh oh!
There was an error while loading. Please reload this page.
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
#6756 (Tracking Issue)
What's changed and what's your intention?
This PR aims to optimize the asynchronous index building process in the Mito engine. The main intention is to improve efficiency and resource management when building indexes for SST files.
The key changes include:
IndexBuildSchedulerthat groups index build tasks into batches and processes them based on priority. This prevents the system from being overwhelmed by numerous small tasks and allows for more efficient resource utilization.index_file_idto FileMeta. This allows an index file to have a different ID from its corresponding data (SST) file, which is crucial for the async build process where an index is created after the SST file already exists. This change has been propagated through read, write, and delete paths.Related Disscussion: feat: introduce IndexBuildTask for async index build #6927 (comment)
PR Checklist
Please convert it to a draft if some of the following conditions are not met.