-
Notifications
You must be signed in to change notification settings - Fork 285
push down ORDER BY LIMIT clause on IVFFlat index to table_scan #22613
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
6b91f4c
to
ebd840d
Compare
PR Compliance Guide 🔍
Below is a summary of compliance checks for this PR:
No security concerns identified
No security vulnerabilities detected by AI analysis. Human verification advised for critical code.🎫 No ticket provided
- Create ticket/issue
Codebase context is not defined
Follow the guide to enable codebase context checks.
No custom compliance provided
Follow the guide to enable custom compliance check.
- Update
Compliance status legend
🟢 - Fully Compliant🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label
PR Code Suggestions ✨Explore these optional code suggestions:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The only question here is single thread can be faster than multi-thread when find centroids?
Uh oh!
There was an error while loading. Please reload this page.
User description
What type of PR is this?
Which issue(s) this PR fixes:
issue #22508
What this PR does / why we need it:
In particular query, number of copied rows from cache to pipeline is reduced from 13560 to 270, and throughput is improved by 10 times.
PR Type
Enhancement, Tests
Description
Push down ORDER BY LIMIT clause on vector index to table scan for IVFFlat
Update mock interfaces to support SetLimit method
Refactor IVFFlat search to use SQL-based distance computation
Update dependencies (StringZilla, SimSIMD)
Diagram Walkthrough
File Walkthrough
17 files
Add vector index ORDER BY LIMIT pushdown logic
Integrate pushdown optimization into query building
Update compilation to handle ORDER BY LIMIT pushdown
Set limit on data source readers
Add Limit field to Source struct
Refactor search to use SQL-based distance computation
Add metric type to distance function name mapping
Add BlockReadTopOp struct for limit operations
Implement SetLimit method for local data source
Add SetLimit to DataSource and Reader interfaces
Implement SetLimit for various reader types
Add SetLimit stub for remote data source
Add orderByLimit parameter to BlockDataRead function
Add SetLimit stub for table meta reader
Add SetLimit stub for sharding local reader
Add SetLimit stub for memory engine reader
Add SetLimit panic stub for backup delta source
1 files
Update mock interfaces with SetLimit and signature fixes
3 files
Update golang.org/x dependencies to newer versions
Update checksums for golang.org/x dependencies
Update StringZilla and SimSIMD library versions