-
Notifications
You must be signed in to change notification settings - Fork 3
fix(query-engine): account for CountMinSketchWithHeap memory usage #684
Open
Description
Parent
What to build
Make CountMinSketchWithHeap memory accounting reflect the actual allocation, including CMS cells, heap entries, and stored key strings. The estimate must use the backend’s actual cell representation rather than assuming f64 storage.
Acceptance criteria
- CMS matrix memory is included.
- Heap capacity and heap-entry metadata are included.
- Key storage is included or explicitly bounded.
- Accounting reflects the actual backend cell type.
- Tests verify accounting for different dimensions and heap sizes.
Blocked by
None - can start immediately.
Concrete example
The current estimate is effectively:
size_of::<CountMinSketchWithHeapAccumulator>() + row_num * col_num * size_of::<f64>()
For a 3x1024 sketch with heap capacity 1000, this omits the heap’s 1000 entries plus their key strings and metadata. It also models cells as f64 even though the heap backend stores integer counts.
The corrected estimate should either measure actual allocations or document a conservative bound that includes the heap and key storage.