-
Notifications
You must be signed in to change notification settings - Fork 4.3k
In Apache Arrow Acero C++, group_by builds an internal aggregation map, while hash join builds an internal hash table.
How can we measure or track the memory usage specifically allocated for these two structures during execution?
Thank you!
All reactions
Replies: 1 comment 2 replies
It’s only supported in Arrow C++ today. You can pass a dedicated MemoryPool via ExecContext for the plan and query bytes_allocated() (and max_memory() for peak). This gives a plan‐level estimate because other operators also allocate from that pool. For hash‐table‐only memory, you’d need to route allocations for SwissJoin (and/or hash aggregate) to a separate pool, which currently requires finer‐grained plumbing or instrumentation.
All reactions
Thank you very much for your inside. I will follow your idea to test it. I am working with CPP, so I probably track a bit deeper into what acero offers.
All reactions
Great. Let me know if you have any further questions! Thanks.