-
-
Couldn't load subscription status.
- Fork 81
Introduce idr.Node caching and recycling
#84
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The key diff/improvement is now we add xpath to the csv reader so that we can perform both positive row selection as well as negative row skipping. Much more powerful than the old row skipping thingy. Next PR will bring the FileFormat implementation in, which will complete the csv fileformat.
As a result, adding `Release(*idr.Node)` to `FormatReader` so that `Ingester` will free up `*idr.Node` allocations for recycling Updated all benchmarks. No degradation so far. The reason the json sample bench mark didn't improve much is because there isn't much nodes allocated per read. However typically in the CSV parsing/transform scenarios, files are usually very long and node allocation caching saving would start to show significantly.
Codecov Report
@@ Coverage Diff @@ ## master #84 +/- ## =========================================== - Coverage 100.00% 99.62% -0.38% =========================================== Files 35 35 Lines 1552 1587 +35 =========================================== + Hits 1552 1581 +29 - Misses 0 6 +6
Continue to review full report at Codecov.
|
somehow the last commit
commit 601656371a3abd8e8810c56d1ca64fbde284439e (HEAD -> nodecache, origin/nodecache)
Author: jf-tech <jf.tech.llc@gmail.com>
Date: Sat Oct 10 10:38:09 2020 +1300
fix cov
refuses to show up.
seems github sync issue. will close and recreate the PR.
As a result, adding
Release(*idr.Node)toFormatReaderso thatIngesterwill free up*idr.Nodeallocations for recyclingUpdated all benchmarks.
No degradation so far. The reason the json sample bench mark didn't improve much is because there isn't much nodes allocated per read. However typically in the CSV parsing/transform scenarios, files are usually very long and node allocation caching saving would start to show significantly.
resolves #83