Skip to content

Navigation Menu

Sign in
Sign up

Dump load - #15

Open
yegct wants to merge 1 commit into
peterc:main from
yegct:dump_load
Open

Dump load #15
yegct wants to merge 1 commit into
peterc:main from
yegct:dump_load

Conversation

@yegct

@yegct yegct commented Apr 13, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

This PR allows BitArray to be dumped to disk and subsequently loaded from disk. It also adds the ability to union two bitarrays. And there's a new class, BitArrayFile, allowing for read-only access to a dumped BitArray, providing a means to query the array without having to load it in to RAM.

Tests

$ ruby test/test_bit_array.rb 
Run options: --seed 52069
# Running:
.................
Finished in 0.007989s, 2127.8684 runs/s, 527461.0175 assertions/s.
17 runs, 4214 assertions, 0 failures, 0 errors, 0 skips
$ ruby test/test_bit_array_file.rb 
Run options: --seed 36015
# Running:
....
Finished in 0.003689s, 1084.2876 runs/s, 37950.0675 assertions/s.
4 runs, 140 assertions, 0 failures, 0 errors, 0 skips

Additionally, I tested this with a bitarray consisting of a little over 2 billion bits. I was able to dump, load, and use BitArrayFile to query the bits.

Comment thread lib/bitarray/bit_array.rb
attr_reader :field, :reverse_byte, :size
include Enumerable

VERSION = "1.4.0"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed from 1.3.0.

Comment thread lib/bitarray/bit_array.rb
include Enumerable

VERSION = "1.4.0"
HEADER_LENGTH = 8 + 1 # QC (@size, @reverse_byte)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't use this here, but need it for BitArrayFile.

Comment thread lib/bitarray/bit_array.rb

def ==(rhs)
@size == rhs.size && @reverse_byte == rhs.reverse_byte && @field == rhs.field
end

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the ability to compare two BitArray objects. This makes it a bit easier to add tests.

Comment thread lib/bitarray/bit_array.rb
end

combined
end

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote this because it allows parallelising bloom filter creation. I don't personally need this method, but figured it might be useful for others.

Comment thread lib/bitarray/bit_array.rb
private def byte_position(position)
@reverse_byte ? position : 7 - position
end

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following two methods, dump and load, are new.

private def seek_to(position)
@io.seek(position + HEADER_LENGTH)
end
end

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's actually fairly easy to implement the rest of BitArray, allowing this to be read/write instead of read-only. But it's quite a bit slower, at least for large bitarrays and nobody's likely to care for small bitarrays. Seemed to be unnecessary bloat.

@@ -0,0 +1,65 @@
require "minitest/autorun"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that I'm more familiar with rspec. Please excuse me if these tests aren't how most people would write Minitest tests.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a way, Minitest is to RSpec as Sinatra is to Rails. Minitest is far less opinionated and freeform which has upsides and downsides, but a big upside is pretty much any approach that works and fits into a project's own style is all good! :)

@yegct yegct mentioned this pull request Apr 13, 2022

peterc commented May 3, 2022

Copy link
Copy Markdown
Owner

I've merged the previous one but this will take a little bit more review - thanks for now though!

yegct force-pushed the dump_load branch 2 times, most recently from 73f65aa to 9c070df Compare May 3, 2022 20:37

yegct commented May 3, 2022

Copy link
Copy Markdown
Contributor Author

I've merged the previous one but this will take a little bit more review - thanks for now though!

Rebased off of main. If you don't like/want the union, let me know and I'll drop that (and associated tests). If you want me to do some cleanup, happy to do that too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@peterc peterc peterc left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

AltStyle によって変換されたページ (->オリジナル) /