Write Ahead Log for LSM or bitcask storage.
- Disk based, support large data volume
- Append only write, high performance
- Fast read, one disk seek to retrieve any value
- Support batch write, all data in a batch will be written in a single disk seek
- Iterate all data in wal with
NewReaderfunction - Extremely fast read
- Support concurrent write and read, all functions are thread safe
Format of a single segment file:
+-----+-------------+--+----+----------+------+-- ... ----+
File | r0 | r1 |P | r2 | r3 | r4 | |
+-----+-------------+--+----+----------+------+-- ... ----+
|<---- BlockSize ----->|<---- BlockSize ----->|
rn = variable size records
P = Padding
BlockSize = 32KB
Format of a single record:
+----------+-------------+-----------+--- ... ---+
| CRC (4B) | Length (2B) | Type (1B) | Payload |
+----------+-------------+-----------+--- ... ---+
CRC = 32-bit hash computed over the payload using CRC
Length = Length of the payload data
Type = Type of record
(FullType, FirstType, MiddleType, LastType)
The type is used to group a bunch of records together to represent
blocks that are larger than BlockSize
Payload = Byte stream as long as specified by the payload size
func main() { wal, _ := wal.Open(wal.DefaultOptions) // write some data chunkPosition, _ := wal.Write([]byte("some data 1")) // read by the position val, _ := wal.Read(chunkPosition) fmt.Println(string(val)) wal.Write([]byte("some data 2")) wal.Write([]byte("some data 3")) // iterate all data in wal reader := wal.NewReader() for { val, pos, err := reader.Next() if err == io.EOF { break } fmt.Println(string(val)) fmt.Println(pos) // get position of the data for next read } }