Encoding and decoding log entries in a Lossy Binary Format

Question

One of the projects I'm working on requires logging a huge amount of information (basically, every function call). The problem is obviously that logging a lot of information has a few side-effects, notably high disk-use (both IO's and amount); tends to require lookup-tables if you're trying to keep things small (i.e. an "integer to page" conversion, etc.); often isn't atomic, that is, concurrency can often become a high target-factor for threaded logging.

To fix some (most) of this, I've worked up a binary encoding for our logs. Basically, it allows us to store a huge amount of information in a tight format, and also allows us to do so atomically and without any concurrency issues. As a result, we can have high-performance logging that doesn't bog anything down. It has a small drawback: encoding of names and some values are "lossy", that is to say, we don't take the entire thing into account.

The specification is a notepad document, which is below:

HEADBYTE (1)
 00000001 - IP Version
 00000010 - Page Name Supplied (1 if supplied)
 01111100 - Parameter Count (Up to 32)
 10000000 - Always 0, indicates "V1", if 1 then there's another headbyte for the next version
IPBYTES (4 or 16)
 If IP Version = 0 Then 4 bytes (IPv4)
 Else 16 bytes (IPv6)
DATETIME (8)
 Unix EPOCH Time in UTC
USERID (5)
 00000011 - User Type
 4 bytes: Integer user ID
PAGENAME (0 or 16)
 If Page Name = 0 Then 0 bytes (not supplied)
 Else 16 bytes for MD5 hash of URL + Folder + Page name
FUNCTION NAME (16)
 16 bytes for MD5 hash of Namespace + Module + Function name
PARAMETERS (0 or *)
 If Parameter Count = 0 Then 0 bytes (not supplied)
 Else Parameters
 PARAMETER (<= 37)
 1-byte for config:
 PARAMETER CONFIG
 00000001 - 0 = Ref, 1 = Val
 00001110 - Type: 0 = Non-numeric type, # = 2^(n - 1) size bytes
 00010000 - Type Extension:
 If Type is non-numeric, 1 indicates data is string, 0 indicates custom class
 If Type is numeric, then 1 = Signed, 0 = Unsigned
 10000000 - Always 0, indicates "V1", if 1 then there's another byte(s) for the next version
 16 bytes for MD5 hash of parameter name
 If Type > 0 Then # bytes of actual numeric value
 Else
 If Type Extension = 1 Then 4 bytes for string length
 Else 16 bytes for MD5 hash of parameter type name (Full type name)

We can encode most of our data in less than 500 bytes, and much of it can be encoded in less than 200 bytes. By using the MD5 hash of the names / values, we can guarantee that there is not lookup or contention over storing them to any sort of "persistent" format, and it means we can build the lookup-table on-the-fly, so-to-speak. (We can store name + hash in the database, but it's no longer required for the logger to do it's job. We can fill it on a regular basis.) We use an MD5 hash for the speed, and the fact that, realistically, there are so few collisions on my use-case that it's not a big deal. Close counts, here.

An example (hexadecimal) log format might look like the following:

0f000000000000000000000000000000
01000001662b4bf83801000000c0c694
689b671598d0809e2935e7ccf4b6fa5c
dcf6485054f1bcd699050864f2d505a5
0691680602baa24206181b31ceb3c201
011029a5b77a07118ffbdbecfac18b73
f7c50000000100b05a9ad34ddf60cadf
2ba9dac8fa3a345592700a1f3b26e424
bc7fe95e50f703

Which, once line-breaks are stripped, can be decoded into the resulting type structure:

type MD5Hash = byte []
type Header =
 { IPV6 : bool
 PageNameSupplied : bool
 ParameterCount : byte }
type IP =
 | V4 of System.Net.IPAddress
 | V6 of System.Net.IPAddress
type UserType =
 | Client
 | Vendor
 | Admin
type DateTime = uint64
type UserId = UserType * uint32
type Number =
 | Byte of byte
 | SByte of sbyte
 | UInt16 of uint16
 | Int16 of int16
 | UInt32 of uint32
 | Int32 of int32
 | UInt64 of uint64
 | Int64 of int64
type ParameterValue =
 | CustomClass of MD5Hash
 | String of int
 | Numeric of Number
type Parameter =
 { ValueType : bool
 ParameterName : MD5Hash
 ParameterValue : ParameterValue }
type LogMessage =
 { IP : IP
 DateTime : DateTime
 UserId : UserId
 PageName : MD5Hash option
 FunctionName : MD5Hash
 Parameters : Parameter list }

To encode and decode we have some utility functions:

let md5Hash (str : string) : MD5Hash =
 use md5 = System.Security.Cryptography.MD5.Create()
 str
 |> System.Text.Encoding.UTF8.GetBytes
 |> md5.ComputeHash
let byteToUserType =
 function
 | 0b00000000uy -> UserType.Client
 | 0b00000001uy -> UserType.Vendor
 | 0b00000011uy -> UserType.Admin
 | _ -> failwith "Invalid User Type"
let userTypeToByte =
 function
 | UserType.Client -> 0b00000000uy
 | UserType.Vendor -> 0b00000001uy
 | UserType.Admin -> 0b00000011uy
let byteArrayToByte =
 function
 | [|b1|] -> b1
 | _ -> failwith "Byte may only have a single element byte-array"
let byteArrayToSByte =
 function
 | [|b1|] -> b1 |> sbyte
 | _ -> failwith "SByte may only have a single element byte-array"
let byteArrayToUInt16 =
 function
 | [|b1; b2|] -> (b1 |> uint16) <<< 8 ||| (b2 |> uint16)
 | _ -> failwith "UInt16 may only have a 2 element byte-array"
let byteArrayToInt16 =
 function
 | [|b1; b2|] -> (b1 |> int16) <<< 8 ||| (b2 |> int16)
 | _ -> failwith "Int16 may only have a 2 element byte-array"
let byteArrayToUInt32 =
 function
 | [|b1; b2; b3; b4|] -> (b1 |> uint32) <<< 24 ||| (b2 |> uint32) <<< 16 ||| (b3 |> uint32) <<< 8 ||| (b4 |> uint32)
 | _ -> failwith "UInt32 may only have a 4 element byte-array"
let byteArrayToInt32 =
 function
 | [|b1; b2; b3; b4|] -> (b1 |> int32) <<< 24 ||| (b2 |> int32) <<< 16 ||| (b3 |> int32) <<< 8 ||| (b4 |> int32)
 | _ -> failwith "Int32 may only have a 4 element byte-array"
let byteArrayToUInt64 =
 function
 | [|b1; b2; b3; b4; b5; b6; b7; b8|] -> (b1 |> uint64) <<< 54 ||| (b2 |> uint64) <<< 48||| (b3 |> uint64) <<< 40 ||| (b4 |> uint64) <<< 32 ||| (b5 |> uint64) <<< 24 ||| (b6 |> uint64) <<< 16 ||| (b7 |> uint64) <<< 8 ||| (b8 |> uint64)
 | _ -> failwith "UInt64 may only have a 8 element byte-array"
let byteArrayToInt64 =
 function
 | [|b1; b2; b3; b4; b5; b6; b7; b8|] -> (b1 |> int64) <<< 54 ||| (b2 |> int64) <<< 48||| (b3 |> int64) <<< 40 ||| (b4 |> int64) <<< 32 ||| (b5 |> int64) <<< 24 ||| (b6 |> int64) <<< 16 ||| (b7 |> int64) <<< 8 ||| (b8 |> int64)
 | _ -> failwith "Int64 may only have a 8 element byte-array"
let byteArrayToNumber signature (bytes : byte []) =
 match bytes, signature with
 | [|b1|], false -> bytes |> byteArrayToByte |> Byte
 | [|b1|], true -> bytes |> byteArrayToSByte |> SByte
 | [|b1; b2|], false -> bytes |> byteArrayToUInt16 |> UInt16
 | [|b1; b2|], true -> bytes |> byteArrayToInt16 |> Int16
 | [|b1; b2; b3; b4|], false -> bytes |> byteArrayToUInt32 |> UInt32
 | [|b1; b2; b3; b4|], true -> bytes |> byteArrayToInt32 |> Int32
 | [|b1; b2; b3; b4; b5; b6; b7; b8|], false -> bytes |> byteArrayToUInt64 |> UInt64
 | [|b1; b2; b3; b4; b5; b6; b7; b8|], true -> bytes |> byteArrayToInt64 |> Int64
 | _ -> failwith "Number must have a 1, 2, 4, or 8 element byte-array"
let byteToByteArray n = [|n|]
let sbyteToByteArray n = [|n |> byte|]
let uint16ToByteArray n = [|(n >>> 8) |> byte; (n) |> byte|]
let int16ToByteArray n = [|(n >>> 8) |> byte; (n) |> byte|]
let uint32ToByteArray n = [|(n >>> 24) |> byte; (n >>> 16) |> byte; (n >>> 8) |> byte; (n) |> byte|]
let int32ToByteArray n = [|(n >>> 24) |> byte; (n >>> 16) |> byte; (n >>> 8) |> byte; (n) |> byte|]
let uint64ToByteArray n = [|(n >>> 56) |> byte; (n >>> 48) |> byte; (n >>> 40) |> byte; (n >>> 32) |> byte; (n >>> 24) |> byte; (n >>> 16) |> byte; (n >>> 8) |> byte; (n) |> byte|]
let int64ToByteArray n = [|(n >>> 56) |> byte; (n >>> 48) |> byte; (n >>> 40) |> byte; (n >>> 32) |> byte; (n >>> 24) |> byte; (n >>> 16) |> byte; (n >>> 8) |> byte; (n) |> byte|]
let numberToByteArray =
 function
 | Byte n -> false, (n |> byteToByteArray)
 | SByte n -> true, (n |> sbyteToByteArray)
 | UInt16 n -> false, (n |> uint16ToByteArray)
 | Int16 n -> true, (n |> int16ToByteArray)
 | UInt32 n -> false, (n |> uint32ToByteArray)
 | Int32 n -> true, (n |> int32ToByteArray)
 | UInt64 n -> false, (n |> uint64ToByteArray)
 | Int64 n -> true, (n |> int64ToByteArray)

Then, we finally get to the heart of the encoding and decoding:

let readLogMessage (bytes : byte []) : Result<byte [] * LogMessage, string> =
 let readLogHeader (bytes : byte []) : Result<byte [] * Header, string> =
 let headerByte = bytes.[0]
 if (headerByte &&& 0b10000000uy) > 0uy then
 Error "Only V1 is supported."
 else
 let remBytes = bytes.[1..]
 let ipVersion = headerByte &&& 0b00000001uy > 0uy
 let pageNameSupplied = headerByte &&& 0b00000010uy > 0uy
 let parameterCount = headerByte &&& 0b01111100uy >>> 2
 (remBytes, { Header.IPV6 = ipVersion; PageNameSupplied = pageNameSupplied; ParameterCount = parameterCount }) |> Ok
 match bytes |> readLogHeader with
 | Ok (bytes, header) ->
 let bytes, ip =
 match header.IPV6 with
 | false -> bytes.[4..], V4 (bytes.[..3] |> System.Net.IPAddress)
 | true -> bytes.[16..], V6 (bytes.[..15] |> System.Net.IPAddress)
 let bytes, datetime = bytes.[8..], bytes.[..7]
 let bytes, userType = bytes.[1..], ((bytes.[0] &&& 0b00000011uy) |> byteToUserType)
 let bytes, userid = bytes.[4..], bytes.[..3]
 let bytes, pageName =
 match header.PageNameSupplied with
 | false -> bytes, None
 | true -> bytes.[16..], (bytes.[..15] |> Some)
 let bytes, functionName = bytes.[16..], bytes.[..15]
 let rec readParameters (bytes : byte []) ps rem =
 match rem with
 | 0 -> ps
 | r -> 
 let bytes, head = bytes.[1..], bytes.[0]
 let valueType = head &&& 0b00000001uy > 0uy
 let typeVal = head &&& 0b00001110uy >>> 1
 let typeExt = head &&& 0b00010000uy > 0uy
 let bytes, parameterName = bytes.[16..], bytes.[..15]
 let bytes, value =
 match typeVal, typeExt with
 | 0uy, false -> bytes.[16..], CustomClass (bytes.[..15])
 | 0uy, true -> bytes.[4..], String (bytes.[..3] |> byteArrayToInt32)
 | i, signed -> bytes.[i |> int..], Numeric (bytes.[..(pown 2 ((i |> int) - 1)) - 1] |> byteArrayToNumber signed)
 let ps = { ValueType = valueType; ParameterName = parameterName; ParameterValue = value } :: ps
 readParameters bytes ps (r - 1)
 Ok (bytes,
 { IP = ip
 DateTime = datetime |> byteArrayToUInt64
 UserId = userType, (userid |> byteArrayToUInt32)
 PageName = pageName
 FunctionName = functionName
 Parameters = readParameters bytes [] (header.ParameterCount |> int) |> List.rev })
 | Error e -> Error e
let writeLogMessage (msg : LogMessage) : Result<byte [], string> =
 let headerByte = match msg.IP with | V4 _ -> 0b00000000uy | V6 _ -> 0b00000001uy
 let headerByte = headerByte ||| match msg.PageName with | Some _ -> 0b00000010uy | None -> 0b00000000uy
 let headerByte = headerByte ||| ((msg.Parameters.Length <<< 3) |> byte >>> 1)
 let result = [|headerByte|]
 let result =
 match msg.IP with
 | V4 ip -> ip.GetAddressBytes()
 | V6 ip -> ip.GetAddressBytes()
 |> Array.append result
 let result = msg.DateTime |> uint64ToByteArray |> Array.append result
 let result =
 let t, id = msg.UserId
 id |> uint32ToByteArray |> Array.append [|0uy ||| (t |> userTypeToByte)|] |> Array.append result
 let result =
 match msg.PageName with
 | Some h -> h
 | None -> [| |]
 |> Array.append result
 let result = msg.FunctionName |> Array.append result
 let rec writeParameters (bytes : byte []) ps =
 match ps with
 | [] -> bytes
 | head::ps ->
 let headByte = 0uy
 let headByte = headByte ||| if head.ValueType then 0b00000001uy else 0b00000000uy
 let addHeadByte, data =
 match head.ParameterValue with
 | CustomClass hash -> 0b00000000uy, hash
 | String length -> 0b00010000uy, (length |> int32ToByteArray)
 | Numeric n ->
 let signed, bytes = n |> numberToByteArray
 (if signed then 0b00010000uy else 0b00000000uy) ||| ((log (bytes.Length |> float) / log 2. + 1.) |> byte <<< 1), bytes
 let headByte = headByte ||| addHeadByte
 let result = [|headByte|]
 let result = head.ParameterName |> Array.append result
 let result = data |> Array.append result
 writeParameters (result |> Array.append bytes) ps
 msg.Parameters
 |> writeParameters [| |]
 |> Array.append result
 |> Ok

And we can verify the whole thing with a simple script:

let input =
 [| //0b00001110uy // IPv4
 0b00001111uy // IPv6
 //0b01111111uy; 0b00000000uy; 0b00000000uy; 0b00000001uy // IPv4
 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000001uy // IPv6
 0b00000000uy; 0b00000000uy; 0b00000001uy; 0b01100110uy; 0b00101011uy; 0b01001011uy; 0b11111000uy; 0b00111000uy // DateTime Stamp
 0b00000001uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b11000000uy // User ID
 0b11000110uy; 0b10010100uy; 0b01101000uy; 0b10011011uy; 0b01100111uy; 0b00010101uy; 0b10011000uy; 0b11010000uy; 0b10000000uy; 0b10011110uy; 0b00101001uy; 0b00110101uy; 0b11100111uy; 0b11001100uy; 0b11110100uy; 0b10110110uy // Page Name
 0b11111010uy; 0b01011100uy; 0b11011100uy; 0b11110110uy; 0b01001000uy; 0b01010000uy; 0b01010100uy; 0b11110001uy; 0b10111100uy; 0b11010110uy; 0b10011001uy; 0b00000101uy; 0b00001000uy; 0b01100100uy; 0b11110010uy; 0b11010101uy // Function Name
 0b00000101uy; 0b10100101uy; 0b00000110uy; 0b10010001uy; 0b01101000uy; 0b00000110uy; 0b00000010uy; 0b10111010uy; 0b10100010uy; 0b01000010uy; 0b00000110uy; 0b00011000uy; 0b00011011uy; 0b00110001uy; 0b11001110uy; 0b10110011uy; 0b11000010uy; 0b00000001uy; 0b00000001uy // Parameter 1
 0b00010000uy; 0b00101001uy; 0b10100101uy; 0b10110111uy; 0b01111010uy; 0b00000111uy; 0b00010001uy; 0b10001111uy; 0b11111011uy; 0b11011011uy; 0b11101100uy; 0b11111010uy; 0b11000001uy; 0b10001011uy; 0b01110011uy; 0b11110111uy; 0b11000101uy; 0b00000000uy; 0b00000000uy; 0b00000000uy; 0b00000001uy // Parameter 2
 0b00000000uy; 0b10110000uy; 0b01011010uy; 0b10011010uy; 0b11010011uy; 0b01001101uy; 0b11011111uy; 0b01100000uy; 0b11001010uy; 0b11011111uy; 0b00101011uy; 0b10101001uy; 0b11011010uy; 0b11001000uy; 0b11111010uy; 0b00111010uy; 0b00110100uy; 0b01010101uy; 0b10010010uy; 0b01110000uy; 0b00001010uy; 0b00011111uy; 0b00111011uy; 0b00100110uy; 0b11100100uy; 0b00100100uy; 0b10111100uy; 0b01111111uy; 0b11101001uy; 0b01011110uy; 0b01010000uy; 0b11110111uy; 0b00000011uy // Parameter 3
 |]
let outMessage = readLogMessage input
let message =
 { //IP = "127.0.0.1" |> System.Net.IPAddress.Parse |> V4 // IPv4
 IP = "::1" |> System.Net.IPAddress.Parse |> V6 // IPv6
 DateTime = System.DateTimeOffset(2018, 9, 30, 16, 24, 51, System.TimeSpan()).ToUnixTimeMilliseconds() |> uint64
 UserId = UserType.Vendor, 192u
 PageName = "Dir/Page.aspx" |> md5Hash |> Some
 FunctionName = "Page_Load" |> md5Hash
 Parameters =
 [{ ValueType = true
 ParameterName = "Parameter 1" |> md5Hash
 ParameterValue = 257us |> UInt16 |> Numeric }
 { ValueType = false
 ParameterName = "Parameter 2" |> md5Hash
 ParameterValue = 1 |> String }
 { ValueType = false
 ParameterName = "Parameter 3" |> md5Hash
 ParameterValue = "Class Type" |> md5Hash |> CustomClass }
 ]
 }
let output = writeLogMessage message
input |> Array.iter (printf "%02x")
printfn ""
match output with
| Ok o ->
 o |> Array.iter (printf "%02x")
 printfn ""
 Seq.fold2 (fun res b1 b2 -> res && b1 = b2) true input o
| Error e -> false

This shows us what the function writing a log would look like (the writeLogMessage part), and what reading a log looks like (the readLogMessage part).

• RubberDuck: Is there a protobuf library for F#? It was designed for your usecase.
• Der Kommissar: @RubberDuck I'd assume the generic .NET one should work, just might be a bit cumbersome to do F# with.
• TeaDrivenDev: Froto is F# specific: github.com/ctaggart/froto
• Der Kommissar: @TeaDrivenDev Sounds like you have the basis of an answer. ;)

Answer 1

At a quick glance, I could nitpick about how I think K&R (Java) style braces are cleaner for type definitions or that hex literals are better than binary literals, but that’s really my own personal preference. For an explanation on why hex is better than binary see this answer on a question of mine.

I feel like some partially applied functions may help clean up this section.

let uint16ToByteArray n = [|(n >>> 8) |> byte; (n) |> byte|]
let int16ToByteArray n = [|(n >>> 8) |> byte; (n) |> byte|]
let uint32ToByteArray n = [|(n >>> 24) |> byte; (n >>> 16) |> byte; (n >>> 8) |> byte; (n) |> byte|]

I’m a little concerned about your User Type. You have 3 types.

01
10
11

Is user type 3 a combination of types 1 and 2? I would expect to be able to do bitwise comparison of values with such a scheme. If you don’t intend this, I would modify 3 -> 4 (100). I’m a little unclear on your intention, so ignore this if I’m wrong.

Ultimately though, I would consider pulling in a dependency on a serialization library like Protobuf.

• Der Kommissar: Yeah the UserType is an enum of mutually exclusive values. Because the serialization is very specific to the binary format I wrote them like that.
• RubberDuck: So, part of the confusion here is they’re defined with a binary literal. If you used a 1 based, 1 incremented enum, I wouldn’t even be thinking of bitwise comparisons.
• RubberDuck: Never mind. I was looking at the byteToUserType func. I’m tired...
• Der Kommissar: To be fair I do need a None case there.

Answer 2

I think there is a problem with some of your byteArrayToXX functions:

As an example:

let byteArrayToInt32 =
 function
 | [|b1; b2; b3; b4|] -> (b1 |> int32) <<< 24 ||| (b2 |> int32) <<< 16 ||| (b3 |> int32) <<< 8 ||| (b4 |> int32)
 | _ -> failwith "Int32 may only have a 4 element byte-array"

When testing with [| 5uy; 3uy; 4uy; 8uy|] it produces 50332680, but the correct result should be 84083720 or binary 0000 ‭0101 0000 0011 0000 0100 0000 1000‬.

The solution is to add parentheses as follows:

let byteArrayToInt32 =
 function
 | [|b1; b2; b3; b4|] -> ((b1 |> int32) <<< 24) ||| ((b2 |> int32) <<< 16) ||| ((b3 |> int32) <<< 8) ||| (b4 |> int32)
 | _ -> failwith "Int32 may only have a 4 element byte-array"

It seems to be a precedence (or more precise: left associativity) problem:

(5 <<< 24 ||| 3 <<< 16) is actually calculated as ((5 <<< 24 ||| 3) <<< 16), but the correct calculation should be ((5 <<< 24) ||| (3 <<< 16))

---

As an exercise I have tried to refactor the converter functions to something that is more maintainable than the originals:

let inline convertBytes caster initial length failMsg bytes = 
 match bytes |> Array.length with
 | x when x = length -> Array.foldBack (fun b (acc, shift) -> acc ||| ((caster b) <<< shift), shift + 8) bytes (initial, 0) |> (fun (acc, _) -> acc)
 | _ -> failwith failMsg
let byteArrayToByte = convertBytes id 0uy 1 "Byte may only have a single element byte-array"
let byteArrayToSByte = convertBytes sbyte 0y 1 "SByte may only have a single element byte-array"
let byteArrayToUInt16 = convertBytes uint16 0us 2 "UInt16 may only have a 2 element byte-array"
let byteArrayToInt16 = convertBytes int16 0s 2 "Int16 may only have a 2 element byte-array"
let byteArrayToUInt32 = convertBytes uint32 0u 4 "UInt32 may only have a 4 element byte-array" 
let byteArrayToInt32 = convertBytes int32 0 4 "Int32 may only have a 4 element byte-array"
let byteArrayToUInt64 = convertBytes uint64 0UL 8 "UInt64 may only have a 8 element byte-array"
let byteArrayToInt64 = convertBytes int64 0L 8 "Int64 may only have a 8 element byte-array"
let byteArrayToNumber signature (bytes : byte []) =
 match bytes.Length, signature with
 | 1, false -> bytes |> byteArrayToByte |> Byte
 | 1, true -> bytes |> byteArrayToSByte |> SByte
 | 2, false -> bytes |> byteArrayToUInt16 |> UInt16
 | 2, true -> bytes |> byteArrayToInt16 |> Int16
 | 4, false -> bytes |> byteArrayToUInt32 |> UInt32
 | 4, true -> bytes |> byteArrayToInt32 |> Int32
 | 8, false -> bytes |> byteArrayToUInt64 |> UInt64
 | 8, true -> bytes |> byteArrayToInt64 |> Int64
 | _ -> failwith "Number must have a 1, 2, 4, or 8 element byte-array"
let inline numberToBytes count n = Array.init (count) (fun i -> (byte (n >>> ((count - 1 - i) * 8))))
let byteToByteArray = numberToBytes 1
let sbyteToByteArray = numberToBytes 1
let uint16ToByteArray = numberToBytes 2
let int16ToByteArray = numberToBytes 2
let uint32ToByteArray = numberToBytes 4
let int32ToByteArray = numberToBytes 4
let uint64ToByteArray = numberToBytes 8
let int64ToByteArray = numberToBytes 8
let numberToByteArray =
 function
 | Byte n -> false, (n |> byteToByteArray)
 | SByte n -> true, (n |> sbyteToByteArray)
 | UInt16 n -> false, (n |> uint16ToByteArray)
 | Int16 n -> true, (n |> int16ToByteArray)
 | UInt32 n -> false, (n |> uint32ToByteArray)
 | Int32 n -> true, (n |> int32ToByteArray)
 | UInt64 n -> false, (n |> uint64ToByteArray)
 | Int64 n -> true, (n |> int64ToByteArray)

I'm not claiming it to be the state of the art or that they have better performance. It's just what it is.

---

If I should reorganize your main read- and writeLogMessage functions, I think I would try something like this:

let readLogMessage (bytes: byte[]) =
 let readHeader result bts = result // TODO implement the function
 let readIp result bts= result // TODO implement the function
 let readUserId result bts= result // TODO implement the function
 let readPageName result bts= result // TODO implement the function
 let readFunctionName result bts= result // TODO implement the function
 let readParameters result bts= result // TODO implement the function
 [readHeader; readIp; readUserId; readPageName; readFunctionName; readParameters; ] 
 |> List.fold (fun (bts, header, msg) fn -> fn (bts, header, msg) bts) 
 (bytes, 
 { IPV6 = false; PageNameSupplied = false; ParameterCount = 0uy }, 
 { IP = System.Net.IPAddress.Parse("::1") |> V6; DateTime = 0UL; UserId = (Client, 0ul); PageName = None; FunctionName = [||]; Parameters = [] })

Here I use List.fold on a list of partial functions where the state object passed each function is a tuple: (remaining bytes, header, message). I think a similar approach can be used in the write function. Again I'm not claiming this to be a better solution, just another way to do things, that I find easier to read and maintain.

• Der Kommissar: I like your reorganization, it'd juts be a matter of properly supplying the header for it to work properly. Should probably fold parameters the same way, too.