How to read structured binary data from a file?

Question

The following C++ code writes a header to a file:

#include <iostream>
struct Header
{
 uint16_t name;
 uint8_t type;
 uint8_t padding;
 uint32_t width, height;
 uint32_t depth1, depth2;
 float dMin, dMax;
};
int main()
{
 Header header;
 header.name = *reinterpret_cast<const uint16_t*>("XO");
 header.type = true;
 header.width = (uint32_t)512;
 header.height = (uint32_t)600;
 header.depth1 = (uint32_t)16;
 header.depth2 = (uint32_t)25;
 header.dMin = 5.0;
 header.dMax = 8.6;
 FILE* f = fopen("header.bin", "wb");
 fwrite(&header, sizeof(Header), 1, f);
}

I am looking to read these header.bin files using Python. In C++ I would be doing something like:

fread(&header, sizeof(Header), 1, f)

But I'm unsure how to read the bytes and convert them into the corresponding fields that the Header struct has in Python?

• MisterMiyagi: Python has an aptly named struct module suitable for such tasks.
• martineau: Use the struct module in the standard library.

Answer 1

Use the struct module to define the binary layout of a C-like struct and de-/serialise it:

import struct
# Format String describing the data layout
layout = "H B x 2L 2L 2f"
# Object representing the layout, including size
header = struct.Struct(layout)
with open("header.bin", "rb") as in_stream:
 print(header.unpack(in_stream.read(header.size))

The layout is a format string describing the fields in-order, e.g. H for uint16_t, B for uint8_t, x for a pad byte, and so on.

• ti7: this is probably the cleanest approach
• Ben Y: Be sure to precede the layout with the appropriate endianness character if the data can be affected by endianness. (see the documentation for the struct module). Typically < or > for forcing little or big-endianness, respectively
• nickponline: Thank you and yes I needed to add the little-endian layout = "<H B x 2L 2L 2f"
• Mark Tolonen: FYI ctypes.Structure also supports bit fields, which struct does not, if that is desired.

Answer 2

I would do this with Python's ctypes, somewhat so you can share the Header header

Create a class from ctypes.Structure to map the types

import ctypes
class StructHeader(ctypes.Structure):
 _fields_ = [
 ("name", ctypes.c_uint16),
 ("type", ctypes.c_uint8),
 ...
 ]

And create a function which does what you want with a signature like

int header(struct Header &buffer)
{
 // open the file and write to buffer
 // opportunity for other features
}

Then you can compile a shared object to read it which returns that type

gcc -shared -Wl,-soname,your_soname \
 -o library_name file_list library_list

And call out with ctypes.CDLL to read the headers

header = ctypes.CDLL("mylib.so.1").header # function named header
header.argtypes = [ctypes.POINTER(StructHeader)]
header.restype = ctypes.c_int
# allocate struct for write
buffer = StructHeader()
# call out to function to write buffer
header(buffer)