python struct unpack length error

Question

I've a bytes object with length 41. I tried to unpack it with:

struct.unpack('2B2B32sBi',data)

But I got a error that:

struct.error: unpack requires a bytes object of length 44

I think the length of 2B2B32sBi should be 2*1+2*1+32*1+1+4=41 after check the python document. Why am I wrong?

• David G. Pickett: unpack must have a buffer of the exact right length for the format, excess is an error, so you must truncate/slice to unpack early info like variable lengths and then create a new format than encompasses them, like: format2 = format1 + str(len) + 's'

Answer 1

You just encountered padding since you've got bytes data first, and then integer (which has stronger alignment constraints)

From the documentation:

Padding is only automatically added between successive structure members. No padding is added at the beginning or the end of the encoded struct.

So you have to specify an endianness to disable padding:

struct.unpack('<2B2B32sBi',data)

Edited for completeness, after reading Galen's excellent answer: just specifying = is better if you don't want to force endianness.

Answer 2

See the parts of the documentation regarding alignment:

By default, C types are represented in the machine’s native format and byte order, and properly aligned by skipping pad bytes if necessary (according to the rules used by the C compiler).

Native size and alignment are determined using the C compiler’s sizeof expression. This is always combined with native byte order.

Note the difference between '@' and '=': both use native byte order, but the size and alignment of the latter is standardized.

To illustrate this:

>>> import struct
>>> struct.calcsize("2B2B32sBi")
44
>>> struct.calcsize("@2B2B32sBi")
44
>>> struct.calcsize("=2B2B32sBi")
41

Answer 3

You may want to read the first note in the struct documentation again. Default is C-aligned data boundaries, so some padding bytes are responsible for the difference. So adding the appropriate byte order should fix the problem.