Decoding packed data into a structure

Question

What would the best way of unpacking a python string into fields

I have data received from a tcp socket, it is packed as follows, I believe it will be in a string from the socket recv function

It has the following format

uint8 - header
uint8 - length
uint32 - typeID
uint16 -param1
uint16 -param2
uint16 -param3
uint16 -param4
char[24] - name string
uint32 - checksum
uint8 - footer

(I also need to unpack other packets with different formats to the above)

How do I unpack these?

I am new to python, have done a bit of 'C'. If I was using 'C' I would probably use a structure, would this be the way to go with Python?

Regards

X

Answer 1

The struct module is designed to unpack heterogeneous data to a tuple based on a format string. It makes more sense to unpack the whole struct at once rather than trying to pull out one field at a time. Here is an example:

fields = struct.unpack('!BBI4H20sIB', data)

Then you can access a given field, for example the first field:

fields[0]

You could also use the tuple to initialize a NamedTuple; look at the documentation for struct for an example. NamedTuples are only available in Python 2.6+, but they behave more like Python structures in that you can access elements as attributes, e.g. fields.header. Of course, you could also accomplish this with a little more work by writing a class to encapsulate the information from the tuple... again, if you care. You can always just index into fields directly, as I showed above.

Answer 2

use struct module

Answer 3

This is an answer to your question-as-an-answer:

It certainly can't be the best way, because it DOESN'T WORK. struct.unpack() always returns a tuple. To pluck out the single item in that tuple, you need to do either field1 = struct.unpack('B',data[0])[0] or field1, = struct.unpack('B',data[0]).

Even with that fix, it's not a good way: too much typing, error proneness of unnecessary [start:end], inefficiency of 10 function calls instead of one.

As you have names, you can use them instead of field1 or field[0] ... like this:

(header, length, typeID, param1, param2,
param3, param4, name_string, checksum, footer,
) = struct.unpack("!2B I 4H 24s I B", data)

• mikip: Thanks John, even better, much appreciated
• John Machin: Thanks for expressing your appreciation in words. Did you know that you can change your selected answer?

Answer 4

Is this the best way of doing this or is there a better way

It is likely that there will be strings with other formats which will require a different unpacking scheme

field1 = struct.unpack('B',data[0])
field2 = struct.unpack('B',data[1])
field3 = struct.unpack('!I',data[2:6])
field4 = struct.unpack('!H',data[6:8])
field5 = struct.unpack('!H',data[8:10])
field6 = struct.unpack('!H',data[10:12])
field7 = struct.unpack('!H',data[12:14])
field8 = struct.unpack('20s',data[14:38])
field9 = struct.unpack('!I',data[38:42])
field10 = struct.unpack('B',data[42])

Regards

• atzz: Sure there is. No need to get each field separately. See musicinmybrain's answer for an example (stackoverflow.com/questions/1879914/…).
• mikip: Thanks, just seen musicbrains response Thanks everyone for your time, much appreciated

Answer 5

Take a look at the module 'struct'.

• Pedro Romano: Same suggestion as previously provided by @SilentGhost.