python concatenate ctype struct and array

Question

I am trying to implement a custom UDP protocol. This protocol has a header and data. The header has information about the structure of the data.

The header is defined using a ctypes struct and the data using 'h' array(signed short).

I am using python sockets.

I tried sending the header ans data using separate calls to "socket.sendto" like this:

s.sendto(header, addr)
s.sendto(data, addr)

But I am not able to receive this as a continuous stream. "socket.recvfrom" is only fetching the header. Maybe if I call "socket.recvfrom" again I will get the data as well. But thats not what I need. I need the full packet as a stream. I think concatenating the header and data in the server itself might fix this.

So I tried different combinations of the following to concatenate the two:

• converting the header to "c_char_p" array.
• Converting the data to bytearray
• using numpy.concatenate
• Standard array concatenation after converting the header to 'B' array.
• Converting header to 'h' array. (This failed with "string length not a multiple of item size"

All the above failed for one reason or the other.

If I need to convert either header or data, I would prefer it to be the header as it is smaller. But if there is no way around, I am ok with converting the data.

Would appreciate any help.

Relevant code snippets:

sdata = array("h")
.
.
.
header=protocol.HEADER(1,50,1,50,1,50,1,50,1,10,1,12,1,1,1,1,1,18,19,10,35,60,85,24,25)
.
.
s.sendto(header+sdata, addr)

Answer 1

You can copy the header struct into a ctypes array of bytes:

>>> buf = (ctypes.c_char * ctypes.sizeof(header)).from_buffer_copy(header)

Now, in Python 2,

>>> buf.raw + sdata.tostring()

should give you what you're looking for.

In Python 3, it would be

>>> buf.raw + sdata.tobytes()

• Eryk Sun: That makes a lot of copies. Using bytearray you can minimize this to copying the header and data only once: buf = bytearray(header); buf.extend((ctypes.c_short * len(sdata)).from_buffer(sdata)). You can't use buf.extend(sdata) directly since array.array doesn't support the new buffer protocol (thus extend falls back on iteration, which won't work since the values can exceed 255), but a ctypes array comes to the rescue by supporting both protocols.
• Eryk Sun: You could also use ctypes.resize to realloc the header and then ctypes.memmove to copy the data buffer from sdata.buffer_info()[0]. While it's closer to how you'd go about this in C, I don't think it's worth the added complexity, and it's probably slower anyway.
• Alex Martelli: Yep, there are other alternatives too (some working only in Python 2 and broken in Python 3), I just tried to present one reasonable one.
• Alex Martelli: @eryksun, in Python 2, I get len(bytearray(header)) equal to zero, when header is an instance of a class subclassing ctypes.Structure -- what do you get?
• Eryk Sun: Did you define _fields_ for the Structure subclass?