Shared memory between C and python

Question

I want to share memory between a program in C and another in python.

The c program uses the following structure to define the data.

struct Memory_LaserFrontal {
 char Data[372]; // original data
 float Med[181]; // Measurements in [m]
 charD; // 'I': Invalid -- 'V': Valid
 charS; // 'L': Clean -- 'S': Dirty
 char LaserStatus[2];
 };

From python I have managed to read the variable in memory using sysv_ipc but they have no structure and is seen as a data array. How can I restructure them?

python code:

from time import sleep
import sysv_ipc
# Create shared memory object
memory = sysv_ipc.SharedMemory(1234)
# Read value from shared memory
memory_value = memory.read()
print (memory_value)
print (len(memory_value))
while True:
 memory_value = memory.read()
 print (float(memory_value[800]))
 sleep(0.1)

I have captured and printed the data in python, I have modified the sensor reading and the read data is also modified, confirming that the read data corresponds to the data in the sensor's shared memory. But without the proper structure y cant use the data.

• Craig Estey: See: docs.python.org/3/library/struct.html

Answer 1

I always like to leave the full source code of the problem solved so others can use it if they have a similar problem. thanks a lot all!

from time import sleep
import sysv_ipc
import struct
import array
# Create shared memory object
while True:
 memory = sysv_ipc.SharedMemory(1234)
 # Read value from shared memory
 memory_value = memory.read()
 #print (memory_value)
 #print (len(memory_value))
 # Get the initial char array - you can turn it into a string if you need to later.
 my_chars = array.array("b") # f for float, c for char etc.
 #my_chars.from_bytes(memory_value[:372]) # happens that 372 chars is 372 bytes.
 Data = my_chars.tolist() # Could be bytes list
 # advance to the member after Data
 end_of_Data = struct.calcsize("372c") 
 # get the length in bytes that 181 floats take up
 end_of_Med = struct.calcsize("181f") + end_of_Data
 # now we know where the floats are
 floats_as_bytes = memory_value[ end_of_Data : end_of_Med ]
 # unpack the remaining parts
 ( D, S, LaserStatus_1, LaserStatus_2 ) = struct.unpack( "cccc", memory_value[end_of_Med:] )
 print(len(floats_as_bytes)/4)
 a=[]
 for i in range(0,len(floats_as_bytes),4):
 a.append(struct.unpack('<f', floats_as_bytes[i:i+4]))
 print (a[0])
 sleep(0.1)

Answer 2

You need to unpack your binary data structure into Python types. The Python modules struct and array can do this for you.

import struct
import array

NB: Some C compilers, but not the comomn ones, may pad your member variables to align each of them with the expected width for your CPU ( almost always 4 bytes ). This means that it may add padding bytes. You may have to experiment with the struct format parameter 'x' between the appropriate parts of your struct if this is the case. Python's struct module does not expect aligned or padded types by default, you need to inform it. See my note at the very end for a guess on what the padding might look like. Again, per @Max's comment, this is unlikely.

NB: I think the members charD and charS are really char D; and char S;

Assuming you want the floats as a Python list or equivalent we have to do some more work with the Python array module . Same for the char[] Data.

 # Get the initial char array - you can turn it into a string if you need to later.
 my_chars = array.array("b") # f for float, b for byteetc.
 my_chars.from_bytes(memory_value[:372]) # happens that 372 C chars is 372 bytes.
 Data = my_chars.tolist() # Could be bytes list
 # advance to the member after Data
 end_of_Data = struct.calcsize("372c") 
 # get the length in bytes that 181 floats take up
 end_of_Med = struct.calcsize("181f") + end_of_Data
 # now we know where the floats are
 floats_as_bytes = memory_value[ end_of_Data : end_of_Med ]
 # unpack the remaining parts
 ( D, S, LaserStatus_1, LaserStatus_2 ) = struct.unpack( "cccc", memory_value[end_of_Med:] )

Now use the array module to unpack to make a Python list

 my_floats = array.array("f") # f for float, c for char etc.
 my_floats.from_bytes(floats_as_bytes)

Now Data might be a list of Python bytes type that you need to convert to your preferred string encoding. Usually .decode('utf-8') is good enough.

 Data_S = "".join(Data).decode('utf-8') # get a usable string in Data_S

Padding

struct Memory_LaserFrontal {
 char Data[372]; // 372 is a multiple of 4, probably no padding 
 float Med[181]; // floats are 4 bytes, probably no padding
 charD; // single char, expect 3 padding bytes after
 charS; // single char, expect 3 padding bytes after
 char LaserStatus[2]; // double char expect 2 padding bytes after.
 };

So the last Python line above might be - where the 'x' indicates a padding byte that can be ignored.

 ( D, S, LaserStatus_1, LaserStatus_2 ) = struct.unpack( "cxxxcxxxccxx", memory_value[end_of_Med:] )

• PBocca: Thenx for the answer. I will experiment and comment the results.
• JimmyNJ: @PBocca - Let me know here, and I'll try to help more, there are many moving parts here, and if they don't line up perfectly, you can get garbage.
• Max: A C compiler won't generally put padding bytes between two characters, because the second char has no alignment requirements. So no padding after char D;. I don't think it'll put padding before laserStatus either, but I'm less sure about that.
• JimmyNJ: @Max agreed, it's not common to have chars padded, but it was worth noting imho. Answer adjusted to your point.
• PBocca: I have to chage c to b because c is a bad typecode (must be b, B, u, h, H, i, I, l, L, q, Q, f or d) b is signed char B unsigned I'm having a problem with line: my_chars.from_bytes(memory_value[:372]) error said: 'array.array' object has no attribute 'from_bytes' If i comment that line and continue i can successfully extract part of the datta getting from: print(D) print(S) print(LaserStatus_1) print(LaserStatus_2) b'V' b'L' b'M' b'I' That are proper messages from the sensor but i have not been able to recover the float values that are the most important ones.