Accessing 'shared memory' with Python 3.x < 3.8

Question

In my application multiple processes rely on one source, either a camera or video stream. 'Shared memory' as introduced with Python 3.8 provides a neat solution as one process shares the frames on a dedicated address and any other process picks it up. Unfortunately, one process is dependent on a library that is not available in Python 3.8.

Is it possible to get the 'shared memory' with Python 3.x < 3.8 by means of the identifier with which the address can be determined?

Any suggestions are welcome!

Answer 1

You can use mmap and share the memory mapped file between the processes.

You can follow this blog post as guidance.

• Patrick Na: Appreciate the hint! What is not clear yet is how a process 'b' is suppose to find the data mapped by a process 'a'? This is what I meant by 'means of the identifier'.
• Joac: mmap implements the file interface, if you write bytes at position 42 on the producer, you will be able to read the data on the consumer at the same position. Hope it helps.
• Patrick Na: Sorry, I read your message after writing this: If I understand it correctly, fileno is the identifier?! I will try that.
• Joac: You are rigth (I just assumed that you were having trouble with the file interface itself, will extend my answer with a short snippet.

Answer 2

@Joac pointed me to the right direction and I learned a little bit more about shared memory, mmap and then also about shmget, shmat, etc. For anyone who is interested in learning more about it I found the following videos and blog-post in particular useful:

How to Map Files into Memory in C (mmap, memory mapped file io)

How to Set up Shared Memory in Your Linux and MacOS Programs. (shmget, shmat, shmdt, shmctl, ftok)

Using python to read shared memory under Linux

In the end I solved the problem by means of the posix_ipc module that was introduced for Python2.x already and in fact works very similar to the native Python3.8 solution I am referring to above.

Example code for sharing a video from a Python 3.8 script and processing it with a script that works with Python 3.x < Python 3.8. Feel free to comment on this example snippet. I believe there are elements that can be improved:

from multiprocessing import shared_memory
import cv2
import numpy as np
if __name__ == "__main__":
 video_path = "/home/john/Videos/sample.mp4"
 video_capture = cv2.VideoCapture(video_path)
 fps = video_capture.get(cv2.CAP_PROP_FPS)
 return_value, frame = video_capture.read()
 
 shm = shared_memory.SharedMemory(name="video_stream", create=True, size=frame.nbytes)
 shared_image = np.ndarray(frame.shape, dtype=frame.dtype, buffer=shm.buf)
 shared_image[:] = frame[:]
 while True:
 return_value, frame = video_capture.read()
 shared_image[:] = frame[:]
 if (return_value):
 cv2.imshow('python3.8', shared_image)
 delay = int((1/fps) * 1000)
 if (cv2.waitKey(delay) & 0xFF == ord('q')):
 break
 video_capture.release()
 cv2.destroyAllWindows()

import mmap
import cv2
import numpy as np
import posix_ipc
if __name__ == "__main__":
 shm = posix_ipc.SharedMemory('/video_stream')
 mapfile = mmap.mmap(shm.fd, shm.size)
 shared_image = np.ndarray((720, 1280, 3), dtype=np.uint8,
 buffer=mapfile.read())
 while True:
 mapfile.seek(0)
 shared_image = np.ndarray((720, 1280, 3), dtype=np.uint8,
 buffer=mapfile.read())
 cv2.imshow('python3.6', shared_image)
 if (cv2.waitKey(1) & 0xFF == ord('q')):
 break
 cv2.destroyAllWindows()