Shared-memory IPC solution for both Linux and Windows

Question

Below is a simple and perfect solution on Windows for IPC with shared memory, without having to use networking / sockets (that have annoying limits on Windows). The only problem is that it's not portable on Linux:

Avoiding the use of the tag parameter will assist in keeping your code portable between Unix and Windows.

Question: is there a simple way built-in in Python, without having a conditional branch "if platform is Windows, if platform is Linux" to have a shared-memory mmap?

Something like

mm = sharedmemory(size=2_000_000_000, name="id1234") # 2 GB, id1234 is a global
 # id available for all processes
mm.seek(1_000_000)
mm.write(b"hello")

that would internally default to mmap.mmap(..., tagname="id1234") on Windows and use /dev/shm on Linux (or maybe even a better solution that I don't know?), and probably something else on Mac, but without having to handle this manually for each different OS.

---

Working Windows-only solution:

#server
import mmap, time
mm = mmap.mmap(-1, 1_000_000_000, tagname="foo")
while True:
 mm.seek(500_000_000)
 mm.write(str(time.time()).encode())
 mm.flush()
 time.sleep(1)
# client
import mmap, time
mm = mmap.mmap(-1, 1_000_000_000, tagname="foo")
while True:
 mm.seek(500_000_000)
 print(mm.read(128))
 time.sleep(1)

Answer 1

The easiest way is to use python with version >=3.8, it has added a built-in abstraction for shared memory, it works on both windows and linux https://docs.python.org/3.10/library/multiprocessing.shared_memory.html

The code will look something like this:

Process #1:

 from multiprocessing import shared_memory
 # create=true to create a new shared memory instance, if it already exists with the same name, an exception is thrown
 shm_a = shared_memory.SharedMemory(name="example", create=True, size=10)
 shm_a.buf[:3] = bytearray([1, 2, 3])
 while True:
 do_smt()
 shm_a.close()

Process #2:

 from multiprocessing import shared_memory
 # create=false, use existing
 shm_a = shared_memory.SharedMemory(name="example", size=10)
 print(bytes(shm.buf[:3]))
 # [0x01, 0x02, 0x03]
 while True:
 do_smt()
 shm_a.close()

Otherwise, I think there are no common good solutions and you will need to reinvent the wheel :)

Answer 2

Personally this has worked well for me

Option 1: http://www.inspirel.com/yami4/

The YAMI4 suite for general computing is a multi-language and multi-platform package.

Several Operating systems:

Sample code

Microsoft Windows, POSIX (Linux, Max OS X, FreeBSD, ...), QNX (with native IPC messaging), FreeRTOS, ThreadX, TI-RTOS. Programming languages: C++, Ada, Java, .NET, Python, Wolfram.

Option 2: ZeroMq https://zeromq.org/