homepage

Sockets have a recv_into() method, io.IOBase has a readinto() method, but there is no os.read_into() function. It would avoid memory copies. It would benefit to the Python implementation FileIO (readall() and readinto() methods), see the issue #21859.

os.read_into() may be used by the following functions.
subprocess.Popen._execute_child():
 # Wait for exec to fail or succeed; possibly raising an
 # exception (limited in size)
 errpipe_data = bytearray()
 while True:
 part = os.read(errpipe_read, 50000)
 errpipe_data += part
 if not part or len(errpipe_data) > 50000:
 break
subprocess.Popen.communicate():
 self._fileobj2output = {}
 if self.stdout:
 self._fileobj2output[self.stdout] = []
 ...
 data = os.read(key.fd, 32768)
 if not data:
 ...
 self._fileobj2output[key.fileobj].append(data)
 ...
 stdout = b''.join(...)
multiprocessing.Connection._recv():
 def _recv(self, size, read=_read):
 buf = io.BytesIO()
 handle = self._handle
 remaining = size
 while remaining > 0:
 chunk = read(handle, remaining)
 n = len(chunk)
 if n == 0:
 if remaining == size:
 raise EOFError
 else:
 raise OSError("got end of file during message")
 buf.write(chunk)
 remaining -= n
 return buf
multiprocessing.read_unsigned():
 def read_unsigned(fd):
 data = b''
 length = UNSIGNED_STRUCT.size
 while len(data) < length:
 s = os.read(fd, length - len(data))
 if not s:
 raise EOFError('unexpected EOF')
 data += s
 return UNSIGNED_STRUCT.unpack(data)[0]
The problem is that some functions still require to return a bytes, not a bytearray or something else. Converting a bytearray to a bytes still require a memory copy...

Why do you want to optimize the pure Python FileIO?

> Why do you want to optimize the pure Python FileIO?
I gave more examples than FileIO in this issue.

Without more interested, I chose to defer this issue. Feel free to reopen it if you need it for more use cases, or if you are interested to implement it.