homepage

The following code displays "Xbc" using io, and "bc" using _pyio (or an unbuffered file, e.g. io.FileIO):
-------------
import _pyio, io
with io.BytesIO(b'abc') as raw:
 #with _pyio.BufferedRandom(raw) as f:
 with io.BufferedRandom(raw) as f:
 f.write(b"X")
 print("pos?", f.tell(), raw.tell())
 print(f.read())
-------------
I expect .write() to change the file position, and so "bc" must be the correct result, not "Wbc".
_pyio.BufferedRandom overrides its write method, whereas io.BufferedRandom doesn't.
I already noticed the implement difference of BufferedRandom.write(), but I don't remember if I reported it or not!?

> I already noticed the implement difference of BufferedRandom.write(),
> but I don't remember if I reported it or not!?
Yes, I already reported the difference:
http://bugs.python.org/issue12062#msg135876 

io_interlaced_read_write.patch:
 - add interlaced read/write tests for BufferedRandom and BufferedRWPair
 - _pyio: move "undo readahead" code into BufferedReader.flush()
 - io: BufferedRandom.flush() doesn't undo readahead if the write buffer is empty, so it's possible to call it in read methods without flusing the readahead buffer when it's not needed
 - read(), read1(), readinto(), peek() calls writer.flush()
 - write() calls reader.flush() (undo readahead)
TODO:
 - tests BufferedRWPair with read-only + write-only files because I'm not sure if my tests using readable and writeable methods
 - _pyio: undo the readahead in BufferedReader.flush() is no perfect, because BufferedReader is supposed to be read-only. I choosed that to factorize the code between BufferedRandom and BufferedRWPair
 - what happens if a write occurs during _pyio.BufferedReader.flush()? "if self._read_buf: <write occurs> with self._read_lock: ...". We may protect the read of self._read_buf with the read lock.
My patch tries to fix interlaced read-write by always calling flush(), but I am not sure that it doesn't change read-only and write-only cases. There are maybe some unnecessary call to flush().

See also issue #12215: TextIOWrapper has also issues on interlaced read-write.

> My patch tries to fix interlaced read-write by always calling flush(),
Why do you need to call flush()? Can't you read from the buffer?

>> My patch tries to fix interlaced read-write by always calling flush(),
>
> Why do you need to call flush()? Can't you read from the buffer?
Hum, my patch does not always call flush of the reader and the writer. On read, it flushs the writer. On write, it "flushes" the reader (undo readahead).
It is maybe possible to do better (do something faster), but there should be some tricky cases with seek().

I think your expectations for BufferedRWPair are wrong. You should not use BufferedRWPair with the same underlying stream (that's the whole point of BufferedRWPair).

Here is a patch.

New changeset 5330af45f777 by Antoine Pitrou in branch '3.2':
Issue #12213: Fix a buffering bug with interleaved reads and writes that
http://hg.python.org/cpython/rev/5330af45f777
New changeset d7f6391954cf by Antoine Pitrou in branch 'default':
Issue #12213: Fix a buffering bug with interleaved reads and writes that
http://hg.python.org/cpython/rev/d7f6391954cf 

This should be fixed for BufferedRandom. As I said, I don't think BufferedRWPair is buggy.

> You should not use BufferedRWPair with the same underlying stream (that's the whole point of BufferedRWPair).
It might be documented. Something like "Warning: don't use the same 
stream as reader and writer, or the BufferedRWPair becomes inconsistent 
on interlaced read-write" ?

New changeset cf2010e9f941 by Antoine Pitrou in branch '2.7':
Issue #12213: Fix a buffering bug with interleaved reads and writes that
http://hg.python.org/cpython/rev/cf2010e9f941 

New changeset 524896c305ce by Antoine Pitrou in branch '3.2':
Issue #12213: make it clear that BufferedRWPair shouldn't be called with the
http://hg.python.org/cpython/rev/524896c305ce
New changeset a423bd492d6c by Antoine Pitrou in branch 'default':
Issue #12213: make it clear that BufferedRWPair shouldn't be called with the
http://hg.python.org/cpython/rev/a423bd492d6c
New changeset dd4f29e39756 by Antoine Pitrou in branch '2.7':
Issue #12213: make it clear that BufferedRWPair shouldn't be called with the
http://hg.python.org/cpython/rev/dd4f29e39756 

I think it can be closed now.

New changeset 8de8945ebfc3 by Antoine Pitrou in branch '3.2':
Issue #12213: make it clear that BufferedRWPair shouldn't be called with the
http://hg.python.org/cpython/rev/8de8945ebfc3