detecting newline character

Sat Apr 23 15:33:33 EDT 2011

Chris Rebert wrote:
> On Sat, Apr 23, 2011 at 11:09 AM, Daniel Geržo <danger at rulez.sk> wrote:
>> I need to detect the newline characters used in the file I am reading.
>> For this purpose I am using the following code:
>>>> def _read_lines(self):
>> with contextlib.closing(codecs.open(self.path, "rU")) as fobj:
>> fobj.readlines()
>> if isinstance(fobj.newlines, tuple):
>> self.newline = fobj.newlines[0]
>> else:
>> self.newline = fobj.newlines
>>>> This works fine, if I call codecs.open() without encoding argument; I am
>> testing with an ASCII enghlish text file, and in such case the
>> fobj.newlines is correctly detected being as '\r\n'. However, when I call
>> codecs.open() with encoding='ascii' argument, the fobj.newlines is None
>> and I can't figure out why that is the case. Reading the PEP at
>> http://www.python.org/dev/peps/pep-0278/ I don't see any reason why would
>> I end up with newlines being None after I call readlines().
>>>> Anyone has an idea?
>> I would hypothesize that it's an interaction bug between universal
> newlines and codecs.open().
>> […]
> I would speculate that the upshot of this is that codecs.open() ends
> up calling built-in open() with a nonsense `mode` of "rUb" or similar,
> resulting in strange behavior.
>> If this explanation is correct, then there are 2 bugs:
> 1. Built-in open() should treat "b" and "U" as mutually exclusive and
> reject mode strings which involve both.
> 2. codecs.open() should either reject modes involving "U", or be fixed
> so that they work as expected.

You might be correct that it is a bug (already fixed in versions newer than 
2.5), since codecs.open() from my Python 2.6 reads as follows:
def open(filename, mode='rb', encoding=None, errors='strict', buffering=1):
 """
 …
 """
 if encoding is not None:
 if 'U' in mode:
 # No automatic conversion of '\n' is done on reading and writing
 mode = mode.strip().replace('U', '')
 if mode[:1] not in set('rwa'):
 mode = 'r' + mode
 if 'b' not in mode:
 # Force opening of the file in binary mode
 mode = mode + 'b'
 file = __builtin__.open(filename, mode, buffering)
 if encoding is None:
 return file
 info = lookup(encoding)
 srw = StreamReaderWriter(file, info.streamreader, info.streamwriter, 
errors)
 # Add attributes to simplify introspection
 srw.encoding = encoding
 return srw
-- 
PointedEars