homepage

The following function calls should raise a TypeError instead. Encoding
functions are fine (they only accept str).
>>> import codecs
>>> codecs.utf_8_decode('aa')
('aa', 2)
>>> codecs.utf_8_decode('éé')
('éé', 4)
>>> codecs.latin_1_decode('éé')
('éé', 4)

Patch replacing "s*" parsing format by "y*" for:
 - utf_7_decode()
 - utf_8_decode()
 - utf_16_decode()
 - utf_16_le_decode()
 - utf_16_be_decode()
 - utf_16_ex_decode()
 - utf_32_decode()
 - utf_32_le_decode()
 - utf_32_be_decode()
 - utf_32_ex_decode()
 - unicode_escape_decode()
 - raw_unicode_escape_decode()
 - latin_1_decode()
 - ascii_decode()
 - charmap_decode()
 - mbcs_decode()
Using run_tests.sh, all tests are ok (with 19 skipped tests). I guess 
that there is not tests for all these functions :-/
Note: codecs documentation was already correct:
.. method:: Codec.decode(input[, errors])
 (...)
 *input* must be a bytes object or one which provides the read-only 
character
 buffer interface -- for example, buffer objects and memory mapped 
files.

On 2009年01月08日 01:59, STINNER Victor wrote:
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> Patch replacing "s*" parsing format by "y*" for:
> - utf_7_decode()
> - utf_8_decode()
> - utf_16_decode()
> - utf_16_le_decode()
> - utf_16_be_decode()
> - utf_16_ex_decode()
> - utf_32_decode()
> - utf_32_le_decode()
> - utf_32_be_decode()
> - utf_32_ex_decode()
> - latin_1_decode()
> - ascii_decode()
> - charmap_decode()
> - mbcs_decode()
These are fine.
> - unicode_escape_decode()
> - raw_unicode_escape_decode()
These changes are in line with their C API codec interfaces as well,
but those particular codecs could well also be made to work on Unicode
input, since unescaping can well be applied to Unicode as well.
I'll probably open a new item for this.
> Using run_tests.sh, all tests are ok (with 19 skipped tests). I guess 
> that there is not tests for all these functions :-/
The mbcs codec is only available on Windows.
All others are tested by test_codecs.py.
Which ones are skipped in your case ?
> Note: codecs documentation was already correct:
> 
> .. method:: Codec.decode(input[, errors])
> (...)
> *input* must be a bytes object or one which provides the read-only 
> character
> buffer interface -- for example, buffer objects and memory mapped 
> files.
That's not entirely correct: codecs are allowed to accept any
object type and can also return any object type. It up to them
to decide, e.g. a codec may accept both bytes and Unicode input
and always generate Unicode output when decoding.
I guess I have to review those doc changes...

The patch is probably fine, but it would be nice to add some unit tests
for the new behaviour.

On 2009年01月13日 14:14, Antoine Pitrou wrote:
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
> The patch is probably fine, but it would be nice to add some unit tests
> for the new behaviour.
+1 from my side.
Thanks for the patch, Viktor.

New patch:
 - Leave unicode_escape_decode() and raw_unicode_escape_decode() 
unchanged (still accept unicode string)
 - Test changes (reject unicode for most codecs decode functions)
 - Write tests for unicode_escape_decode() and 
raw_unicode_escape_decode() (there was no test for these functions)

Committed in r68855, r68856. Thanks!

IMO, Modules/cjkcodecs/multibytecodec.c should be changed as well.

Right, I hadn't thought of that.

"Fixing" multibytecodec.c produces a TypeError in the following test:
 def test_errorcallback_longindex(self):
 dec = codecs.getdecoder('euc-kr')
 myreplace = lambda exc: ('', sys.maxsize+1)
 codecs.register_error('test.cjktest', myreplace)
 self.assertRaises(IndexError, dec,
 'apple\x92ham\x93spam', 'test.cjktest')
TypeError: decode() argument 1 must be bytes or buffer, not str
Since the test is meant to test recovery from a misbehaving callback, I
guess the type of the input string is not really important and can be
changed to a bytes string instead. What do you think?
(in any case, here is a patch)

The patch looks good. 
I think the missing b in test_errorcallback_longindex is an overlook
when the tests were updated for py3k. You are right to change the test.

Committed in r68857, r68858.