homepage

As shown in a patch in issue15431, frozen.c does not output the same data on different platforms.
The first difference looks like this (extracted from the patch):
- 101,73,255,255,255,255,0,0,0,0,40,10,0,0,0,117,
+ 101,108,3,0,0,0,255,127,255,127,3,0,40,10,0,0,
On first row, 'I' followed by 0xFFFFFFFF on 8 bytes.
On second row, 'l' followed by 3 followed by 0xFFFFFFFF (in 3 chunks of 15 bits).
The Python number 0xFFFFFFFF is marshalled with TYPE_INT64 when SIZEOF_LONG>4 (Unix 64bit), and with TYPE_LONG on other platforms (32bit, or win64)
The C "long" type has much less importance in 3.x Python, because PyIntObject does not exist anymore.
I suggest to remove this distinction, and allow TYPE_INT64 on all platforms.
I don't see any compatibility issue, on unmarshalling both methods produce the same object.
I'll try to suggest a patch later today.

I realize that uppercase I and lowercase L are easily confused:
On first row, 'I' (=TYPE_INT64) followed by 0xFFFFFFFF on 8 bytes.
On second row, 'l' (=TYPE_LONG) followed by 3 followed by 0xFFFFFFFF (in 3 chunks of 15 bits).

> I suggest to remove this distinction, and allow TYPE_INT64 on all
> platforms.
You have to find a 64-bit C int type that works on all platforms and for which a PyLongAsXXX() function exists, though. Or perhaps you want to restrict yourself to systems which have "long long" and where it is at least 64 bits wide.

My primary goal is to have a stable importlib.h, and currently all developer work on machines where PY_LONG_LONG is >64bit. So the restriction is OK.

I suggest the opposite: never emit TYPE_INT64 at all. For compatibility, we can still support reading it in 3.3 (and drop that support in 3.4).

Here is a patch to stop generating TYPE_INT64. Ok for 3.30b2?

Here is a patch that write TYPE_INT64 on most platforms (where either long or long long is 64bit).
It is admittedly much larger than Martin's...

I can see yet another problem under the hood. Marshalling will output different data on platforms with a different representation of negative numbers. Worse still, these data are non-portable, written on one platform will be read incorrectly on the other. Mark, time for your inquisitor sword.

Are there really platforms which don't use two's complement?
(If there are, I'm not sure to want to share binary data with them. Same for EBCDIC)

Serhiy: this is safe to ignore.

> I suggest the opposite: never emit TYPE_INT64 at all. For
> compatibility, we can still support reading it in 3.3 (and drop
> that support in 3.4).
+1 essentially we maintain the status quo

New changeset 461e84fc8c60 by Martin v. Löwis in branch 'default':
Issue #15466: Stop using TYPE_INT64 in marshal,
http://hg.python.org/cpython/rev/461e84fc8c60 

Removing TYPE_INT64 was indeed the most reasonable choice.

I put the complete removal of TYPE_INT64 into issue15480 

Looks like the Misc/NEWS entry got truncated. Also, does this change need a new PYC magic number (now in Lib/importlib/_bootstrap.py)?

For the truncation, see ec7267fa8b84. I don't see why a new pyc magic number should be necessary. Python continues to support the existing files.

> I don't see why a new pyc magic number should be necessary. Python
> continues to support the existing files.
Good point.