homepage

On Fedora 8 and 10 using Python 2.5.1 and 2.5.2 (64 bit):
$ grep nfsnobody /etc/passwd
nfsnobody:x:4294967294:4294967294:Anonymous NFS
User:/var/lib/nfs:/sbin/nologin
So the UID of nfsnobody is 4294967294 (-2 if viewed as signed 32-bit int).
>>> import pwd, os
>>> print pwd.getpwnam('nfsnobody').pw_uid
4294967294
>>> os.chown('some file', pwd.getpwnam('nfsnobody').pw_uid,
pwd.getpwnam('nfsnobody').pw_gid)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
OverflowError: signed integer is greater than maximum
The reason for this error is that os.chown uses the "i" format to
convert the second and third arguments. But the valued do not fit in a
32-bit signed integer. uid_t and gid_t are defined as unsigned
quantities on this system.
The bug does not occur on 32 bit Fedora since there the uid and gid of
nfsnobody are 65534.

This was already corrected by r61540, and will be released with 2.5.3
and 2.6.2.
BUT -- fchown() was not modified. patch is similar to r61540, but I
don't know how to test:
Index: posixmodule.c
===================================================================
--- posixmodule.c	(revision 67068)
+++ posixmodule.c	(working copy)
@@ -1902,9 +1902,10 @@
 static PyObject *
 posix_fchown(PyObject *self, PyObject *args)
 {
-	int fd, uid, gid;
+	int fd;
+	long uid, gid;
 	int res;
-	if (!PyArg_ParseTuple(args, "iii:chown", &fd, &uid, &gid))
+	if (!PyArg_ParseTuple(args, "ill:fchown", &fd, &uid, &gid))
 		return NULL;
 	Py_BEGIN_ALLOW_THREADS
 	res = fchown(fd, (uid_t) uid, (gid_t) gid);

2^31-2 doesn't fit in long on 32 bits CPU. On my Linux box, uid_t is an 
unsigned integer (32 bits unsigned integer). Why not using "unsigned 
int"? chown prototype:
 int chown(const char *path, uid_t owner, gid_t group);

I'm sure you meant 2^32-2 ;-).
The "fix" to use long doesn't seem right to me either. unsigned int is
a better match with uid_t and gid_t.

There is not only chown! There are also lchown(), fchown(), stat() and 
lstat().
Attached patch:
 - use "unsigned int" to store Windows st_uid/st_gid in the win32_stat 
structure
 - use PyLong to store an unsigned group in the stat result (because 
2^32-2 doesn't fit in a PyInt)
 - use "unsigned int" (instead of long) for uid/gid in chown, lchown 
and fchown
Note: *chown() accepts -1 for an argument, eg. chown("file", 10, -1) 
only changes the user group.

I tested my patch (for Python trunk) on Ubuntu Gutsy with a 32 bits 
CPU: chown(), lchwon(), fchown(), stat() and lstat() works as expected 
(support identifier > 2**31-1).

Anyone available for a patch review? Do you need anything else: tests, 
documentation updates, etc.?

The documentation for PyArg_ParseTuple states that the "I" code doesn't
do any overflow checking. Does it mean the input parameters can be
silently truncated?

pitrou> The documentation for PyArg_ParseTuple states that the 
pitrou> "I" code doesn't do any overflow checking. Does it mean 
pitrou> the input parameters can be silently truncated?
I'm sorry, but you're right :-) My patch was wrong. New patch 
introduces functions parse_uid() and parse_gid() doing the right thing 
to get an unsigned uid and gid. I reused the parsing code from 
posix_setgroups().

Oops, i forgot to ignore space changes in my diff... new patch.

New patch using PyNumber_Index()+PyLong_AsLong() for parse_uid() and 
parse_gid() as proposed by antoine (on IRC).
Changes between python trunk and posix_unsigned_uid-3.patch:
- os.chown() and os.fchown() accepts uid and gid > 2^31: identifiers 
can be in [-1; 2^32-1] (-1 means: don't change uid/gid)
- fix os.*stat(): st_uid and st_gid are unsigned long integers (in [0; 
2^32-1]) instead of signed integers
- os.chown(), os.fchown() and os.setgroups() accepts any objects 
implementing __index__() method (instead of just int/long) and display 
the valid range on error ("group id is to big" => "group id is not in 
range [-1; 2^32-1]")

I'm not sure hardcoding "2^32 - 1" in the error message is a good idea.
I also think it would be nice to have tests for the "-1" special value:
- check that chown(-1, current_gid) succeeds (and leaves the uid intact)
- check that chown(current_uid, -1) succeeds (and leaves the gid intact)

See also #3586: "pwd.getpwnam('nobody') produces incorrect results if
sizeof(uid_t) < sizeof(long)" (now closed).

Here is a simplified version of patch from issue2005. Support of uid_t/gid_t larger than unsigned long dropped. Signed uid_t/gid_t support dropped. The converters moved to Objects/longobject.c. Some tests added.

I don't think _Py_Uid_Converter and _Py_Gid_Converter belong in Objects/longobject.c. Is there a better place to put these? (Though _PyLong_FromUid and _PyLong_FromGid seem fine.)

> Though _PyLong_FromUid and _PyLong_FromGid seem fine.
Hmm. I take that back. I don't think any of this really belongs in Objects/longobject.c. Right now that module contains entirely portable C code that knows almost nothing about operating systems. In particular, the knowledge that -1 has special meaning doesn't really have a place in the longobject implementation.
Is it possible to just use PyLong_FromLong / PyLong_FromLongLong etc. depending on the precision of uid_t / gid_t?

> Hmm. I take that back. I don't think any of this really belongs in Objects/longobject.c.
I am also not happy with that. Originally they were in Modules/posixmodule.c. However where should we place the declarations? We have not posixmodule.h. PyLong_FromPid()/PyLong_AsPid() placed right in Include/longobject.h. Are you have any suggestions?
> Is it possible to just use PyLong_FromLong / PyLong_FromLongLong etc. depending on the precision of uid_t / gid_t?
No. uid_t/gid_t can be large than maximal signed long, and long long can be not defined. See issue2005 where more complex patch proposed.

Perhaps we should add a new header file for declarations of converters to/from all platform specific integers (pid_t, time_t, uid_t, gid_t).

Do we know of any systems where none of int, long, long long (if defined) or their unsigned variants match gid_t / uid_t?

A new header file sounds reasonable. We could then move the PID stuff out of longobject.h

> Do we know of any systems where none of int, long, long long (if defined) or their unsigned variants match gid_t / uid_t?
No, I don't.

The value -1 is used is some functions like chown() (to only change the
user, only the group or none, to check if you are allowed or not). How do
you handle this value?
Le 24 nov. 2012 19:29, "Serhiy Storchaka" <report@bugs.python.org> a écrit :
>
> Serhiy Storchaka added the comment:
>
> Here is a simplified version of patch from issue2005. Support of
> uid_t/gid_t larger than unsigned long dropped. Signed uid_t/gid_t support
> dropped. The converters moved to Objects/longobject.c. Some tests added.
>
> ----------
> components: +Extension Modules, Interpreter Core -Library (Lib)
> nosy: +serhiy.storchaka
> stage: -> patch review
> versions: +Python 3.4
> Added file: http://bugs.python.org/file28096/posix_uid_gid_conv_3.patch
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue4591>
> _______________________________________
>

Patch updated. The converters declaration moved to the new file Modules/posixmodule.h (Modules/grpmodule.c, Modules/posixmodule.c, Modules/pwdmodule.c, and Modules/signalmodule.c use it) and the implementation moved to Modules/posixmodule.c.

> The value -1 is used is some functions like chown() (to only change the
user, only the group or none, to check if you are allowed or not). How do
you handle this value?
Yes, there is a special case for -1. Look at the code.

Are there any objections, nitpicks or suggestions?

No objections here.

New changeset b322655a4a88 by Serhiy Storchaka in branch '3.3':
Issue #4591: Uid and gid values larger than 2**31 are supported now.
http://hg.python.org/cpython/rev/b322655a4a88
New changeset 94256de0aff0 by Serhiy Storchaka in branch 'default':
Issue #4591: Uid and gid values larger than 2**31 are supported now.
http://hg.python.org/cpython/rev/94256de0aff0 

Looks like this broke some buildbots:
======================================================================
ERROR: test_chown (test.test_shutil.TestShutil)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/home/buildbot/buildarea/3.3.krah-freebsd/build/Lib/test/test_shutil.py", line 1209, in test_chown
 shutil.chown(filename, 3.14)
 File "/usr/home/buildbot/buildarea/3.3.krah-freebsd/build/Lib/shutil.py", line 1024, in chown
 os.chown(path, _user, _group)
PermissionError: [Errno 1] Operation not permitted: '/tmp/tmp3f82m0/tmp_ttyx5'

New changeset 4ef048f4834e by Serhiy Storchaka in branch '3.3':
Reject float as uid or gid.
http://hg.python.org/cpython/rev/4ef048f4834e
New changeset 3fdcffdfd3e6 by Serhiy Storchaka in branch 'default':
Reject float as uid or gid.
http://hg.python.org/cpython/rev/3fdcffdfd3e6 

Not technically the topic of this issue, but should we just lock
down _Py_Uid_Converter() to ints?
This is still accepted:
os.setuid(Decimal("1000.2"))

> Not technically the topic of this issue, but should we just lock
> down _Py_Uid_Converter() to ints?
The right way to do it should be to use PyNumber_Index().
(perhaps we need as PyNumber_AsLongIndex() to do the direct conversion
to a C long, it would make things easier in many cases)

> (perhaps we need as PyNumber_AsLongIndex() to do the direct conversion
> to a C long, it would make things easier in many cases)
+1, but drop the 'Index' part of the name, to match the existing PyNumber_AsSsize_t.

> > (perhaps we need as PyNumber_AsLongIndex() to do the direct conversion
> > to a C long, it would make things easier in many cases)
> 
> +1, but drop the 'Index' part of the name, to match the existing
> PyNumber_AsSsize_t.
There's already PyNumber_Long() (which doesn't use __index__), hence the
possible confusion.
Note there is also PyIndex_Check(), so we could call it
PyIndex_AsLong().

Hmm; good point. Well, +1 to the functionality, anyway; I'll leave the discussion about the name.

There is a problem with test_pwd on 64-bit platform. It expects KeyError on pwd.getpwuid(sys.maxsize). Actually the test is not looks robust. On 32-bit platform sys.maxsize = 2**31-1 < 2**32 and this value only by chance was not in the user database. On 64-bit platform sys.maxsize > 2**32 and in old code it was wrapped to 2**31-1 C unsigned int value and this value only by chance was not in the user database. New code doesn't wrap the value to C unsigned int and raises an overflow exception.
What should I change, a test or a raised exception?

+1 for PyIndex_AsLong()

> (perhaps we need as PyNumber_AsLongIndex() to do the direct conversion
to a C long, it would make things easier in many cases)
In this case we need PyNumber_AsLongAndOverflowIndex() and PyNumber_AsUnsignedLongIndex(). And a lot of other parallel functions for other cases. This can exponentially increase a number of functions.

> Not technically the topic of this issue, but should we just lock
> down _Py_Uid_Converter() to ints?
I just copied this code from PyArg_ParseTuple*() for 'l' format.
> This is still accepted:
>
> os.setuid(Decimal("1000.2"))
Any C implemented function which parses an integer argument with PyArg_ParseTuple*() accepts decimals.
>>> chr(65.2)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: integer argument expected, got float
>>> chr(Decimal('65.2'))
'A'
I think this is an offtopic for this issue.

Serhiy Storchaka <report@bugs.python.org> wrote:
> > Not technically the topic of this issue, but should we just lock
> > down _Py_Uid_Converter() to ints?
> 
> I just copied this code from PyArg_ParseTuple*() for 'l' format.
> > os.setuid(Decimal("1000.2"))
I know that this behavior wasn't introduced by you. It's perfectly fine
to use PyArg_ParseTuple() for guidance or to try to preserve the existing
behavior.
> >>> chr(Decimal('65.2'))
> 'A'
I just happen to think that PyArg_ParseTuple() is wrong here and we shouldn't
enable this "feature" in new code. Fixing these issues one by one is probably
the best way forward.

> There is a problem with test_pwd on 64-bit platform.
Fixed in changesets a0983e46feb1 and 1e9fa629756c: Raise KeyError instead of OverflowError when getpwuid's argument is out of uid_t range.

> In this case we need PyNumber_AsLongAndOverflowIndex() and
> PyNumber_AsUnsignedLongIndex(). And a lot of other parallel functions
> for other cases. This can exponentially increase a number of functions.
I don't think it will be exponential :-)

New changeset 3893ab574c55 by Serhiy Storchaka in branch '3.2':
Issue #4591: Uid and gid values larger than 2**31 are supported now.
http://hg.python.org/cpython/rev/3893ab574c55
New changeset 035cbc654889 by Serhiy Storchaka in branch '2.7':
Issue #4591: Uid and gid values larger than 2**31 are supported now.
http://hg.python.org/cpython/rev/035cbc654889 

Revision 035cbc654889 added useless '#ifndef Py_LIMITED_API' check in 2.7 branch.
Support for Py_LIMITED_API was introduced in Python 3.2.