homepage

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Undocumented KeyError from os.path.expanduser
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Python startup should not require passwd entry
View: 10496
Assigned To: Nosy List: acdha, docs@python, serhiy.storchaka, terry.reedy, vstinner, zaytsev
Priority: normal Keywords:

Created on 2014年01月07日 16:15 by acdha, last changed 2022年04月11日 14:57 by admin. This issue is now closed.

Messages (7)
msg207552 - (view) Author: Chris Adams (acdha) Date: 2014年01月07日 16:15
This is a more general version of #10496: os.path.expanduser is documented as returning the unmodified string if it cannot be expanded (http://docs.python.org/3/library/os.path.html#os.path.expanduser) but there's one edge case where this can fail: when running without HOME in the environment:
https://github.com/python/cpython/blob/3.3/Lib/posixpath.py#L259-L263
In this case, pwd.getpwuid is called and although it's not documented as such, it raises a KeyError anytime the underlying POSIX getpwuid() call fails:
https://github.com/python/cpython/blob/3.3/Modules/pwdmodule.c#L119-L120
Reproducing this deliberately:
cadams@Io:~ $ sudo -H python
Python 2.7.5 (default, Aug 25 2013, 00:04:04) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import posix
>>> posix.setuid(1001)
>>> posix.seteuid(1001)
>>> os.environ.pop('HOME')
'/var/root'
>>> os.path.expanduser("~")
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 269, in expanduser
 userhome = pwd.getpwuid(os.getuid()).pw_dir
KeyError: 'getpwuid(): uid not found: 1001'
Context:
* https://github.com/toastdriven/django-haystack/issues/924
* https://github.com/kennethreitz/requests/issues/1846.
msg207555 - (view) Author: Yury V. Zaytsev (zaytsev) Date: 2014年01月07日 16:38
Amusingly, it's also the case on BG/Q compute nodes. Only this morning, I cooked up the following patch:
--- a/Lib/posixpath.py
+++ b/Lib/posixpath.py
@@ -237,7 +237,11 @@ def expanduser(path):
 if i == 1:
 if 'HOME' not in os.environ:
 import pwd
- userhome = pwd.getpwuid(os.getuid()).pw_dir
+ try:
+ userhome = pwd.getpwuid(os.getuid()).pw_dir
+ except KeyError:
+ import warnings
+ warnings.warn('Unable to auto-detect $HOME, export it or functionality such as user-installed modules will not work!', RuntimeWarning)
 else:
 userhome = os.environ['HOME']
 else:
I think it's a bit harsh to fail completely, but returning silently also doesn't sound like a good idea, so I decided to issue a warning instead; the wording should be probably tweaked by a native speaker of English though.
What do you think?
msg207579 - (view) Author: Chris Adams (acdha) Date: 2014年01月07日 18:29
Other than hoisting the warnings import to the top (PEP-8) that seems entirely reasonable to me. The user report which we got was confusing because it was non-obvious where it came from - a warning or other pointer would have helped the original reporter get an idea as to what was failing earlier.
The other two changes I'd suggest would be documentation updates:
os.path.expanduser noting that KeyError may be raised when HOME is undefined and pwd.getpwuid fails for the active user
pwd.getpwuid noting that KeyErrors will be raised when the underlying POSIX call fails
msg227746 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014年09月28日 01:49
I think the doc should be considered correct and posixpath buggy. The ntpath code checks for the existence of HOME and substitutes before returning the arg unchanged, as documented.
 if 'HOME' in os.environ:
 userhome = os.environ['HOME']
 elif 'USERPROFILE' in os.environ:
 userhome = os.environ['USERPROFILE']
 elif not 'HOMEPATH' in os.environ:
 return path
 <create userhome from HOMEDRIVE and HOMEPATH
--
The pwd doc already says " KeyError is raised if the entry asked for cannot be found.", so this is not an issue.
msg227953 - (view) Author: Chris Adams (acdha) Date: 2014年09月30日 17:16
I agree that making the code match the docs would be preferable – an unexpected KeyError might be easier to track down that way but it'd still surprise most developers.
re:pwd docs, the formatting in https://hg.python.org/cpython/file/8e9df3414185/Doc/library/pwd.rst makes it easy to miss the KeyError note – it's at the end of a paragraph after a table and since "entry" is a generic term it's possible to misread it as applying to fields within the record rather than the record itself. I think it would at least be worth adding a newline and perhaps explicitly noting that it applies to any function which returns a password database entry. 
Given that there are only two functions which it applies to, it might be best to simply add a note to each of them for people who use "pydoc pwd.getpwduid" or an IDE and don't see the module-level docs but that's getting a bit contrived.
msg331924 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018年12月16日 11:34
The behavior was changed in f2f4555d8287ad217a1dba7bbd93103ad4daf3a8 as a part of issue10496.
msg331938 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018年12月16日 21:59
Even if it's not exactly a duplicate of bpo-10496, the reported bug has been fixed. I close the issue.
History
Date User Action Args
2022年04月11日 14:57:56adminsetgithub: 64363
2018年12月16日 21:59:17vstinnersetstatus: pending -> closed
superseder: Python startup should not require passwd entry
messages: + msg331938

resolution: duplicate
stage: test needed -> resolved
2018年12月16日 11:34:46serhiy.storchakasetstatus: open -> pending
nosy: + serhiy.storchaka, vstinner
messages: + msg331924

2014年09月30日 17:16:21acdhasetmessages: + msg227953
2014年09月28日 01:49:08terry.reedysetnosy: + terry.reedy
messages: + msg227746

assignee: docs@python ->
components: - Documentation
stage: needs patch -> test needed
2014年09月27日 14:29:14serhiy.storchakasetversions: + Python 3.5, - Python 3.3
nosy: + docs@python

assignee: docs@python
components: + Documentation
stage: needs patch
2014年01月07日 18:29:08acdhasetmessages: + msg207579
2014年01月07日 16:38:31zaytsevsetversions: + Python 3.4
nosy: + zaytsev

messages: + msg207555

type: behavior
2014年01月07日 16:15:27acdhacreate

AltStyle によって変換されたページ (->オリジナル) /