This issue tracker has been migrated to GitHub ,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2012年01月15日 04:29 by ferringb, last changed 2022年04月11日 14:57 by admin.
| Files | ||||
|---|---|---|---|---|
| File name | Uploaded | Description | Edit | |
| closerange-optimization.patch | ferringb, 2012年01月15日 04:43 | for non-windows systems, if /proc/${PID}/fd is available make use of it | review | |
| Messages (12) | |||
|---|---|---|---|
| msg151273 - (view) | Author: Ferringb (ferringb) * | Date: 2012年01月15日 04:29 | |
The current implementation of closerange essentially is a bruteforce invocation of close for every integer in the range.
While this works, it's rather noisy for stracing, and for most invocations, is near a thousand close invocations more than needed.
As such it should be aware of /proc/${PID}/fd, and use that to isolate down just what is actually open, and close that.
|
|||
| msg151274 - (view) | Author: Ferringb (ferringb) * | Date: 2012年01月15日 04:43 | |
Fixed tabs/spaces... |
|||
| msg151275 - (view) | Author: Ross Lagerwall (rosslagerwall) (Python committer) | Date: 2012年01月15日 04:48 | |
Thanks for the patch. However, this cannot as far as I understand be used for the subprocess implementation due to the limitation of what can be called after a fork() and before an exec(). Take a look at #8052 for some more discussion of this. |
|||
| msg151287 - (view) | Author: Georg Brandl (georg.brandl) * (Python committer) | Date: 2012年01月15日 12:38 | |
fwiw, s/MSDOS_WINDOWS/MS_WINDOWS/. |
|||
| msg151289 - (view) | Author: Gregory P. Smith (gregory.p.smith) * (Python committer) | Date: 2012年01月15日 15:25 | |
Reopening. Comments added to the code review. This issue is independent of the subprocess module issue in #8052. The _posixsubprocess.c has its own fd closing loop. http://hg.python.org/cpython/file/050c07b31192/Modules/_posixsubprocess.c#l118 |
|||
| msg192738 - (view) | Author: Ronald Oussoren (ronaldoussoren) * (Python committer) | Date: 2013年07月09日 10:53 | |
Two small technical comments:
1) I'd add a configure or compile-time check to determine if the procfs
interface might be available. I don't like probing for features that
you know are not available.
2) MacOSX has simular functionality using /dev/fd instead of
/proc/${PID}/fd (and other BSD systems might have this as well)
|
|||
| msg192739 - (view) | Author: Christian Heimes (christian.heimes) * (Python committer) | Date: 2013年07月09日 11:30 | |
In case someone is wondering if the approach really reduces the amount of syscalls: yes, it does. readdir() doesn't do a syscall for each entry. On Linux it uses the internal syscall getdents() to fill a buffer of directory entry structs. http://man7.org/linux/man-pages/man2/getdents.2.html On my system os.listdir() does four syscalls: $ strace python -c "import os; os.listdir('/home/heimes')" openat(AT_FDCWD, "/home/heimes", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3 getdents(3, /* 381 entries */, 32768) = 12880 getdents(3, /* 0 entries */, 32768) = 0 close(3) On Linux you can also use /proc/self/fd instead of /proc/YOURPID/fd. Other operating systems have different APIs to get a list of open FDs. AFAK /dev/fd is static on FreeBSD and Mac OS X: FreeBSD: http://www.manualpages.de/FreeBSD/FreeBSD-7.4-RELEASE/man3/kinfo_getfile.3.html Darwin / Mac OS X: proc_pidinfo() |
|||
| msg192760 - (view) | Author: Gregory P. Smith (gregory.p.smith) * (Python committer) | Date: 2013年07月09日 16:03 | |
_posixsubprocess already uses the Linux getdent64 syscall when available (though for different reasons: readdir is not safe in that context). http://hg.python.org/cpython/file/3f3cbfd52f94/Modules/_posixsubprocess.c#l227 Probing for procfs at configure time could be problematic. It is a virtual filesystem. It is entirely possible for a system to choose not to mount it. It might be reasonable to assume that it "might be present" only if the system had it mounted at compile time but a configure flag to override that might be desirable for some systems (not the Linux systems I usually deal with). If we're going through all of these hoops for closerange: I'd love to see an API exposed in the os module to return a list of open fd's. It is an abstraction nobody should have to write for themselves. |
|||
| msg192776 - (view) | Author: STINNER Victor (vstinner) * (Python committer) | Date: 2013年07月09日 20:16 | |
FreeBSD and other OSes provide closefrom(). Why not exposing this function which is probably implemented as a single syscall? |
|||
| msg351354 - (view) | Author: STINNER Victor (vstinner) * (Python committer) | Date: 2019年09月09日 07:29 | |
See also bpo-38061: "FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()". |
|||
| msg396819 - (view) | Author: William Manley (wmanley) | Date: 2021年07月01日 16:41 | |
Linux has a close_range syscall since v5.9 (Oct 2020): https://man7.org/linux/man-pages/man2/close_range.2.html |
|||
| msg396963 - (view) | Author: Gregory P. Smith (gregory.p.smith) * (Python committer) | Date: 2021年07月05日 00:19 | |
Code needed in a modern patch: 1) Use the system call if compile time configure detected it may be available. (if we just use syscall() rather than a libc wrapper, a configure check may not be necessary, but various #ifdefs likely are) 2) If (1) produces an error saying it isn't available at runtime (ie: running on a kernel that doesn't support it, regardless of where we were built), fall back to another approach (3) 3) if iterating over /proc/$pid/fd works at runtime, use that; else (4) 4) the existing brute force code. |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022年04月11日 14:57:25 | admin | set | github: 57997 |
| 2021年07月05日 00:19:38 | gregory.p.smith | set | stage: needs patch messages: + msg396963 versions: + Python 3.11, - Python 3.8 |
| 2021年07月01日 16:41:13 | wmanley | set | nosy:
+ wmanley messages: + msg396819 |
| 2019年09月09日 09:26:42 | gregory.p.smith | set | versions: + Python 3.8, - Python 3.4 |
| 2019年09月09日 07:29:20 | vstinner | set | messages: + msg351354 |
| 2013年11月23日 19:31:36 | gregory.p.smith | set | assignee: gregory.p.smith -> |
| 2013年07月09日 20:16:54 | vstinner | set | nosy:
+ vstinner messages: + msg192776 |
| 2013年07月09日 16:03:57 | gregory.p.smith | set | messages: + msg192760 |
| 2013年07月09日 11:31:00 | christian.heimes | set | messages: + msg192739 |
| 2013年07月09日 10:53:29 | ronaldoussoren | set | nosy:
+ ronaldoussoren messages: + msg192738 |
| 2013年07月08日 17:17:15 | christian.heimes | set | nosy:
+ christian.heimes stage: resolved -> (no value) versions: + Python 3.4 |
| 2012年01月15日 15:25:49 | gregory.p.smith | set | status: closed -> open superseder: subprocess close_fds behavior should only close open fds -> assignee: gregory.p.smith nosy: + gregory.p.smith messages: + msg151289 resolution: duplicate -> |
| 2012年01月15日 12:38:01 | georg.brandl | set | nosy:
+ georg.brandl messages: + msg151287 |
| 2012年01月15日 09:15:30 | neologix | set | status: open -> closed resolution: duplicate superseder: subprocess close_fds behavior should only close open fds stage: resolved |
| 2012年01月15日 04:48:53 | rosslagerwall | set | nosy:
+ rosslagerwall messages: + msg151275 |
| 2012年01月15日 04:43:56 | ferringb | set | files: - closerange-optimization.patch |
| 2012年01月15日 04:43:26 | ferringb | set | files:
+ closerange-optimization.patch messages: + msg151274 |
| 2012年01月15日 04:29:24 | ferringb | create | |