homepage

I created this issue to group all issues related to the implementation of the PEP 475 "Retry system calls failing with EINTR".
https://www.python.org/dev/peps/pep-0475/
- Issue #23646: the time module, retry sleep()
- Issue #23618: the socket module
- Issue #23285: the io and the os modules, and open()
- Issue #23485: the select and selectors module

In msg196555, Charles-François Natali wrote:
"""
From a cursory look, the main files affected would be:
Modules/fcntlmodule.c
Modules/ossaudiodev.c
Modules/posixmodule.c
Modules/selectmodule.c
Modules/selectmodule.c
Modules/signalmodule.c
Modules/socketmodule.c
Modules/syslogmodule.c
"""
For syslog, I tested the following code and I don't get any InterruptedError. The return type of the C functions openlog(), syslog() and closelog() is void: no result. So I don't see how they could fail because of a signal. Code of my test:
---
import syslog
import signal
hit = 0
def noop(*args):
 global hit
 hit += 1
signal.signal(signal.SIGALRM, noop)
t = 1e-6
nlog = 10**2
signal.setitimer(signal.ITIMER_REAL, t, t)
for i in range(nlog):
 syslog.openlog()
 syslog.syslog("test %s")
 syslog.closelog()
signal.signal(signal.SIGALRM, signal.SIG_IGN)
print(nlog, "logs")
print(hit, "signals")
---

test_eintr.py: program to check if a Python function can fail with InterruptedError. Modify func() to test a function. (The program checks select.select(), currently it fails with InterruptedError.)

I found a way to get a list of functions which can fail with EINTR: search for TEMP_FAILURE_RETRY in the source code of the glibc. TEMP_FAILURE_RETRY:
https://www.gnu.org/software/libc/manual/html_node/Interrupted-Primitives.html#Interrupted-Primitives
Following functions are called with TEMP_FAILURE_RETRY (in glibc code, doc and examples):
- accept, send, sendfile, sendmsg, accept4, connect, sendto, recvmsg
- write, writev, read, readv, pwrite
- fdatasync, fsync
- waitpid
- select, poll
- mq_send, mq_timedsend, mq_receive, mq_timedreceive
- clock_nanosleep, timer_settime, nanosleep
- posix_fallocate
- sem_wait, sem_timedwait, sem_trywait
- aio_suspend
- fcntl(F_SETLK)

stat manual page of SCO documentation says that stat() can fail with EINTR, but I don't see this error in latest Linux manual page. I tested manually, and I failed to see stat() failing with InterruptedError.
http://uw714doc.sco.com/en/man/html.2/stat.2.html
So only a few system calls on the filesystem (open, read, write, close, fsync, ...) can fail with EINTR?

More issues:
- Issue #23707: os.urandom() doesn't handle EINTR correctly
- Issue #23708: Add _Py_read() and _Py_write() functions
- Issue #23709: Refactor ossaudiodev: use _Py_read and _Py_write with the Py_buffer

signal.sigwaitinfo() raises InterruptedError if the wait was interrupted by a signal handler (this handler was for a signal other than one of those in set).
IMO we should also retry sigwaitinfo() if it fails with EINTR.

I searched for "EINTR" in all manual pages of my Fedora 21. I found a long list of functions (filename of manual pages). But I didn't check each manual page.
There are false positive. For example, "stat" is listed, but in fact EINTR is only mentionned in commented text (not visible using the "man" command): "SVr4 documents additional stat() error conditions EINTR, ...".
Another example of false positive with system:
"""
Early drafts required, or allowed, system() to return with errno set to
[EINTR] if it was interrupted with a signal. This error return was removed, and
a requirement that system() not return until the child has terminated was
added. This means that if a waitpid() call in system() exits with errno
set to [EINTR], system() must reissue the waitpid().
"""
Shell command to search for all EINTR:
find /usr/share/man -name "*.gz"|xargs zgrep -H EINTR > EINTR.txt
Shell command to extract file names from the output:
sed 's!.*/!!g' EINTR.txt|sed 's!\.[0-9][a-z]\?m\?\.gz:.*!!g'|sort -u > EINTR_files.txt
Filename of manual pages:
accept
aio_suspend
catclose
catgets
chmod
chown
chroot
clock_nanosleep
close
closedir
connect
curs_getch
dup
endgrent
epoll_wait
exec
execve
fallocate
fchdir
fchmod
fchown
fclose
fcntl
fflush
fgetc
fgetwc
flock
fopen
fork
fprintf
fputc
fputwc
freopen
fseek
fsetpos
fstatvfs
fsync
ftruncate
futex
getch
getgrent
getgrgid
getgrnam
getmsg
getpwent
getpwnam
getpwuid
has_key
inotify
ioctl
io_getevents
lchown
lio_listio
lockf
mknod
mq_open
mq_receive
mq_send
msgop
msgrcv
msgsnd
mvgetch
mvwgetch
nanosleep
open
pause
pclose
poll
posix_fallocate
posix_mem_offset
posix_trace_close
posix_trace_create
posix_trace_get_filter
posix_trace_getnext_event
posix_trace_start
posix_typed_mem_get_info
posix_typed_mem_open
pselect
pthread_atfork
pthread_attr_destroy
pthread_attr_getdetachstate
pthread_attr_getguardsize
pthread_attr_getinheritsched
pthread_attr_getschedparam
pthread_attr_getschedpolicy
pthread_attr_getscope
pthread_attr_getstack
pthread_attr_getstacksize
pthread_barrierattr_destroy
pthread_barrierattr_getpshared
pthread_barrier_destroy
pthread_barrier_wait
pthread_cancel
pthread_cleanup_pop
pthread_condattr_destroy
pthread_condattr_getclock
pthread_condattr_getpshared
pthread_cond_broadcast
pthread_cond_destroy
pthread_cond_init
pthread_cond_timedwait
pthread_create
pthread_detach
pthread_equal
pthread_getconcurrency
pthread_getschedparam
pthread_getspecific
pthread_join
pthread_key_create
pthread_key_delete
pthread_kill
pthread_mutexattr_destroy
pthread_mutexattr_getprioceiling
pthread_mutexattr_getprotocol
pthread_mutexattr_getpshared
pthread_mutexattr_getrobust
pthread_mutexattr_gettype
pthread_mutex_consistent
pthread_mutex_destroy
pthread_mutex_getprioceiling
pthread_mutex_lock
pthread_mutex_timedlock
pthread_once
pthread_rwlockattr_destroy
pthread_rwlockattr_getpshared
pthread_rwlock_destroy
pthread_rwlock_rdlock
pthread_rwlock_timedrdlock
pthread_rwlock_timedwrlock
pthread_rwlock_trywrlock
pthread_rwlock_unlock
pthreads
pthread_setcancelstate
pthread_setschedprio
pthread_sigmask
pthread_spin_destroy
pthread_spin_lock
pthread_spin_unlock
pthread_tryjoin_np
ptrace
putmsg
read
recv
recvfrom
recvmsg
request_key
restart_syscall
scanf
select
select_tut
semop
sem_open
sem_timedwait
sem_trywait
sem_wait
send
sendmsg
sendto
shm_open
sigaction
sighold
siginterrupt
signal
sigpause
sigsuspend
sigtimedwait
sigvec
sigwaitinfo
spu_run
stat
statfs
statvfs
strace
system
tcdrain
tcsetattr
tmpfile
truncate
ualarm
ungetch
unlink
usleep
ustat
wait
wait4
waitid
wgetch
write
Not interesting for Python:
Expect
HTTP::Tiny
Net::HTTP
POSIX
Tcl_AppendAllObjTypes
Tcl_ConvertToType
Tcl_GetObjType
Tcl_RegisterObjType
nfs
perlapio
perlfunc
perlipc
zmq_ctx_destroy
zmq_getsockopt
zmq_msg_recv
zmq_msg_send
zmq_poll
zmq_recv
zmq_recvmsg
zmq_send
zmq_sendmsg
zmq_setsockopt
zmq_term
zshmodules

unittest_setitimer.patch: hack unittest.TestCase.run() to inject signals every 1 ms, send the first signal in 100 ms. It should help to detect functions which doesn't retry on EINTR.

New issue #23715: handle EINTR in the signal module.

pthread_* are false positive, they cannot fail with EINTR:
"""
Pthreads function return values
 Most pthreads functions return 0 on success, and an error number of failure. Note that the pthreads functions do not set errno. For each of the pthreads functions that can return an error, POSIX.1-2001 specifies that the function can never fail with the error EINTR.
"""
Example of pthread_create:
"""
The pthread_create() function shall not return an error code of [EINTR].
"""
pthread_atfork
pthread_attr_destroy
pthread_attr_getdetachstate
pthread_attr_getguardsize
pthread_attr_getinheritsched
pthread_attr_getschedparam
pthread_attr_getschedpolicy
pthread_attr_getscope
pthread_attr_getstack
pthread_attr_getstacksize
pthread_barrierattr_destroy
pthread_barrierattr_getpshared
pthread_barrier_destroy
pthread_barrier_wait
pthread_cancel
pthread_cleanup_pop
pthread_condattr_destroy
pthread_condattr_getclock
pthread_condattr_getpshared
pthread_cond_broadcast
pthread_cond_destroy
pthread_cond_init
pthread_cond_timedwait
pthread_create
pthread_detach
pthread_equal
pthread_getconcurrency
pthread_getschedparam
pthread_getspecific
pthread_join
pthread_key_create
pthread_key_delete
pthread_kill
pthread_mutexattr_destroy
pthread_mutexattr_getprioceiling
pthread_mutexattr_getprotocol
pthread_mutexattr_getpshared
pthread_mutexattr_getrobust
pthread_mutexattr_gettype
pthread_mutex_consistent
pthread_mutex_destroy
pthread_mutex_getprioceiling
pthread_mutex_lock
pthread_mutex_timedlock
pthread_once
pthread_rwlockattr_destroy
pthread_rwlockattr_getpshared
pthread_rwlock_destroy
pthread_rwlock_rdlock
pthread_rwlock_timedrdlock
pthread_rwlock_timedwrlock
pthread_rwlock_trywrlock
pthread_rwlock_unlock
pthreads
pthread_setcancelstate
pthread_setschedprio
pthread_sigmask
pthread_spin_destroy
pthread_spin_lock
pthread_spin_unlock
pthread_tryjoin_np

New issue #23719: "port test_eintr to Windows".

Well, all the syscalls which can blocki can fail with EINTR, so all
I:O related one.

> Well, all the syscalls which can blocki can fail with EINTR, so all
> I:O related one.
This is also what I expect, but how do you explain that I'm unable to see os.stat() failing with EINTR? I'm testing on ext4 file system. May it occur with other file systems (ex: NFS)? Or maybe not on Linux, but on other platforms?

fstat_not_eintr.py: run this script from a NFS share and unplug the network cable, wait, replug. Spoiler: fstat() hangs until the network is back, CTRL+c or setitimer() don't interrupt it.
By the way, it looks like the itimer is interrupted during fstat!
Extract of strace output:
---
<cable unplugged>
rt_sigreturn() = -1 EINTR (Interrupted system call)
select(0, NULL, NULL, NULL, {0, 100000}) = ? ERESTARTNOHAND (To be restarted if no handler)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
rt_sigreturn() = -1 EINTR (Interrupted system call)
write(1, "fstat...\n", 9) = 9
fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
<cable replugged>
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
rt_sigreturn() = 0
---
The fstat() took 3 seconds, I expected 30 SIGALRM, but none occurred during fstat().

> fstat_not_eintr.py: run this script from a NFS share and unplug the network cable, wait, replug. Spoiler: fstat() hangs until the network is back, CTRL+c or setitimer() don't interrupt it.
You have to mount the share with the eintr option.
Getting EINTR on a local FS is probably more challenging.

Charles-François Natali added the comment:
> You have to mount the share with the eintr option.
Oh, I didn't know this option. Unlikely, it looks like the option was
deprecated something like 7 years ago, in Linux kernel 2.6.25.
Related commits:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/nfs?id=d33e4dfeab5eb0c3c1359cc1a399f2f8b49e18de
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/nfs?id=c381ad2cf2d5dcd3991bcc8a18fddd9d5c66ccaa
The patch on the NFS mailing list:
http://comments.gmane.org/gmane.linux.nfs/20011
> Getting EINTR on a local FS is probably more challenging.
Another option would be to play with FUSE. But I'm not convinced that
it's useful if fstat() never fails EINTR on filesystems used in
production.

New changeset b7c0137cccbe by Victor Stinner in branch 'default':
Issue #23648: Document the PEP 475 in the "Porting to Python 3.5" section and
https://hg.python.org/cpython/rev/b7c0137cccbe 

inject_signals.patch: Quick-and-dirty patch to send signals (SIGALRM) every milliseconds when running unit tests.
On Linux, results are very good. Only multiprocessing fails because of the issue #23618 (socket connect) which has a pending patch. I hope that we will quickly reach a point where the full test suite completes without failure while sending millions of signals to it!

New issue #23836: Enhance faulthandler to handle better reentrant calls.

New issue #23851: _posixsubprocess retries close() on EINTR.

New changeset 66e4ef9a7467 by Victor Stinner in branch 'default':
Issue #23648: Complete the list of modified functions for the PEP 475
https://hg.python.org/cpython/rev/66e4ef9a7467 

New issue #23853: handle EINTR in the ssl module.

> New changeset 66e4ef9a7467 by Victor Stinner in branch 'default':
> Issue #23648: Complete the list of modified functions for the PEP 475
> https://hg.python.org/cpython/rev/66e4ef9a7467
>
> + * special cases: :func:`os.close` and :func:`os.dup2` now ignore
> + :py:data:`~errno.EINTR` error, the syscall is not retried (see the PEP
> + for the rationale)
But os.dup2 is not mentioned in PEP 475.

> But os.dup2 is not mentioned in PEP 475.
Currently, What's in Python 3.5 is more complete than the PEP 475. Another example, the PEP doesn't mention signal.sigtimedwait(). The PEP should be updated when the implementation will be finished.

I updated the list of modified functions in the PEP 475.
Except of the issue #23719, all other issues related to the PEP 475 have been fixed. It's time to close this meta issue.