POSIX places an obscure requirement on popen which is like a limited version of close-on-exec: "The popen() function shall ensure that any streams from previous popen() calls that remain open in the parent process are closed in the new child process." if the POSIX-future 'e' mode flag is passed, producing a pipe FILE with FD_CLOEXEC on the underlying pipe, this requirement is automatically satisfied. however, for applications which use multiple concurrent popen pipes but don't request close-on-exec, fd leaks from earlier popen calls to later ones could produce deadlock situations where processes are waiting for a pipe EOF that will never happen. to fix this, iterate through all open FILEs and add close actions for those obtained from popen. this requires holding a lock on the open file list across the posix_spawn call so that additional popen FILEs are not created after the list is traversed. note that it's still possible for another popen call to start and create its pipe while the lock is held, but such pipes are created with O_CLOEXEC and only drop close-on-exec status (when 'e' flag is omitted) under control of the lock.
the newly allocated FILE * has not yet leaked to the application and is only visible to stdio internals until popen returns. since we do not change any fields of the structure observed by libc internals, only the pipe_pid member, locking is not necessary.
popen was special-casing the possibility (only possible when the parent closed stdin and/or stdout) that the child's end of the pipe was already on the final desired fd number, in which case there was no way to get rid of its close-on-exec flag in the child. commit 6fc6ca1a323bc0b6b9e9cdc8fa72221ae18fe206 made this unnecessary by implementing the POSIX-future requirement that dup2 file actions with equal source and destination fd values remove the close-on-exec flag.
this mirrors the stdio_impl.h cleanup. one header which is not strictly needed, errno.h, is left in pthread_impl.h, because since pthread functions return their error codes rather than using errno, nearly every single pthread function needs the errno constants. in a few places, rather than bringing in string.h to use memset, the memset was replaced by direct assignment. this seems to generate much better code anyway, and makes many functions which were previously non-leaf functions into leaf functions (possibly eliminating a great deal of bloat on some platforms where non-leaf functions require ugly prologue and/or epilogue).
signal mask was not being restored after fork, but instead blocked again.
__release_ptc() is only valid in the parent; if it's performed in the child, the lock will be unlocked early then double-unlocked later, corrupting the lock state.
since we target systems without overcommit, special care should be taken that system() and popen(), like posix_spawn(), do not fail in processes whose commit charges are too high to allow ordinary forking. this in turn requires special precautions to ensure that the parent process's signal handlers do not end up running in the shared-memory child, where they could corrupt the state of the parent process. popen has also been updated to use pipe2, so it does not have a fd-leak race in multi-threaded programs. since pipe2 is missing on older kernels, (non-atomic) emulation has been added. some silly bugs in the old code should be gone too.
close was the only cancellation point called from popen, but it left popen with major resource leaks if any call to close got cancelled. the easiest, cheapest fix is just to use a non-cancellable close function.