Jaunty i386 popen() misbehaves on x86_64 kernel 2.6.26

Bug Description

SRU justification:

Impact: A bug in the syscall implementation can cause incorrect behavior when trying to use syscalls that are not implemented.

Fix: Cherry pick from upstream to correctly return an error code in that situation.

Testcase: see test below.

---

There is something very wrong with the popen() implementation in Jaunty on i386. It works fine when run on the Jaunty kernel, but when run on the Lenny 2.6.26-1-amd64 kernel, it misbehaves as follows. The output of the subcommand, instead of being fed to the pipe, is fed directly to stdout; the pipe either returns errors or hangs forever.

This causes weird errors in many programs running in a Jaunty i386 chroot on our Lenny build server.

Attached is a small 5-line test program. Run
gcc -static popen-test.c -o popen-test
and then copy popen-test to a Lenny machine to see the problem.

Correct output:
$ uname -r
2.6.28-8-generic
$ ./popen-test
popen returned 0x1ec6400
ferror returned 0
fread returned 19
ferror returned 0
pclose returned 0
Incorrect output:
$ uname -r
2.6.26-1-amd64
$ ./popen-test
popen returned 0x935f688
ferror returned 0
SHOULD NOT DISPLAY

This has been reproduced independently by two of us with two different Jaunty i386 installs and two different Lenny installs.

+++ This bug was initially created as a clone of Bug #481682 +++

Created an attachment (id=330051)
proposed fix

in kernel-2.6.18-128.el5 and prior versions, arch/x86_64/ia32/ia32entry.S hunk of the linux-2.6-misc-utrace-update.patch contains incorrect optimization.
As result, out-of-table 32-bit syscalls on the x86_64 kernel do not return ENOSYS (unless the caller is being ptraced).

For example, glibc-2.9+ popen() goes mad when pipe2 syscall returns its number 331 instead of fail with ENOSYS.
As result, FC10+ 32-bit processes on RHEL5 x86_64 kernel break once popen(3) is called.

--- Additional comment from <email address hidden> on 2009年01月27日 04:29:59 EDT ---

The issue was found while running Fedora 10 containers on an RHEL5+OpenVZ kernel.

Relative OpenVZ bug: http://bugzilla.openvz.org/show_bug.cgi?id=1150

--- Additional comment from <email address hidden> on 2009年02月06日 20:59:52 EDT ---

Created an attachment (id=331182)
test case source

Simple test case, compile with -m32 and run on x86-64 kernel.

The RHEL5 code is the same (broken) as upstream. I'll fix it upstream and then RHEL5 should backport the change so it continues to match upstream.

--- Additional comment from <email address hidden> on 2009年02月06日 21:02:48 EDT ---

Created an attachment (id=331183)
test case source

Test case fixed to exit 0 for correct and nonzero for bug.

Also, I forgot to note that to reproduce you have to have auditd disabled:
/sbin/chkconfig auditd off; reboot

--- Additional comment from <email address hidden> on 2009年02月06日 21:34:13 EDT ---

Created an attachment (id=331187)
fix posted upstream

Wait to see if upstream takes this as is, but almost certain they will.
With s,/x86/,/x86_64/,g this same patch applies to RHEL5 fine.

--- Additional comment from <email address hidden> on 2009年02月07日 04:57:47 EDT ---

> The RHEL5 code is the same (broken) as upstream.

RHEL5 is not broken, I wasn't able to reproduce it. As I understood, this is due to this check:

testl $(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SECCOMP),threadinfo_flags(%r10)
jnz ia32_tracesys

And we have TIF_SYSCALL_AUDIT set.

Also, it's possible to make the patch shorter:

--- a/arch/x86/ia32/ia32entry.S
+++ b/arch/x86/ia32/ia32entry.S
@@ -417,6 +417,7 @@ ENTRY(ia32_syscall)
GET_THREAD_INFO(%r10)
orl $TS_COMPAT,TI_status(%r10)
testl $_TIF_WORK_SYSCALL_ENTRY,TI_flags(%r10)
+ movq $-ENOSYS,RAX-ARGOFFSET(%rsp)
jnz ia32_tracesys
ia32_do_syscall:
cmpl $(IA32_NR_syscalls-1),%eax

I've tested it yesterday, but didn't post upstream yet.

--- Additional comment from <email address hidden> on 2009年02月07日 17:21:41 EDT ---

It is broken. See comment#3 on how to reproduce it.
Shorter patch does not mean shorter code path, which is what matters.
Anyway, I've already posted upstream.

--- Additional comment from <email address hidden> on 2009年02月07日 17:25:55 EDT ---

Upstream fix was merged: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c09249f8d1b84344eca882547afdbffee8c09d14

I don't know what the point of this bug report is. The fix is already in upstream, and rawhide rebases from upstream daily.

I think you want to file this bug against Fedora 10 ??

(In reply to comment #2)
> I think you want to file this bug against Fedora 10 ??

Yes. I don't know if this will be backported to 2.6.27.x

Fixed in 2.6.27.18-170.2.28 and will send to -stable.

kernel-2.6.27.19-170.2.35.fc10 has been pushed to the Fedora 10 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
su -c 'yum --enablerepo=updates-testing update kernel'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F10/FEDORA-2009-2003

kernel-2.6.27.19-170.2.35.fc10 has been pushed to the Fedora 10 stable repository. If problems still persist, please make note of it in this bug report.

I would be the second person who tested this on a second Jaunty install and a second Lenny install

I reproduced this with a vanilla v2.6.26 kernel (on x86_64 with defconfig). I used git bisect to find the first commit on which the problem is exhibited:

commit 8ab32bb89b5b9bf06147c31947eba65f0f21c3c0
Author: Roland McGrath <email address hidden>
Date: Sun Mar 16 21:57:41 2008 -0700

x86: ia32 ptrace vs -ENOSYS

When we're stopped at syscall entry tracing, ptrace can change the %eax
value from -ENOSYS to something else. If no system call is actually made
because the syscall number (now in orig_eax) is bad, then the %eax value
set by ptrace should be returned to the user. But, instead it gets reset
to -ENOSYS again. This is a regression from the native 32-bit kernel.

This change fixes it by leaving the return value alone after entry tracing.

Signed-off-by: Roland McGrath <email address hidden>
Signed-off-by: Ingo Molnar <email address hidden>

By defconfig, I meant defconfig plus CONFIG_IA32_EMULATION=y (without which it is not possible to run the i386 binary); that is the only option I changed.

And the first commit after v2.6.26 on which the problem is NOT exhibited is:

commit ed8cae8ba01348bfd83333f4648dd807b04d7f08
Author: Ulrich Drepper <email address hidden>
Date: Wed Jul 23 21:29:30 2008 -0700

flag parameters: pipe

This patch introduces the new syscall pipe2 which is like pipe but it also
takes an additional parameter which takes a flag value. This patch implements
the handling of O_CLOEXEC for the flag. I did not add support for the new
syscall for the architectures which have a special sys_pipe implementation. I
think the maintainers of those archs have the chance to go with the unified
implementation but that's up to them.

The implementation introduces do_pipe_flags. I did that instead of changing
all callers of do_pipe because some of the callers are written in assembler.
I would probably screw up changing the assembly code. To avoid breaking code
do_pipe is now a small wrapper around do_pipe_flags. Once all callers are
changed over to do_pipe_flags the old do_pipe function can be removed.

It seems that __have_pipe2 is being incorrectly set to 1 when running under kernels inside this range (8ab32bb..ed8cae8). Before 8ab32bb, __have_pipe2 is set to -1 and the program works correctly; starting with ed8cae8, pipe2() is actually implemented so the program works correctly.

Fascinating. The problem is that pipe2() is not actually returning -1 and setting errno = ENOSYS like it should. Instead, it is returning 331, which is __NR_pipe2, and leaving errno == 0.

This affects every x86_64 kernel from v2.6.26 (8ab32bb) through v2.6.29-rc3, and was fixed by

commit c09249f8d1b84344eca882547afdbffee8c09d14
Author: Roland McGrath <email address hidden>
Date: Fri Feb 6 18:15:18 2009 -0800

x86-64: fix int 0ドルx80 -ENOSYS return

One of my past fixes to this code introduced a different new bug.
When using 32-bit "int 0ドルx80" entry for a bogus syscall number,
the return value is not correctly set to -ENOSYS. This only happens
when neither syscall-audit nor syscall tracing is enabled (i.e., never
seen if auditd ever started).

The fix makes the int 0ドルx80 path match the sysenter and syscall paths.

Reported-by: Dmitry V. Levin <email address hidden>
Signed-off-by: Roland McGrath <email address hidden>

To fix this on the kernel side, please add commit c09249f8d1b84344eca882547afdbffee8c09d14 to the Intrepid and Jaunty kernels.

It may be possible for glibc to work around this problem on unpatched kernels, though it would not be very pretty.

> Correct output:
> $ uname -r
> 2.6.28-8-generic
> $ ./popen-test
> popen returned 0x1ec6400
> ferror returned 0
> fread returned 19
> ferror returned 0
> pclose returned 0
> Incorrect output:
> $ uname -r
> 2.6.26-1-amd64
> $ ./popen-test
> popen returned 0x935f688
> ferror returned 0
> SHOULD NOT DISPLAY

I am confused. Your own output above shows that this works fine on the
Ubuntu Jaunty kernel. We don't need to patch Jaunty kernels to get the
right result?

It is possible this is triggered by the fact that we do indeed default to running with auditing. That would make the problem moot for a default Intrepid and Jaunty install. I would need a complelling use case to support and SRU of this change given the default install is unaffected.

Kernels starting with 2.6.27 do not have that particular problem with popen() because they implement the pipe2() system call, so popen() does not tickle the missing syscall detection bug. But the attached source can be used to reproduce the missing syscall detection bug directly, by displaying the return value of syscall(6666666).

There are likely to be more and more problems arising from this bug as new syscalls get implemented.

To reproduce, statically compile this on an Intrepid or Jaunty i386 system:
gcc -static bad-syscall-test.c -o bad-syscall-test
and then run the resulting binary on an Intrepid or Jaunty amd64 system.

Correct output:
$ uname -rm
2.6.24-etchnhalf.1-amd64 x86_64
$ ./bad-syscall-test
syscall(6666666) returned -1
errno is 38 (Function not implemented)
Incorrect output:
$ uname -rm
2.6.28-8-generic x86_64
$ ./bad-syscall-test
syscall(6666666) returned 6666666
errno is 0 (Success)

It seems that the only sensible fix is to fix the kernel. If this will affect new system calls then this does seem like something worth fixing correctly.

@Anders -- I have pulled down that kernel fix and applied it to Jaunty and Intrepid, could you test the kernels at the URL below and confirm you are happy that they fix the issue to your satisfaction. Please report any testing back here. Kernels are at the URLs below:

http://people.ubuntu.com/~apw/lp339743-intrepid/
http://people.ubuntu.com/~apw/lp339743-jaunty/

Thanks. I tested the Jaunty amd64 kernel using the same bad-syscall-test binary; it works correctly and fixes the problem. I will test the Intrepid kernel soon.

Yep, the Intrepid amd64 kernel also fixes the problem.

http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-jaunty.git;a=commit;h=73eec19c0301e46b46184cc8c16704ee6b9e26a5

This bug was fixed in the package linux - 2.6.28-10.32

---------------
linux (2.6.28-10.32) jaunty; urgency=low

[ Amit Kucheria ]

* Delete prepare-ppa-source script

[ Andy Isaacson ]

* SAUCE: FSAM7400: select CHECK_SIGNATURE
* SAUCE: LIRC_PVR150: depends on VIDEO_IVTV
- LP: #341477

[ Ayaz Abdulla ]

* SAUCE: forcedeth: msi interrupt fix
- LP: #288281

[ Brad Figg ]

* Updating armel configs to remove PREEMPT

[ Catalin Marinas ]

* Fix the VFP handling on the Feroceon CPU

[ Huaxu Wan ]

* SAUCE: (drop after 2.6.28) [Jaunty] iwlagn: fix iwlagn DMA mapping
direction

[ Ike Panhc ]

* squashfs: correct misspelling
- LP: #322306

[ Theodore Ts'o ]

* SAUCE: (drop after 2.6.28) ext4: add EXT4_IOC_ALLOC_DA_BLKS ioctl
* SAUCE: (drop after 2.6.28) ext4: Automatically allocate delay allocated
blocks on close
* SAUCE: (drop after 2.6.28) ext4: Automatically allocate delay allocated
blocks on rename
- LP: #317781

[ Tyler Hicks ]

* SAUCE: (drop after 2.6.28) eCryptfs: Don't encrypt file key with
filename key
- LP: #342128

[ Upstream Kernel Changes ]

* ALS: hda - Add support of iMac 24 Aluminium
* USB: fix broken OTG makefile reference
* ALSA: hda - add another MacBook Pro 3,1 SSID
* ALSA: hda - Add model entry for HP dv4
* x86-64: fix int 0ドルx80 -ENOSYS return
- LP: #339743

-- Tim Gardner <email address hidden> 2009年3月12日 19:16:07 -0600

Accepted linux into intrepid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

I have reproduced the bad return value of int 0ドルx080 calls on the amd64 kernels in intrepid-updates, currently 2.6.27-14.37, and confirmed that the kernels in intrepid-proposed, version 2.6.27-14.36 (needing to be updated), fixes the issue. I've added a slightly modified testcase to the lp:qa-regression-testing bzr tree (I invoke int 0ドルx080 directly via inline asm rather than call syscall(2) so that binaries can be built directly on amd64). Marking verification-done.

This bug was fixed in the package linux - 2.6.27-14.41

---------------
linux (2.6.27-14.41) intrepid-proposed; urgency=low

[ Stefan Bader ]

* Revert "SAUCE: input: Blacklist digitizers from joydev.c"
- LP: #300143

linux (2.6.27-14.40) intrepid-proposed; urgency=low

[ Amit Kucheria ]

* Disable DEVKMEM for all archs on Intrepid
- LP: #354221
* SAUCE: Quirk for BT USB device on MacbookPro to be reset before use
- LP: #332443

[ Andy Isaacson ]

* LIRC_PVR150: depends on VIDEO_IVTV
- LP: #341477
* SAUCE: FSAM7400: select CHECK_SIGNATURE
- LP: #341712

[ Andy Whitcroft ]

* SAUCE: hotkey quirks for various Zepto Znote and Fujitsu Amilo laptops
- LP: #330259
* SAUCE: unusual devs: add an entry for the ScanLogic SL11R-IDE 0.78
- LP: #336189

[ Anton Veretenenko ]

* SAUCE: sony-laptop: add support for Sony Vaio FW series function/media
keys
- LP: #307592

[ Ayaz Abdulla ]

* SAUCE: forcedeth: msi interrupt fix
- LP: #288281

[ Chuck Short ]

* SAUCE: [USB] Unusual Device support for Gold MP3 Player Energy
- LP: #125250

[ Ike Panhc ]

* squashfs: correct misspelling
- LP: #322306
* SAUCE: Fixing symbol name in HECI module
- LP: #336549
* Copy header files for various kernel media driver
- LP: #322732

[ Stefan Bader ]

* SAUCE: vgacon: Return the upper half of 512 character fonts
- LP: #355057
* SAUCE: input: Blacklist digitizers from joydev.c
- LP: #300143

[ Upstream Kernel Changes ]

* libata: make sure port is thawed when skipping resets
- LP: #269652
* x86-64: fix int 0ドルx80 -ENOSYS return
- LP: #339743
* rt2x00: Fix race conditions in flag handling
- LP: #258985
* USB: cdc-acm: Add another conexant modem to the quirks
- LP: #323829
* Bluetooth: Add fine grained mem_flags usage to btusb driver
- LP: #268502
* Bluetooth: Handle bulk URBs in btusb driver from notify callback
- LP: #268502
* Bluetooth: Submit bulk URBs along with interrupt URBs
- LP: #268502
* hwmon: (abituguru3) Match partial DMI board name strings
- LP: #298798
* x86: mtrr: don't modify RdDram/WrDram bits of fixed MTRRs
- LP: #292619
* sis190: add identifier for Atheros AR8021 PHY
- LP: #247889
* ath9k: implement IO serialization
- LP: #373034
* ath9k: AR9280 PCI devices must serialize IO as well
- LP: #373034
* acer-wmi: fix regression in backlight detection
- LP: #333386

-- Stefan Bader <email address hidden> 2009年8月26日 11:48:11 +0200

I just checked sources of karmic and don't see there any prebuilt openvz enabled kernels. I thought it was this bug that was preventing the openvz-enablde kernel to be built and put into the repository.