Kernel Oops - BUG: unable to handle kernel paging request at ffffffffffffffb8; RIP: 0010:[<ffffffffa05a5839>] [<ffffffffa05a5839>] nfs_have_delegation+0x9/0x40 [nfs]
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Linux |
Invalid
|
Undecided
|
Unassigned | ||
| linux (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
| Precise |
Fix Released
|
Medium
|
Unassigned | ||
Bug Description
== Precise SRU Justification ==
This bug is preventing users from using NFS clients on Precise.
Several users have reported the issue, which has been already fixed
upstreams but did not made it into stable yet.
== Fix ==
There are four commits that are relevant to fix this issue. From
mainline kernel:
- 14977489ffdb80d
- 96dcadc2fdd111d
From linux-nfs git tree
(git://
- 487790f27df9bb2
- 5de4815015e550b
== Impact ==
There are several users reporting the same issue when running NFS
clients on Precise. Other reports have also been found with the same
issue:
https:/
== Test Case ==
According to one of the bug reporters:
"Logging into a 12.04 client system with autofs5, LDAP, kerberos authenticated
nfs4 mounts. However this is to a server which is running 10.04 (contrary to #4
it seems) In my case the triggering process is gnome-keyring-d."
=======
I have no idea.
ProblemType: KernelOops
DistroRelease: Ubuntu 12.04
Package: linux-image-
ProcVersionSign
Uname: Linux 3.2.0-22-generic x86_64
NonfreeKernelMo
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
Annotation: Your system might become unstable now and might need to be restarted.
ApportVersion: 2.0-0ubuntu4
Architecture: amd64
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Thu Apr 5 14:35:46 2012
Failure: oops
HibernationDevice: RESUME=
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release amd64 (20111012)
IwConfig:
lo no wireless extensions.
eth2 no wireless extensions.
MachineType: Gigabyte Technology Co., Ltd. GA-MA78GM-US2H
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageV
RfKill:
SourcePackage: linux
Title: BUG: unable to handle kernel paging request at ffffffffffffffb8
UpgradeStatus: Upgraded to precise on 2012年03月21日 (15 days ago)
dmi.bios.date: 10/08/2009
dmi.bios.vendor: Award Software International, Inc.
dmi.bios.version: F8
dmi.board.name: GA-MA78GM-US2H
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.modalias: dmi:bvnAwardSof
dmi.product.name: GA-MA78GM-US2H
dmi.sys.vendor: Gigabyte Technology Co., Ltd.
- AcpiTables.txt Edit (137.2 KiB, text/plain; charset="utf-8")
- AlsaDevices.txt Edit (711 bytes, text/plain; charset="utf-8")
- AplayDevices.txt Edit (382 bytes, text/plain; charset="utf-8")
- ArecordDevices.txt Edit (415 bytes, text/plain; charset="utf-8")
- AudioDevicesInUse.txt Edit (192 bytes, text/plain; charset="utf-8")
- BootDmesg.txt Edit (77.7 KiB, text/plain; charset="utf-8")
- Card0.Amixer.info.txt Edit (171 bytes, text/plain; charset="utf-8")
- Card0.Amixer.values.txt Edit (4.5 KiB, text/plain; charset="utf-8")
- Card0.Codecs.codec.0.txt Edit (13.8 KiB, text/plain; charset="utf-8")
- Card1.Amixer.info.txt Edit (178 bytes, text/plain; charset="utf-8")
- Card1.Amixer.values.txt Edit (124 bytes, text/plain; charset="utf-8")
- Card1.Codecs.codec.0.txt Edit (1.1 KiB, text/plain; charset="utf-8")
- CurrentDmesg.txt Edit (5.9 KiB, text/plain; charset="utf-8")
- Dependencies.txt Edit (1.9 KiB, text/plain; charset="utf-8")
- Lspci.txt Edit (28.1 KiB, text/plain; charset="utf-8")
- Lsusb.txt Edit (705 bytes, text/plain; charset="utf-8")
- OopsText.txt Edit (2.6 KiB, text/plain; charset="utf-8")
- PciMultimedia.txt Edit (3.3 KiB, text/plain; charset="utf-8")
- ProcCpuinfo.txt Edit (2.6 KiB, text/plain; charset="utf-8")
- ProcInterrupts.txt Edit (1.9 KiB, text/plain; charset="utf-8")
- ProcModules.txt Edit (2.8 KiB, text/plain; charset="utf-8")
- UdevDb.txt Edit (120.8 KiB, text/plain; charset="utf-8")
- UdevLog.txt Edit (242.0 KiB, text/plain; charset="utf-8")
- WifiSyslog.txt Edit (167.5 KiB, text/plain; charset="utf-8")
rvaliant, thank you for reporting this bug and helping make Ubuntu better. Please answer these questions:
* Is this reproducible?
* If so, what specific steps should we take to recreate this bug?
* Could you also please test the latest upstream kernel available? It will allow additional upstream developers to examine the issue. Refer to https:/
Thanks in advance.
+ Kernel Oops - BUG: unable to handle kernel paging request at
+ ffffffffffffffb8; RIP: 0010:[<ffffffffa05a5839>] [<ffffffffa05a5839>]
+ nfs_have_delegation+0x9/0x40 [nfs]
Description of problem:
Kernel traceback on NFS4 locking attempt
Version-Release number of selected component (if applicable):
kernel 3.3.1-2 (and 3.3.1-3)
How reproducible:
Always
Steps to Reproduce:
1. Install kernel 3.3.1-{2,3}; mount a users home over NFS4
2. Start e.g. pidgin, which tries to lock some file
3.
Actual results:
*) Pidgin crashes with 'Killed'. Strace shows that the last activity
is:
[...]
open("/
flock(11, LOCK_EX <unfinished ...>
+++ killed by SIGKILL +++
Killed
*) In dmesg, you see:
[ 322.987247] BUG: unable to handle kernel paging request at ffffffffffffffb8
[ 322.987330] IP: [<ffffffffa0e14
[ 322.987418] PGD 1c07067 PUD 1c08067 PMD 0
[ 322.987495] Oops: 0000 [#1] SMP
[ 322.987565] CPU 1
[ 322.987574] Modules linked in: nfs fscache nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack sha256_generic dm_crypt nvidia(PO) nfsd uinput lockd nfs_acl auth_rpcgss sunrpc snd_hda_
[ 322.988230]
[ 322.988230] Pid: 1697, comm: pidgin Tainted: P O 3.3.1-3.fc16.x86_64 #1 transtec AG /DG31PR
[ 322.988230] RIP: 0010:[<
[ 322.988230] RSP: 0018:ffff880112
[ 322.988230] RAX: ffff880122411800 RBX: ffff880112e55e68 RCX: 00000000ffffd8ca
[ 322.988230] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
[ 322.988230] RBP: ffff880112e55dd8 R08: 0000000000016560 R09: ffffea00044b8400
[ 322.988230] R10: ffffffffa0e0457a R11: 0000000000000000 R12: 00000000ffffd8ca
[ 322.988230] R13: ffff8801218be000 R14: ffff88011573cc00 R15: ffff8801224b0480
[ 322.988230] FS: 00007fdda206098
[ 322.988230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 322.988230] CR2: ffffffffffffffb8 CR3: 0000000112c07000 CR4: 00000000000006e0
[ 322.988230] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 322.988230] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 322.988230] Process pidgin (pid: 1697, threadinfo ffff880112e54000, task ffff880112dd2e60)
[ 322.988230] Stack:
[ 322.988230] ffff880112e55e28 ffffffffa0e01fa1 ffff880112e11e60 0000000000000000
[ 322.988230] ffff8801224b0480 ffff8801224b0780 ffff8801224b0480 0000000000000082
[ 322.988230] ffff880112e55e68 00000000fffffff5 ffff880112e55eb8 ffffffffa0e04b9c
[ 322.988230] Call Trace:
[ 322.988230] [<ffffffffa0e01
[ 322.988230] [<ffffffffa0e04
[ 322.988230] [<ffffffffa0de5
[ 322.988230] [<ffffffffa0de5
[ 322.988230] [<ffffffff811cb
[ 322.988230] [<ffffffff815fb
We are suffering the same problem, it occurs right after login when gnome-keyring tries to access the home directory mounted via NFSv4 with kernel 3.3.1-3 on the client. Server is CentOS 6.2. Client-side kernel 3.2.10-
Apr 10 14:26:32 morgan kernel: [ 54.179536] BUG: unable to handle kernel paging request at ffffffffffffffb8
Apr 10 14:26:32 morgan kernel: [ 54.179575] IP: [<ffffffffa0380
Apr 10 14:26:32 morgan kernel: [ 54.179621] PGD 1c07067 PUD 1c08067 PMD 0
Apr 10 14:26:32 morgan kernel: [ 54.179646] Oops: 0000 [#1] SMP
Apr 10 14:26:32 morgan kernel: [ 54.179665] CPU 1
Apr 10 14:26:32 morgan kernel: [ 54.179675] Modules linked in: fuse nfs fscache auth_rpcgss nfs_acl lockd nf_conntrack_ipv4 nf_defrag_ipv4 ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables s
nd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device ppdev snd_pcm joydev i2c_i801 snd_timer snd soundcore snd_page_alloc serio_raw dcdbas parport_pc e1000e iTCO_wdt iTCO_vendor_support parport microcode sunr
pc uinput usb_storage ata_generic pata_acpi radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
Apr 10 14:26:32 morgan kernel: [ 54.179957]
Apr 10 14:26:32 morgan kernel: [ 54.179966] Pid: 1345, comm: gnome-keyring-d Not tainted 3.3.1-3.fc16.x86_64 #1 Dell Inc. OptiPlex 755 /0GM819
Apr 10 14:26:32 morgan kernel: [ 54.180015] RIP: 0010:[<
Apr 10 14:26:32 morgan kernel: [ 54.180061] RSP: 0018:ffff880074
Apr 10 14:26:32 morgan kernel: [ 54.180084] RAX: ffff8800608f4400 RBX: ffff880074255e68 RCX: 00000000ffffd8ca
Apr 10 14:26:32 morgan kernel: [ 54.180112] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
Apr 10 14:26:32 morgan kernel: [ 54.180139] RBP: ffff880074255dd8 R08: 0000000000016560 R09: ffffea0001dc9a00
Apr 10 14:26:32 morgan kernel: [ 54.180167] R10: ffffffffa037057a R11: 0000000000000000 R12: 00000000ffffd8ca
Apr 10 14:26:32 morgan kernel: [ 54.180195] R13: ffff8800608b6000 R14: ffff88006041d000 R15: ffff88007793e600
Apr 10 14:26:32 morgan kernel: [ 54.180223] FS: 00007ffb5b18080
Apr 10 14:26:32 morgan kernel: [ 54.180255] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 10 14:26:32 morgan kernel: [ 54.180277] CR2: ffffffffffffffb8 CR3: 00000000779d9000 CR4: 00000000000006e0
Apr 10 14:26:32 morgan kernel: [ 54.180305] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 10 14:26:32 morgan kernel: [ 54.180333] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr 10 14:26:32 morgan kernel: [ 54.180362] Process gnome-keyring-d (pid: 1345, threadinfo ffff880074254000, task ffff880078782e60)
Apr 10 14:26:32 morgan kernel: [ 54.180396] Stack:
Apr 10 14:26:32 morgan kernel: [ 54.180405] ffff880074255e28 ffffffffa036dfa1 ffff880077268660 0000000000000000
Apr 10 14:26:32 morgan kernel: [ 54.180443] ffff88007793e600 f...
*** Bug 811484 has been marked as a duplicate of this bug. ***
This bug has also been reported here: https:/
Any news on this? It's kinda urgent, had to make all of our desktop machines boot an old kernel now...
On our desktop clients kernel 3.3.1-5.fc16.x86_64 shows the same nfs_have_delegation error when using an NFS mounted home directory. This system was just installed today with fedora-updates repo enabled. To reproduce edit any existing file in the home directory with gedit and try to save it. I can confirm that kernel 3.3.0-8.fc16.x86_64 does not have this problem.
Apr 14 22:40:11 client kernel: [ 913.752179] BUG: unable to handle kernel paging request at ffffffffffffffb8
Apr 14 22:40:11 client kernel: [ 913.752204] IP: [<ffffffffa039c
Apr 14 22:40:11 client kernel: [ 913.752234] PGD 1c07067 PUD 1c08067 PMD 0
Apr 14 22:40:11 client kernel: [ 913.752245] Oops: 0000 [#1] SMP
Apr 14 22:40:11 client kernel: [ 913.752256] CPU 1
Apr 14 22:40:11 client kernel: [ 913.752260] Modules linked in: nfs fscache auth_rpcgss nfs_acl lockd ip6t_REJECT nf_conntrack_ipv6 nf_defr
ag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack tpm_infineon snd_hda_
ymap rfkill ppdev snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm microcode snd_timer snd soundcore snd_page_alloc seri
o_raw iTCO_wdt iTCO_vendor_support parport_pc parport e1000e tpm_tis tpm tpm_bios sunrpc uinput ata_generic pata_acpi nouveau ttm drm_kms_he
lper drm i2c_core mxm_wmi video wmi [last unloaded: scsi_wait_scan]
Apr 14 22:40:11 client kernel: [ 913.752401]
Apr 14 22:40:11 client kernel: [ 913.752405] Pid: 2786, comm: gedit Tainted: G I 3.3.1-5.fc16.x86_64 #1 ***REMOVED***
Apr 14 22:40:11 client kernel: [ 913.752424] RIP: 0010:[<
Apr 14 22:40:11 client kernel: [ 913.752447] RSP: 0018:ffff88005a
Apr 14 22:40:11 client kernel: [ 913.752453] RAX: ffff88006428b400 RBX: ffff88005a6a3e68 RCX: 00000000ffffd8ca
Apr 14 22:40:11 client kernel: [ 913.752461] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
Apr 14 22:40:11 client kernel: [ 913.752476] RBP: ffff88005a6a3dd8 R08: 0000000000016560 R09: ffffea0001694980
Apr 14 22:40:11 client kernel: [ 913.752483] R10: ffffffffa038c57a R11: 0000000000000000 R12: 00000000ffffd8ca
Apr 14 22:40:11 client kernel: [ 913.752656] R13: ffff8800659ea800 R14: ffff880064289400 R15: ffff8800523599c0
Apr 14 22:40:11 client kernel: [ 913.752664] FS: 00007f584675498
Apr 14 22:40:11 client kernel: [ 913.752674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 14 22:40:11 client kernel: [ 913.752680] CR2: ffffffffffffffb8 CR3: 00000000642bb000 CR4: 00000000000006e0
Apr 14 22:40:11 client kernel: [ 913.752691] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 14 22:40:11 client kernel: [ 913.752699] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr 14 22:40:11 client kernel: [ 913.752707] Process gedit (pid: 2786, threadinfo ffff88005a6a2000, task ffff8800646fdcc0)
Apr 14 22:40:11 client kernel: [ 913.752725] Stack:
Apr 14 22:40:11 client kernel: [ 913.752729] ffff88005a6a3e28 ffffffffa0389fa1 ffff88005a527...
Some more tests:
1) kernels 3.3.2-1.
2) With 3.4.0-0.
but a hang, and this stuff in the dmesg
[ 216.350962] nfs4_reclaim_
[ 216.350965] NFS: nfs4_reclaim_
[ 216.352550] NFS: nfs4_reclaim_
[ 216.354033] NFS: nfs4_reclaim_
[ 216.355580] NFS: nfs4_reclaim_
[ 216.357055] NFS: nfs4_reclaim_
[ 216.358522] NFS: nfs4_reclaim_
[ 216.360042] NFS: nfs4_reclaim_
[ 216.361513] NFS: nfs4_reclaim_
[ 216.363006] NFS: nfs4_reclaim_
[ 216.364582] NFS: nfs4_reclaim_
This repeats endlessly till the offending program is killed.
Abt. 120 machines affected here.
*** Bug 812633 has been marked as a duplicate of this bug. ***
This problem is reproduced by logging into Ubuntu with a user's home directory mounted over NFS... though only if the server is also running 12.04.
I have a stable server running 10.04 and 12.04 clients can still mount NFS home directories from there, moving to the test server, configured in the same way but running 12.04, clients crash as above.
Has anyone affected by this bug had a chance to test the latest mainline kernel:
http://
Actually, my server is Ubuntu 11.10, so if this is an NFS related issue, it's not confined to 12.04. I have some time today and will try the latest mainline kernel on my client as suggested by jsalisbury and report back - if someone doesn't beat me to it.
Looks like this is probably fixable by pushing commit 14977489 to stable.
I'm seeing the same OOps -- again Logging into a 12.04 client system with autofs5, LDAP, kerberos authenticated nfs4 mounts. However this is to a server which is running 10.04 (contrary to #4 it seems) In my case the triggering process is gnome-keyring-d.
Running the mainline kernel (linux-
[ 48.701213] NFS: nfs4_reclaim_
[ 48.701990] NFS: nfs4_reclaim_
[ 53.696076] nfs4_reclaim_
Until the client was upgraded to 12.04, it was running 11.10 perfectly in this configuration.
cc'ing Trond...
It looks like this regression crept in with commit 3114ea7a. That commit was
sent to stable kernel series, but commit 149774 (which fixed a bug in the
above commit) was not.
Trond, any chance you can send 149774 to stable as well? We might also want
to take that into Fedora for now until it makes its way into stable kernels.
It looks like commit 14977489ffdb80d
Re: #8, tried this kernel with the same effect as running the mainline kernel in #7 specifically many many messages of the type:
NFS: nfs4_reclaim_
I've added 149774 to F15-F17 now. Rawhide already has it.
I've started a scratch build here:
http://
testing would be appreciated.
Ian: Thanks for testing the kernel. Apparently, these messages are harmless and the mainline kernel has already a ratelimiting patch for this. I have backported the ratelimiting commit (96dcadc2fdd111
http://
Please let me know if this new kernel fixes it.
- Packet capture Edit (211.3 KiB, image/png)
Re: #10, Luis, thanks for your efforts on this. By running http://
I'm not entirely sure about the "harmless" nature of the messages however. My perception is that logging in is somewhat slower -- this may be down to the beta nature of the release at present however what I am observing, even after logon of the user has completed is a high level of network activity. Running a packet trace and a wireshark dump of this activity, it seems that the activity is down to NFS and it seems (see screenshot attached) to be stuck in somewhat of a loop -- specifically continually getting an error when trying to lock the file (each time the same file). I wonder if the slowness has the same root cause (the shere number of failed retries -- looking at the packet trace it seems, if I'm not mistaken, there's about 1000 per second of these!)
I observe the file being processed is user.keystore and in my case, the prior oops was triggered by gnome-keyring-d so I can't help feeling these issues are related. However I will admit I'm no expert on NFS so I could be seeing ghosts :-)
Let me know if there is any information I can provide that might help.
Ian, thanks again for testing and helping triagging this bug.
When I said "harmless" (and I'm far from being an expert on NFS as well!) I was not aware that the flood of messages being logged would be kept that high. Obviously, logging such a huge amount of messages will at least impact the performance. My comment referred to the fact that the log was actually on an error path that would be retried again.
I'll ping upstreams for further information and post any development here.
I'm also seeing this (didn't report; tainted kernel, though I now have another box set up with an untainted kernel to play with)
I grabbed the SRPM and built myself a kernel-
Alas, I can't try my 100% reliable reproducer of "hit the New Message button in kmail" because this kernel gives me my NFS home directory owned by an unfeasibly large UID :-)
Under 3.3.0-8.fc17.x86_64 the files are owned by 'tim' but under kernel-
I know about getting /etc/idmapd.conf correct. I can flip back and forth between the states by rebooting and selecting the different kernels with no other change.
(In reply to comment #11)
> I'm also seeing this (didn't report; tainted kernel, though I now have another
> box set up with an untainted kernel to play with)
>
> I grabbed the SRPM and built myself a kernel-
>
> Alas, I can't try my 100% reliable reproducer of "hit the New Message button in
> kmail" because this kernel gives me my NFS home directory owned by an
> unfeasibly large UID :-)
F17 has additional NFS patches that are needed to work with the F17 userspace utilities. You would need to have an F17 SRPM for it to work properly.
To Josh Boyer in reply to comment #10:
I tried your 3.3.2-3.fc16.x86_64 build from Koji. It behaves like the 3.4.0
kernel from F18: the program (pidgin in this case) hangs, and dmesg spits, by the thousands:
[ 96.542908] nfs4_reclaim_
[ 96.543687] nfs4_reclaim_
[ 96.544439] nfs4_reclaim_
[ 96.545258] nfs4_reclaim_
[ 96.546066] nfs4_reclaim_
So no solution yet ...
(In reply to comment #12)
> (In reply to comment #11)
> > I'm also seeing this (didn't report; tainted kernel, though I now have another
> > box set up with an untainted kernel to play with)
> >
> > I grabbed the SRPM and built myself a kernel-
> >
> > Alas, I can't try my 100% reliable reproducer of "hit the New Message button in
> > kmail" because this kernel gives me my NFS home directory owned by an
> > unfeasibly large UID :-)
>
> F17 has additional NFS patches that are needed to work with the F17 userspace
> utilities. You would need to have an F17 SRPM for it to work properly.
Any instructions for finding/making one? I furtled through koji.fedoraproj
"Re: #10, Luis, thanks for your efforts on this. By running http://
I can confirm that here too, using Luis' kernel on the client and the latest, untouched, untainted 3.2.0-23-generic on the server.
Can this not be marked as confirmed now?
Ian, there's something I forgot to ask: could you take a look at the server logs (running Lucid, I believe) and check whether there's some extra information there? Please attach any relevant information (in particular the kernel logs).
- Trace - nfs_have_delegation Edit (3.2 KiB, text/plain)
Same Problem here:
Server 10.04 LTS Stock-Kernel
Clients: 11.04 - Custom Kernel, nfs-home
Kernel 3.1, 3.2 from kernel.org:
BUG: unable to handle kernel paging request at ffffffffffffffb8
IP: [<ffffffffa0e83
Kernel 3.4 from kernel.org:
NFS: nfs4_reclaim_
Appears when trying to run for example tomboy, gedit, pidgin. No 3.x (3.1, 3.2, 3.4) - Kernel allows to start the apps with a nfs-home.
No entries in Serverlogs.
*** Bug 813732 has been marked as a duplicate of this bug. ***
(In reply to comment #14)
> > F17 has additional NFS patches that are needed to work with the F17 userspace
> > utilities. You would need to have an F17 SRPM for it to work properly.
>
> Any instructions for finding/making one? I furtled through
> koji.fedoraproj
> as comment #13 has? (I'm guessing here that f18 has the NFS patches of which
> you speak) Or should I drop that spare machine back to f16 to try this out?
I started an F17 scratch build here:
(In reply to comment #13)
> To Josh Boyer in reply to comment #10:
>
> I tried your 3.3.2-3.fc16.x86_64 build from Koji. It behaves like the 3.4.0
> kernel from F18: the program (pidgin in this case) hangs, and dmesg spits, by
> the thousands:
>
> [ 96.542908] nfs4_reclaim_
> [ 96.543687] nfs4_reclaim_
> [ 96.544439] nfs4_reclaim_
> [ 96.545258] nfs4_reclaim_
> [ 96.546066] nfs4_reclaim_
>
> So no solution yet ...
Well, it's not oopsing at least.
Jeff?
Re: #15, Correct -- server running lucid with stock kernel (from updates). There's no messages in the server logs that I can see (same as #16)
OK. Same "nfs4_reclaim_
Even more trivial way to reproduce, BTW:
bash$ /usr/bin/flock /file/on/nfs echo Fish
With kernel 3.3.2-3.fc16.x86_64 and saving a file on NFS mounted partition using gedit I have a similar result as described in comment #13: no longer a kernel oops and dmesg showing many lines of "Lock reclaim failed!":
[ 302.568318] nfs4_reclaim_
[ 302.569517] nfs4_reclaim_
[ 302.570724] nfs4_reclaim_
[ 302.571894] nfs4_reclaim_
[ 302.573091] nfs4_reclaim_
[ 302.574305] nfs4_reclaim_
[ 302.575437] nfs4_reclaim_
I cannot reproduce the "Lock reclaim failed!" error with the flock command in comment #18 though. Next I tried running
/usr/bin/flock ~/a sleep 10
in one terminal and
/usr/bin/flock ~/a echo foo
in another terminal and this kind of locking seems to work fine since the second flock waits with executing the echo command until the sleep command has exited.
I have ping'ed upstreams about the issue and two new patches (not yet upstream) have been provided. I have uploaded a new test kernel here:
http://
It contains the two mentioned patches and the other two you have already tested.
Please let me know if this new kernel finally solves the issue.
Re: #18, Luis, I think you've cracked it! I have just installed that kernel and I am happy to report that I'm not getting an oops, Nor am I getting any messages on the client, the network traffic is back down to normal levels, I'm not getting any messages on the server side and last but not least, my subjective test of the time it takes to login time seems to be back to normal!
Thats it, Luis! Bug is gone with your kernel.
Where can i get the patches to compile my own? When are the fixes @kernel.org?
Thanks!
I can confirm as well that using the kernel from #18 and performing the same steps as before fails to generate the bug and everything seems to work correctly.
Thanks as well!
That's great, thank you for testing. So, to summarise, here's the list of commits that were present in this last kernel you have tested:
- From Linux mainline:
- 14977489ffdb80d
- 96dcadc2fdd111d
- From linux-nfs, branch bugfixes:
- 487790f27df9bb2
- 5de4815015e550b
Same here, thank you so much Luis!
Is there any way this will make it into precise before launch now? It seems almost unthinkable that an LTS could be released without the ability to mount NFS home directories :s
Is there anything we can do to help this process?
Hmm. I have more interesting information....
On 3.3.0-8.
open("/
flock(3, LOCK_EX) = -1 EIO (Input/output error)
access(
close(3) = 0
open("/
flock(3, LOCK_EX) = 0
So it opens the file read-only, then tries to LOCK_EX and the NFS server (entirely reasonably) says "NFS4ERR_OPENMODE, you silly person". Then and only then does /usr/bin/flock get around to checking the permissions, opening it read/write, and doing another LOCK_EX, which works.
On my 3.3.2-4.
open("/
flock(3, LOCK_EX ....never returns
while a sniffer trace shows a continuous stream of OPEN(readonly)
So I would venture to suggest that the problem lurks around the -NFS4ERR_OPENMODE case in nfs4_handle_
Someone more familiar with the code will probably beat me to working out what the precise circumstances are in which it should be retrying and not returning the error, but I shall have a go at working that out anyway, 'cos I have a warped sense of fun...
There are a couple of patches that Trond has proposed upstream and for stable for the other (non-oops) problem. Would any of the folks suffering from it be able to test this and see if it fixes the issue?
(In reply to comment #21)
> There are a couple of patches that Trond has proposed upstream and for stable
> for the other (non-oops) problem. Would any of the folks suffering from it be
> able to test this and see if it fixes the issue?
>
> http://
Applied and tried.
This looks like correct behaviour:
[tim@groke ~]$ /usr/bin/flock ~/tmp/spon echo Fish
flock: /home/tim/tmp/spon: Bad file descriptor
(flock() should never have worked, really :-)
I will try this out with my original kmail problem when I have access to that desktop again in a few hours.
By default, nfs.ko does flock() emulation on top of POSIX locks.
If you find that programs that use flock don't work properly on top of NFS, you can mount with local_lock=flock and that will disable that behavior. Of course, no other client will be aware of flock locks at that point...
(In reply to comment #23)
> By default, nfs.ko does flock() emulation on top of POSIX locks.
>
> If you find that programs that use flock don't work properly on top of NFS, you
> can mount with local_lock=flock and that will disable that behavior. Of course,
> no other client will be aware of flock locks at that point...
In which case this may still be a problem. I mount with local_lock=none
The only way to get correct flock() emulation would be to silently reopen the FD as writable on the NFS server if LOCK_EX is requested (but presumably keep letting the VFS believe it's read-only). flock() will cheerfully let you exclusive-lock a read-only FD. Ick :-)
Aha! flock in f17's util-linux-2.21.1 has a test for EIO, and retries read-write. That might need an extra
case EBADF:
The check is not present in f16's util-linux-2.20.1
Was wondering why f16 behaved differently.
(In reply to comment #22)
> I will try this out with my original kmail problem when I have access to that
> desktop again in a few hours.
OK, the original kmail problem is fixed too. This appears simply to ignore the error it gets back from flock() anyway, once it actually returns :-)
Interestingly, it too attempts an exclusive lock on a read-only open of a file with read-write permission. Depending on how common this behaviour is, it might be less than useless to silently upgrade the NFS open to read-write on a flock(fd,LOCK_EX) IF (a) the file is actually writable AND (b) it is somehow possible to know that it *was* flock() requesting this operation and not fcntl().
...wanders off mumbling...
Hello rvaliant, or anyone else affected,
Accepted linux into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https:/
3.3.2-6.fc16.x86_64 is all now. These
nfs4_reclaim_
messages are filling /var/log/messages at about 1MB/sec. So with
an average / filesystem of 8 or 16GB, you can estimate the remaining
uptime after it hits you ...
So can I have this kernel OOPS back please?
This bug was fixed in the package linux - 3.2.0-24.37
---------------
linux (3.2.0-24.37) precise-proposed; urgency=low
[ Herton Ronaldo Krzesinski ]
* d-i: Add hid-logitech-dj to input-modules
- LP: #975198
* d-i: Add rtl8187 driver to nic-usb-modules
- LP: #971719
[ Ian Abbott ]
* SAUCE: staging: comedi: Add module parameters for default buffer size
- LP: #981234
* SAUCE: staging: comedi: Add kernel config for default buffer sizes
- LP: #981234
[ K. Y. Srinivasan ]
* SAUCE: hv_storvsc: Account for in-transit packets in the RESET path
- LP: #978394
[ Leann Ogasawara ]
* [Config] Set CONFIG_
- LP: #981234
[ Luis Henriques ]
* SAUCE: ite-cir: postpone ISR registration
- LP: #984387
[ Manoj Iyer ]
* SAUCE: Bluetooth: btusb: Add vendor specific ID (0489 e042) for
BCM20702A0
- LP: #980965
[ Tim Gardner ]
* Extract firmware module info during getabi
* [Config] Remove hiq-quanta module references
- LP: #913164
* [Config] powerpc-smp: build in ATI and RADEON frame buffer drivers
- LP: #949288
[ Trond Myklebust ]
* SAUCE: NFSv4: Ensure that the LOCK code sets exception->inode
- LP: #974664
* SAUCE: NFSv4: Ensure that we check lock exclusive/shared type against
open modes
- LP: #974664
[ Upstream Kernel Changes ]
* Input: psmouse - allow drivers to use psmouse_
- LP: #969334
* Input: psmouse - use psmouse_
drivers
- LP: #969334
* Input: sentelic - refactor code for upcoming new hardware support
- LP: #969334
* Input: sentelic - enabling absolute coordinates output for newer
hardware
- LP: #969334
* Input: sentelic - minor code cleanup
- LP: #969334
* Input: sentelic - improve packet debugging information
- LP: #969334
* Input: sentelic - filter taps in absolute mode
- LP: #969334
* drm/i915: Fixes distorted external screen image on HP 2730p
- LP: #796030
* NFSv4: Minor cleanups for nfs4_handle_
nfs4_
- LP: #974664
* NFSv4: Rate limit the state manager for lock reclaim warning messages
- LP: #974664
* HID: multitouch: merge quanta driver into hid-multitouch
- LP: #913164
* HID: usbhid: add quirk no_get for quanta 3008 devices
- LP: #913164
-- Leann Ogasawara <email address hidden> 2012年4月24日 07:47:49 -0700
(In reply to comment #27)
>
> 3.3.2-6.fc16.x86_64 is all now. These
>
> nfs4_reclaim_
>
> messages are filling /var/log/messages at about 1MB/sec. So with
> an average / filesystem of 8 or 16GB, you can estimate the remaining
> uptime after it hits you ...
>
> So can I have this kernel OOPS back please?
It looks like this is avoided in mainline due to a previous commit:
commit 96dcadc2fdd111d
Author: William Dauchy <email address hidden>
Date: Wed Mar 14 12:32:04 2012 +0100
NFSv4: Rate limit the state manager for lock reclaim warning messages
This should presumably go into stable as well, though it seems to me that that still leaves a bug to fix - either the warning is real and we should try to avoid the problem, or the warning is bogus and should be removed.
(In reply to comment #28)
> (In reply to comment #27)
> >
> > 3.3.2-6.fc16.x86_64 is all now. These
> >
> > nfs4_reclaim_
> >
>
> It looks like this is avoided in mainline due to a previous commit:
>
> commit 96dcadc2fdd111d
> Author: William Dauchy <email address hidden>
> Date: Wed Mar 14 12:32:04 2012 +0100
>
> NFSv4: Rate limit the state manager for lock reclaim warning messages
Either that or the patch mentioned in comment #21 isn't in yet. If you have stuff hanging on a flock() attempt, you will need that one or one very similar.
Hi,
I've applied the following patches on top of the 3.3.4 kernel I got from Koji:
They seem to resolve this bug. I believe these patches have been queued by Greg KH for the 3.3 stable queue.
I added the following patches:
NFS: put open context on error in nfs_flush_multi
commit 8ccd271f7a3a846
nfs: Enclose hostname in brackets when needed in
commit 98a2139f4f4d7b5
NFSv4: Ensure that we check lock exclusive/shared type against open modes
commit 55725513b5ef9d4
NFSv4: Ensure that the LOCK code sets exception->inode
commit 05ffe24f5290dc0
I believe only the last two are needed to fix this issue.
Please apply these patches to the next Fedora kernel update.
Rik
(In reply to comment #30)
> Hi,
>
> I've applied the following patches on top of the 3.3.4 kernel I got from Koji:
>
> https:/
>
> They seem to resolve this bug. I believe these patches have been queued by Greg
> KH for the 3.3 stable queue.
>
> I added the following patches:
>
> NFS: put open context on error in nfs_flush_multi
> commit 8ccd271f7a3a846
>
> nfs: Enclose hostname in brackets when needed in
> commit 98a2139f4f4d7b5
>
> NFSv4: Ensure that we check lock exclusive/shared type against open modes
> commit 55725513b5ef9d4
>
> NFSv4: Ensure that the LOCK code sets exception->inode
> commit 05ffe24f5290dc0
>
> I believe only the last two are needed to fix this issue.
>
> Please apply these patches to the next Fedora kernel update.
Justin put all of those into the F17 kernel last night. We'll pick them up in f16 relatively soon.
Solved in kernel-
Tnx all!
- extract /var/log/syslog with the bug description Edit (4.1 KiB, text/plain)
I observe the same kernel bug on Ubuntu 11.10 (oneiric) with kernel 3.0.0-19-generic, see the attached syslog. Interestingly, it's not there in the vanilla install of Ubuntu 11.10 with kernel 3.0.0-12-generic (tested by booting an older kernel from grub). This issue manifests itself when I log in using an LDAP user with home on an NFS mount.
Any chance of fixing it on Ubuntu 11.10?
Tagging this bug as verified for Precise. Patch is already released with Precise, but showing up on ti-omap4 SRU report where it isn't a specific ti-omap4 change.
The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.