sighals and sockets

Wed Aug 4 18:10:00 GMT 2004

I've been trying to track down a bug in my code for a few days now and I
was looking at a pretty strange stack. It looked like the
PlainSocketImpl::write would get as far as __libc_write, enter a signal
handler, segv, enter the segv signal handler, then recurse forever until
the process crashed. It was pretty confusing because the runtime sets an
ignore on SIGPIPE and the args going into the write looked fine from
gdb. After some thought, I realized that since the app uses a good deal
of native (CNI) code, it's possible someone else is setting a sighandler
for SIGPIPE. It turns out that the syslog call will do just that in
glibc. It installs a temporary sighandler for PIPE, then restores
whatever was there before returning to the caller. So there's a race
condition that exists between a thread writing to a socket in the java
runtime and a thread calling syslog. The only problem I see here is that
the socket writing thread may inadvertantly close the syslog connection,
which would then be re-stablished the next time someone calls syslog. 
Would it be safer to use the send(...) call instead of the write(...) in
the socket code? Are there any other uses of the ignore on the SIGPIPE
other that the socket writer?
I still have not discovered why the application has crashed. But here's
a snippet of the stack. This is 3.3.1
-jake
(gdb) bt -40
#9158 0x2ac5106e in uw_frame_state_for (context=0x797fea3c,
fs=0x797fe97c) at ../../gcc/gcc/unwind-dw2.c:939
#9159 0x2ac51768 in _Unwind_RaiseException (exc=0x429b7560) at
../../gcc/gcc/unwind.inc:95
#9160 0x2afd4ab2 in _Jv_Throw (value=0x4008fc30) at
../../../gcc/libjava/exception.cc:100
#9161 0x2afc698a in _Jv_ThrowSignal (throwable=0x4008fc30) at
../../../gcc/libjava/prims.cc:152
#9162 0x2afc69c2 in catch_segv (_dummy=11) at
../../../gcc/libjava/prims.cc:162
#9163 <signal handler called>
#9164 0x2ac5106e in uw_frame_state_for (context=0x797fef94,
fs=0x797feed4) at ../../gcc/gcc/unwind-dw2.c:939
#9165 0x2ac51768 in _Unwind_RaiseException (exc=0x429b75a0) at
../../gcc/gcc/unwind.inc:95
#9166 0x2afd4ab2 in _Jv_Throw (value=0x4008fc30) at
../../../gcc/libjava/exception.cc:100
#9167 0x2afc698a in _Jv_ThrowSignal (throwable=0x4008fc30) at
../../../gcc/libjava/prims.cc:152
#9168 0x2afc69c2 in catch_segv (_dummy=11) at
../../../gcc/libjava/prims.cc:162
#9169 <signal handler called>
#9170 0x00000003 in ?? ()
#9171 0x2b19db2e in pthread_sighandler (signo=13, ctx=
 {gs = 0, __gsh = 0, fs = 0, __fsh = 0, es = 43, __esh = 0, ds =
43, __dsh = 0, edi = 157, esi = 1095844640, ebp = 2038429524, esp =
2038429476, ebx = 157, edx = 200, ecx = 1095844640, eax = 4294967264,
trapno = 0, err = 0, eip = 727775108, cs = 35, __csh = 0, eflags = 530,
esp_at_signal = 2038429476, ss = 43, __ssh = 0, fpstate = 0x797ff4a8,
oldmask = 2147483648, cr2 = 0}) at signals.c:97
#9172 <signal handler called>
#9173 0x2b60f784 in __libc_write () from /lib/libc.so.6
#9174 0x2b19f730 in write (fd=157, buf=0x41514320, n=200) at
wrapsyscall.c:178
#9175 0x2aff8a80 in java::net::PlainSocketImpl::write (this=0x42728ac8,
b=0x41514318, offset=0, len=200) at include/platform.h:132
#9176 0x2b0cab57 in
java.net.PlainSocketImpl$SocketOutputStream.write(byte[], int, int)
(this=0x41d55228, buffer=0x41514318, offset=0,
 length=200) at
../../../gcc/libjava/java/net/PlainSocketImpl.java:222
#9177 0x2b0122bc in java.io.BufferedOutputStream.flush()
(this=0x423a8ad0) at
../../../gcc/libjava/java/io/BufferedOutputStream.java:141
#9178 0x2b02640a in java.io.OutputStreamWriter.writeChars(char[], int,
int) (this=0x4236d150, buf=0x43400f00, offset=0, count=100)
 at ../../../gcc/libjava/java/io/OutputStreamWriter.java:116
#9179 0x2b026698 in java.io.OutputStreamWriter.write(int)
(this=0x4236d150, ch=69)
 at ../../../gcc/libjava/java/io/OutputStreamWriter.java:178
#9180 0x2b02919e in java.io.PrintWriter.write(int) (this=0x423a8ae0,
ch=69) at ../../../gcc/libjava/java/io/PrintWriter.java:500
#9181 0x08290dab in com.spinnakernet.mgmt.cli.CliWriter.write(int)
(this=0x423a8ae0, ch=69)
 at build/framework/com/spinnakernet/mgmt/cli/CliWriter.java:40
#9182 0x08290dcf in com.spinnakernet.mgmt.cli.CliWriter.write(char)
(this=0x423a8ae0, ch=69)
 at build/framework/com/spinnakernet/mgmt/cli/CliWriter.java:44
#9183 0x0829f9b1 in
com.spinnakernet.mgmt.cli.reports.TextBox.drawLine(int, int)
(this=0x40174750, x1=0, x2=20)
 at build/framework/com/spinnakernet/mgmt/cli/reports/TextBox.java:80
#9184 0x0829eb23 in
com.spinnakernet.mgmt.cli.reports.Frame.drawLine(int, int)
(this=0x40174780, x1=0, x2=2147483647)
 at build/framework/com/spinnakernet/mgmt/cli/reports/Frame.java:164
#9185 0x082a0c25 in
com.spinnakernet.mgmt.cli.reports.TableFormatter.flush()
(this=0x4001f540)
 at
build/framework/com/spinnakernet/mgmt/cli/reports/TableFormatter.java:16
1
#9186 0x082a0dd1 in
com.spinnakernet.mgmt.cli.reports.TableFormatter.displayIt(com.spinnaker
net.mgmt.cim.Report) (this=0x4001f540,
 rp=0x401747e0) at
build/framework/com/spinnakernet/mgmt/cli/reports/TableFormatter.java:19
3
#9187 0x08292357 in com.spinnakernet.mgmt.cli.ResponsePdu.TransmitPdu()
(this=0x4155db00)
 at build/framework/com/spinnakernet/mgmt/cli/ResponsePdu.java:87
#9188 0x0829a54b in
com.spinnakernet.mgmt.cli.actions.CimShowAction.applyIt(com.spinnakernet
.mgmt.cli.NameSpaceNode, com.spinnakernet.mgmt.cli.UserSession,
com.spinnakernet.mgmt.cli.CmdTokenEnumeration,
com.spinnakernet.mgmt.form.Form) (this=0x4173b300, node=0x408eea20,
 sess=0x40289360, st=0x42a208e8, form=0x40658fa0) at
build/framework/com/spinnakernet/mgmt/cli/actions/CimShowAction.java:135
#9189 0x08293e4b in
com.spinnakernet.mgmt.cli.keycommands.ExecuteCommand.applyIt(com.spinnak
ernet.mgmt.cli.UserSession, char) (
 this=0x40bde1f0, sess=0x40289360, c=120) at
build/framework/com/spinnakernet/mgmt/cli/keycommands/ExecuteCommand.jav
a:113
---Type <return> to continue, or q <return> to quit---
#9190 0x08292c66 in
com.spinnakernet.mgmt.cli.InteractiveSession.ProcessEvents()
(this=0x40289360)
 at
build/framework/com/spinnakernet/mgmt/cli/InteractiveSession.java:297
#9191 0x082911ca in
com.spinnakernet.mgmt.cli.ShellServerImpl$ShellUserSession.run()
(this=0x41fbdb00)
 at
build/framework/com/spinnakernet/mgmt/cli/ShellServerImpl.java:137
#9192 0x2b00ec33 in java.lang.Thread.run() (this=0x42728a00) at
../../../gcc/libjava/java/lang/Thread.java:134
#9193 0x2aff193c in _Jv_ThreadRun (thread=0x42728a00) at
../../../gcc/libjava/java/lang/natThread.cc:285
#9194 0x2afc61f0 in really_start (x=0x429664b8) at
../../../gcc/libjava/posix-threads.cc:392
#9195 0x2b1847de in GC_start_routine (arg=0x4276cec0) at
../../../gcc/boehm-gc/linux_threads.c:1375
#9196 0x2b19b057 in pthread_start_thread (arg=0x797ffc00) at
manager.c:284
#9197 0x2b61cc0a in thread_start () from /lib/libc.so.6
Current language: auto; currently c++
(gdb)