2009年3月15日星期日

WLS_033:常见故障之一:Server Crash

Server Crash时,JVM强行退出,并产生一个 server core 文件。该文件是 Server Crash 时,JVM中所有活动线程的快照,即 Thread Dump。

1. core文件位置
(1)在Unix环境下,该文件位于Server启动的目录下。
(2)在Windows环境下,该文件位于\Windows\Drwatson目录下,名称类似drwtsn32.log。


2. Server Crash 原因
Java代码不会引起Server Crash,只有Native 代码(C、C++)才有可能引起Server Crash。
在WebLogic Server中,出现
Native 代码的地方有:
(1)WebLogic Server native performance packs
(2)Type 2 JDBC drivers
(3)应用程序代码使用
Java Native Interface (JNI调用Native libraries
(4)JVM 自身,大多数JVM都使用了JIT提高性能,如果怀疑JVM引起Crash,可以在启动Server的脚本里,禁用JIT: -Djava.compiler=none。


3. Hotspot JVM Crash
Hotspot JVM发生Crash时,会产生一个文件 hs_err_pid<WLSpid>.log。 
在该文件中,一般会告诉我们引起Crash的原因。比如:
An unexpected exception has been detected in native code
outside the VM.
Unexpected Signal : 11 occurred at PC=0x5a4cf2e4
Function name=Java_HelloWorld_displayHelloWorld

Library=/home/spoz/wls70/linuxAS/user_projects/mydomain/lib/lib
hello.so
Current Java thread:
at HelloWorld.displayHelloWorld(Native Method)
at servlets.NativeServlet.doGet(NativeServlet.java:85)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
. . .
Local Time = Wed 17 09:35:39 2004
Elapsed Time = 186
# The exception was detected in native code outside the VM
# Java VM: Java HotSpot(TM) Client VM (1.3.1_06-b01 mixed mode)


4. Debugger JVM
如果问题可以重现,可以跟踪JVM:
java -Xdebug -Xrunjdwp:transport=dt_socket,address=1000,server=y,suspend=n

5. 如何模拟 Server Crash?
最简单的方法就是用C/C++编写一个除零的方法,编译成DLL,放到启动Server的bin目录下,然后在Java Servlet中通过JNI调用该方法。此时,Server 会Crash掉,JVM也会随之Crash掉。

(1)如果是Hotspot JVM,
在Server的bin目录下会生成类似的文件hs_err_pid1348.log,内容如下:
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# EXCEPTION_INT_DIVIDE_BY_ZERO (0xc0000094) at pc=0x04b61011, pid=1800, tid=1196
#
# Java VM: Java HotSpot(TM) Client VM (1.4.2_11-b06 mixed mode)
# Problematic frame:
# C [kboom.dll+0x1011]
#

--------------- T H R E A D ---------------

Current thread (0x02f39dc0): JavaThread "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon [_thread_in_native, id=1196]

siginfo: ExceptionCode=0xc0000094

Registers:
EAX=0x00000001, EBX=0x15804550, ECX=0x02f488c0, EDX=0x00000000
ESP=0x037ef9b4, EBP=0x037ef9b8, ESI=0x15804598, EDI=0x02f39dc0
EIP=0x04b61011, EFLAGS=0x00010293

Top of Stack: (sp=0x037ef9b4)
0x037ef9b4: 00000000 037ef9e4 00ac81a6 02f39e5c
0x037ef9c4: 037ef9f4 037ef9c8 15804598 037ef9f4
0x037ef9d4: 15804840 00000000 15804550 037ef9f4
0x037ef9e4: 037efa14 00ac2cb3 00000000 00ac64a9
0x037ef9f4: 102fc158 037ef9f8 158045ef 037efa1c
0x037efa04: 15804840 00000000 158045a0 037efa1c
0x037efa14: 037efa3c 00ac2cb3 102fc150 037efa20
0x037efa24: 157fc6d4 037efa50 157fc968 00000000

Instructions: (pc=0x04b61011)
0x04b61001: 8b ec 51 c7 45 fc 00 00 00 00 b8 01 00 00 00 99
0x04b61011: f7 7d fc 89 45 fc 8b e5 5d c2 08 00 8b 44 24 08


Stack: [0x037b0000,0x037f0000), sp=0x037ef9b4, free space=254k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [kboom.dll+0x1011]
j troubleshooting.common.KBoom.killMeNow()V+0
j troubleshooting.common.KBoom.terminateWebLogic()V+7
j troubleshooting.servlets.WebLogicCoreDump.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+20
j javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+30
j weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run()Ljava/lang/Object;+12
j weblogic.servlet.internal.ServletStubImpl.invokeServlet(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lweblogic/servlet/internal/FilterChainImpl;)V+387
j weblogic.servlet.internal.ServletStubImpl.invokeServlet(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+4
j weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run()Ljava/lang/Object;+22
j weblogic.security.acl.internal.AuthenticatedSubject.doAs(Lweblogic/security/subject/AbstractSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;+34
j weblogic.security.service.SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;+18
j weblogic.servlet.internal.WebAppServletContext.invokeServlet(Lweblogic/servlet/internal/ServletRequestImpl;Lweblogic/servlet/internal/ServletResponseImpl;)V+685
j weblogic.servlet.internal.ServletRequestImpl.execute(Lweblogic/kernel/ExecuteThread;)V+141
j weblogic.kernel.ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V+23
j weblogic.kernel.ExecuteThread.run()V+66
v ~StubRoutines::call_stub
V [jvm.dll+0x72846]
V [jvm.dll+0xac976]
V [jvm.dll+0x72753]
V [jvm.dll+0x7244f]
V [jvm.dll+0x72488]
V [jvm.dll+0x8a3a0]
V [jvm.dll+0xd0de8]
V [jvm.dll+0xd0db6]
C [MSVCRT.dll+0x2a3b0]
C [kernel32.dll+0xb50b]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j troubleshooting.common.KBoom.killMeNow()V+0
j troubleshooting.common.KBoom.terminateWebLogic()V+7
j troubleshooting.servlets.WebLogicCoreDump.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+20
j javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+30
j weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run()Ljava/lang/Object;+12
j weblogic.servlet.internal.ServletStubImpl.invokeServlet(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lweblogic/servlet/internal/FilterChainImpl;)V+387
j weblogic.servlet.internal.ServletStubImpl.invokeServlet(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+4
j weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run()Ljava/lang/Object;+22
j weblogic.security.acl.internal.AuthenticatedSubject.doAs(Lweblogic/security/subject/AbstractSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;+34
j weblogic.security.service.SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;+18
j weblogic.servlet.internal.WebAppServletContext.invokeServlet(Lweblogic/servlet/internal/ServletRequestImpl;Lweblogic/servlet/internal/ServletResponseImpl;)V+685
j weblogic.servlet.internal.ServletRequestImpl.execute(Lweblogic/kernel/ExecuteThread;)V+141
j weblogic.kernel.ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V+23
j weblogic.kernel.ExecuteThread.run()V+66
v ~StubRoutines::call_stub

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x00a9b970 JavaThread "ExecuteThread: '3' for queue: 'JMS.TimerClientPool'" daemon [_thread_blocked, id=1692]
0x00a9b408 JavaThread "ExecuteThread: '2' for queue: 'JMS.TimerClientPool'" daemon [_thread_blocked, id=1032]
0x03351008 JavaThread "ExecuteThread: '1' for queue: 'JMS.TimerClientPool'" daemon [_thread_blocked, id=500]
0x030cdd00 JavaThread "ExecuteThread: '0' for queue: 'JMS.TimerClientPool'" daemon [_thread_blocked, id=468]
0x03099358 JavaThread "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon [_thread_in_native, id=1492]
0x03098700 JavaThread "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon [_thread_in_native, id=1544]
0x031068a8 JavaThread "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon [_thread_in_native, id=208]
0x03106040 JavaThread "SSLListenThread.Default" [_thread_in_native, id=1156]
0x032eb348 JavaThread "ListenThread.Default" [_thread_in_native, id=1524]
0x033487f0 JavaThread "Thread-7" daemon [_thread_blocked, id=1532]
0x02fcaf48 JavaThread "ExecuteThread: '0' for queue: 'JMS.TimerTreePool'" daemon [_thread_blocked, id=1688]
0x032e1538 JavaThread "Thread-6" [_thread_blocked, id=1108]
0x0309d510 JavaThread "weblogic.health.CoreHealthMonitor" daemon [_thread_blocked, id=1552]
0x0309d1e0 JavaThread "Thread-5" [_thread_blocked, id=340]
0x02f07630 JavaThread "DoSManager" [_thread_blocked, id=588]
0x02f012a0 JavaThread "VDE Transaction Processor Thread" [_thread_blocked, id=1376]
0x02fc8408 JavaThread "ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon [_thread_blocked, id=1288]
0x030962d0 JavaThread "ExecuteThread: '1' for queue: 'weblogic.admin.RMI'" daemon [_thread_blocked, id=1368]
0x02ff8e88 JavaThread "ExecuteThread: '0' for queue: 'weblogic.admin.RMI'" daemon [_thread_blocked, id=1264]
0x02f1a008 JavaThread "ExecuteThread: '1' for queue: 'weblogic.admin.HTTP'" daemon [_thread_blocked, id=120]
0x03096448 JavaThread "ExecuteThread: '0' for queue: 'weblogic.admin.HTTP'" daemon [_thread_blocked, id=776]
0x02f4e1f0 JavaThread "weblogic.security.SpinnerRandomSource" daemon [_thread_blocked, id=1148]
0x02f4e4e8 JavaThread "weblogic.time.TimeEventGenerator" daemon [_thread_blocked, id=1152]
0x02f4c4d0 JavaThread "ExecuteThread: '4' for queue: 'weblogic.kernel.System'" daemon [_thread_blocked, id=1188]
0x02f4b8d8 JavaThread "ExecuteThread: '3' for queue: 'weblogic.kernel.System'" daemon [_thread_blocked, id=1916]
0x02f4ace0 JavaThread "ExecuteThread: '2' for queue: 'weblogic.kernel.System'" daemon [_thread_blocked, id=1604]
0x02f4a0e8 JavaThread "ExecuteThread: '1' for queue: 'weblogic.kernel.System'" daemon [_thread_blocked, id=280]
0x02f49d08 JavaThread "ExecuteThread: '0' for queue: 'weblogic.kernel.System'" daemon [_thread_blocked, id=236]
0x02f48960 JavaThread "ExecuteThread: '14' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=204]
=>0x02f39dc0 JavaThread "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon [_thread_in_native, id=1196]
0x02f39218 JavaThread "ExecuteThread: '12' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=1192]
0x02f38670 JavaThread "ExecuteThread: '11' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=1184]
0x02f37ac8 JavaThread "ExecuteThread: '10' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=1412]
0x02f36f90 JavaThread "ExecuteThread: '9' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=976]
0x02f47498 JavaThread "ExecuteThread: '8' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=1116]
0x02f46948 JavaThread "ExecuteThread: '7' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=584]
0x02f45ea0 JavaThread "ExecuteThread: '6' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=1484]
0x02f441f0 JavaThread "ExecuteThread: '5' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=512]
0x02f43888 JavaThread "ExecuteThread: '4' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=416]
0x02f362c8 JavaThread "ExecuteThread: '3' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=1752]
0x02f36160 JavaThread "ExecuteThread: '2' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=1248]
0x02f36970 JavaThread "ExecuteThread: '1' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=1224]
0x03147758 JavaThread "ExecuteThread: '0' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=1180]
0x03147f60 JavaThread "Thread-1" daemon [_thread_blocked, id=464]
0x00a105e0 JavaThread "CompilerThread0" daemon [_thread_blocked, id=824]
0x00a5ed80 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=968]
0x00a0d488 JavaThread "Finalizer" daemon [_thread_blocked, id=1880]
0x00a0c9b0 JavaThread "Reference Handler" daemon [_thread_blocked, id=184]
0x00036a18 JavaThread "main" [_thread_blocked, id=924]

Other Threads:
0x00a4b6f8 VMThread [id=1304]
0x00a5db40 WatcherThread [id=1584]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
def new generation total 4544K, used 3283K [0x10010000, 0x104f0000, 0x104f0000)
eden space 4096K, 73% used [0x10010000, 0x102fc1e0, 0x10410000)
from space 448K, 64% used [0x10480000, 0x104c8a38, 0x104f0000)
to space 448K, 0% used [0x10410000, 0x10410000, 0x10480000)
tenured generation total 60544K, used 17040K [0x104f0000, 0x14010000, 0x14010000)
the space 60544K, 28% used [0x104f0000, 0x11594160, 0x11594200, 0x14010000)
compacting perm gen total 24576K, used 24530K [0x14010000, 0x15810000, 0x18010000)
the space 24576K, 99% used [0x14010000, 0x158048d0, 0x15804a00, 0x15810000)

Dynamic libraries:
0x00400000 - 0x0040b000 C:\bea\jdk142_11\bin\java.exe
0x7c920000 - 0x7c9b4000 C:\WINDOWS\system32\ntdll.dll
0x7c800000 - 0x7c91c000 C:\WINDOWS\system32\kernel32.dll
0x77da0000 - 0x77e49000 C:\WINDOWS\system32\ADVAPI32.dll
0x77e50000 - 0x77ee1000 C:\WINDOWS\system32\RPCRT4.dll
0x77be0000 - 0x77c38000 C:\WINDOWS\system32\MSVCRT.dll
0x08000000 - 0x0813f000 C:\bea\jdk142_11\jre\bin\client\jvm.dll
0x77d10000 - 0x77d9f000 C:\WINDOWS\system32\USER32.dll
0x77ef0000 - 0x77f36000 C:\WINDOWS\system32\GDI32.dll
0x76b10000 - 0x76b3a000 C:\WINDOWS\system32\WINMM.dll
0x76300000 - 0x7631d000 C:\WINDOWS\system32\IMM32.DLL
0x62c20000 - 0x62c29000 C:\WINDOWS\system32\LPK.DLL
0x73fa0000 - 0x7400b000 C:\WINDOWS\system32\USP10.dll
0x10000000 - 0x10007000 C:\bea\jdk142_11\jre\bin\hpi.dll
0x76bc0000 - 0x76bcb000 C:\WINDOWS\system32\PSAPI.DLL
0x008a0000 - 0x008ae000 C:\bea\jdk142_11\jre\bin\verify.dll
0x008b0000 - 0x008c9000 C:\bea\jdk142_11\jre\bin\java.dll
0x008d0000 - 0x008de000 C:\bea\jdk142_11\jre\bin\zip.dll
0x02d20000 - 0x02d2f000 C:\bea\jdk142_11\jre\bin\net.dll
0x71a20000 - 0x71a37000 C:\WINDOWS\system32\WS2_32.dll
0x71a10000 - 0x71a18000 C:\WINDOWS\system32\WS2HELP.dll
0x719c0000 - 0x719fe000 C:\WINDOWS\System32\mswsock.dll
0x76ef0000 - 0x76f17000 C:\WINDOWS\system32\DNSAPI.dll
0x76f80000 - 0x76f88000 C:\WINDOWS\System32\winrnr.dll
0x76f30000 - 0x76f5c000 C:\WINDOWS\system32\WLDAP32.dll
0x76f90000 - 0x76f96000 C:\WINDOWS\system32\rasadhlp.dll
0x02e40000 - 0x02e48000 C:\bea\jdk142_11\jre\bin\nio.dll
0x02e50000 - 0x02e56000 C:\bea\jdk142_11\jre\bin\ioser12.dll
0x60fd0000 - 0x61025000 C:\WINDOWS\system32\hnetcfg.dll
0x71a00000 - 0x71a08000 C:\WINDOWS\System32\wshtcpip.dll
0x04570000 - 0x04575000 C:\bea\weblogic81\server\bin\stackdump.dll
0x04580000 - 0x04585000 C:\bea\weblogic81\server\bin\wlntio.dll
0x04b60000 - 0x04b67000 C:\student\course_wls_troubleshooting\bin\kboom.dll

VM Arguments:
jvm_args: -Dweblogic.security.SSL.verbose=false -Dssl.debug=false -Dweblogic.StdoutDebugEnabled=true -Xms64m -Xmx64m -Dweblogic.Name=AdminServer -Dbea.home=C:\bea -Dweblogic.ProductionModeEnabled=false -Djava.security.policy==C:\bea\weblogic81\lib\weblogic.policy -Dweblogic.RootDirectory=C:\student\course_wls_troubleshooting\domains\dizzyworld
java_command: weblogic.Server
Launcher Type: SUN_STANDARD

Environment Variables:
CLASSPATH=.;C:\bea\weblogic81\server\lib\weblogic.jar;C:\bea\weblogic81\common\eval\pointbase\lib\pbclient44.jar;C:\bea\weblogic81\common\eval\pointbase\lib\pbtools44.jar;C:\bea\weblogic81\common\eval\pointbase\lib\pbserver44.jar;C:\student\course_wls_troubleshooting\lib;C:\bea\jdk142_11\lib\tools.jar
PATH=C:\bea\weblogic81\server\bin;C:\bea\jdk142_11\bin;C:\student\course_wls_troubleshooting\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\IDM Computer Solutions\UltraEdit\
USERNAME=pma
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 23 Stepping 10, GenuineIntel


--------------- S Y S T E M ---------------

OS: Windows XP Build 2600 Service Pack 2

CPU:total 2 family 6, cmov, cx8, fxsr, mmx, sse, sse2

Memory: 4k page, physical 2096620k(1320460k free), swap 4034924k(2087084k free)

vm_info: Java HotSpot(TM) Client VM (1.4.2_11-b06) for windows-x86, built on Feb 13 2006 10:19:30 by "java_re" with MS VC++ 6.0

如果在启动Server的脚本里,加上-XX:+ShowMessageBoxOnError,增加的位置在'java' 之后,'weblogic.Server' 之前。这样在Server Crash时,会弹出窗口:


不要选是或否,获取此时的Thread Dump。

(2)如果是JRockit JVM,在Server的bin目录下会生成类似的文件jrockit.4012.dump和jrockit.4012.mdmp文件。其中jrockit.4012.dump是可读文本,比较小,jrockit.4012.mdmp是二进制文件,比较大。jrockit.4012.dump 内容如下:

===== BEGIN DUMP =============================================================
JRockit dump produced after 0 days, 00:01:07 on Sat Apr 30 14:46:25 2011

Additional information is available in:
C:\student\course_wls_troubleshooting\bin\jrockit.3420.dump
C:\student\course_wls_troubleshooting\bin\jrockit.3420.mdmp
If you see this dump, please open a support case with BEA and
supply as much information as you can on your system setup and
the program you were running. You can also search for solutions
to your problem at http://forums.bea.com in
the forum jrockit.developer.interest.general.

Error Message: Unhandled native exception [84]
Exception Rec: EXCEPTION_INT_DIVIDE_BY_ZERO (c0000094) at 0x01AE1011
Minidump : Wrote mdmp. Size is 141MB
Version : BEA JRockit(R) R26.3.0-32_CR283061-63465-1.4.2_10-20060620-1811-win-ia32
GC : System optimized over throughput (initial strategy singleparpar)
: GC strategy for GC 0 was singleparpar
: GC strategy for GC 1 was genparpar
: GC strategy for GC 2 was genparpar
: GC strategy for GC 3 was genparpar
: GC strategy for GC 4 was genparpar
: mmHeap->data = 0x78000000, mmHeap->top = 0x7C000000
: The nurserylist starts at 0x785EEFA8 and ends at 0x7AB0AF60
: mmStartCompaction = 0x7B400000, mmEndCompaction = 0x7B800008
CPU : Intel Pentium Pro SSE SSE2 EM64T
Number CPUs : 2
Tot Phys Mem : 2146938880 (2047 MB)
OS version : Microsoft Windows XP version 5.1 Service Pack 2 (Build 2600)
Thread System: Windows Threads
State : JVM is running
Command Line : -Dweblogic.security.SSL.verbose=false -Dssl.debug=false -Dweblogic.StdoutDebugEnabled=true -Xms64m -Xmx64m -Dweblogic.Name=AdminServer -Dbea.home=C:\bea -Dweblogic.ProductionModeEnabled=false -Djava.security.policy==C:\bea\weblogic81\lib\weblogic.policy -Dweblogic.RootDirectory=C:\student\course_wls_troubleshooting\domains\dizzyworld weblogic.Server
JAVA_HOME :
java.home : C:\bea\jrockit81sp6_142_10\jre
j.class.path : .;C:\bea\weblogic81\server\lib\weblogic.jar;C:\bea\weblogic81\common\eval\pointbase\lib\pbclient44.jar;C:\bea\weblogic81\common\eval\pointbase\lib\pbtools44.jar;C:\bea\weblogic81\common\eval\pointbase\lib\pbserver44.jar;C:\student\course_wls_troubleshooting\lib;C:\bea\jrockit81sp6_142_10\lib\tools.jar
j.lib.path : C:\bea\jrockit81sp6_142_10\bin;.;C:\WINDOWS\system32;C:\WINDOWS;C:\bea\weblogic81\server\bin;C:\bea\jrockit81sp6_142_10\bin;C:\student\course_wls_troubleshooting\bin;C:\bea\weblogic81\server\bin;C:\bea\jrockit81sp6_142_10\bin;C:\student\course_wls_troubleshooting\bin;C:\bea\weblogic81\server\bin;C:\bea\jrockit81sp6_142_10\bin;C:\student\course_wls_troubleshooting\bin;C:\bea
\weblogic81\server\bin;C:\bea\jdk142_11\bin;C:\student\course_wls_troubleshooting\bin;C:\bea\weblogic81\server\bin;C:\bea\jdk142_11\bin;C:\student\course_wls_troubleshooting\bin;C:\bea\weblogic81\server\bin;C:\bea\jdk142_11\bin;C:\student\course_wls_troubleshooting\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\IDM Computer Solutions\UltraEdit\
C Heap : Good; no memory allocations have failed
StackOverFlow: 0 StackOverFlowErrors have occured

Registers (from ThreadContext: 0x029BF56C / OS context: 0x029BF950):
eax = 00000001 ecx = 029bfc34 edx = 00000000 ebx = 79800640
esp = 029bfc1c ebp = 029bfc20 esi = 029bfc28 edi = 0135c5d4
es = 00000023 cs = 0000001b ss = 00000023 ds = 00000023
fs = 0000003b gs = 00000000
eip = 01ae1011 eflags = 00010216

Stack:
(* marks the word pointed to by the stack pointer)
029bfc1c: 00000000* 79763df0 74432b14 0135c5d4 029bfc44 029bfc18
029bfc34: 03401ff8 74432b0f 0135c68c 029bfc28 79800640 0135c490
029bfc4c: 79763df0 79800640 74432a9b 79800638 744329ae 797d6848
029bfc64: 79763df0 797f7900 74432908 79763df0 797d6848 79754c20
029bfc7c: 79763df0 797f7900 74432882 79763df0 744316cc 79763df0
029bfc94: 797d6848 78af24f8 00000000 797d6848 a5280611 0000012f
029bfcac: 797b6a58 7844d7d0 79754c20 79763df0 00000000 79754c20
029bfcc4: 00bc3820 74020350 0135c68c 029bfcb8 0135c490 74020350
029bfcdc: 7844d7d0 797b6a58 74020dfb 79763df0 79763df0 79754c20
029bfcf4: 7844d7d0 797b6a58 74431317 79763df0 00000000 7feaf738
029bfd0c: 744312c5 79763df0 7844d7d0 797b6a58 741d20fa 7844d7d0
029bfd24: 7844f3c0 00000000 7402721c 784168f8 0135c490 00000000
029bfd3c: 797b6a58 7844d7d0 741d2068 797b6a58 7443121d 7844d7d0
029bfd54: 7442c13f 797b6a58 79763df0 78af24f8 79754c20 784168f8
029bfd6c: 7846ea88 78a1e8d8 79763df0 78af24f8 00000000 79763df0
029bfd84: 7844d7d0 79754c20 00000008 784168f8 0135c490 79763df0
029bfd9c: 7442b9aa 79763df0 740203b1 79754c20 00000001 79754c20
029bfdb4: 0135c490 784d30f8 74360614 79754c20 0135c490 784d30f8
029bfdcc: 741f7d9e 0135c5d4 0135c490 784168f8 74020014 784d30f9
029bfde4: 0135c490 fffffffc 74021b61 fffffffc 029bfe04 029bfe98
029bfdfc: 741f7d00 016241d0 0135c5d4 0135c490 029bfe30 029bfe98
029bfe14: 0056e8c4 0135c490 741f7d00 029bfe98 029bff30 00000000
029bfe2c: 00000000 01623f08 0053aabd 74021b20 0135c490 741f7d00
029bfe44: 029bfe98 029bff30 7633e230 0135c5d4 029bff30 0135c490
029bfe5c: 00000000 00000000 00000001 00bc9c00 00000000 029bff04
029bfe74: 00000001 00b7eccc 00000006 00000001 00000001 00000001
029bfe8c: 00000000 00000001 00000000 0135c490 00000000 00000000
029bfea4: 00000000 00000000 00000000 00000000 00000000 004fd067
029bfebc: 0135c5d4 7c150470 004a0363 0135c5d4 7c150470 7c151848
029bfed4: 029bff04 004a049e 0135c5d4 7633e230 01623f08 0135c490
029bfeec: 00000000 029bff48 0053a7c0 029bff30 0135c490 029bff60
029bff04: 01623f08 00000000 00000000 004a229b 00000000 029bff48
029bff1c: 0053a7c0 00000000 029bff30 0135c5d4 0007fa4c 00000000
029bff34: 00000000 004d4cf2 0135c5d4 0135c490 7633e230 0135c490
029bff4c: 00000000 00000e38 0135c490 004d469f 00000000 0007f908
029bff64: 004d51fa 004d4c90 00000000 01624e20 00000000 029bffb4
029bff7c: 01354fe0 00000000 004d4c90 00527997 003e5958 01624e20
029bff94: 00000000 01354fe0 029bff90 029bf848 029bffdc 00594072
029bffac: 005af0d8 00000000 029bffec 7c80b50b 01354fe0 01624e20
029bffc4: 00000000 01354fe0 7ff92000 c0000094 029bffc0 029bf848
029bffdc: ffffffff 7c8399f3 7c80b518 00000000 00000000 00000000
029bfff4: Unreadable
029c000c: 00000000 00000000 00000000 00000000 00000000 00000000
029c0024: 00000000 00000000 00000000 00000000 00000000 00000000
029c003c: 00000000 00000000 00000000 00000000 00000000 00000000
029c0054: 00000000 00000000 00000000 00000000 00000000 00000000
029c006c: 00000000 00000000 00000000 00000000 00000000 00000000
029c0084: 00000000 00000000 00000000 00000000 00000000 00000000
029c009c: 00000000 00000000 00000000 00000000 00000000 00000000
029c00b4: 00000000 00000000 00000000 00000000 00000000 00000000
029c00cc: 00000000 00000000 00000000 00000000 00000000 00000000
029c00e4: 00000000 00000000 00000000 00000000 00000000 00000000
029c00fc: 00000000 00000000 00000000 00000000 00000000 00000000
029c0114: 00000000 00000000 00000000 00000000 00000000 00000000
029c012c: 00000000 00000000 00000000 00000000 00000000 00000000
029c0144: 00000000 00000000 00000000 00000000 00000000 00000000
029c015c: 00000000 00000000 00000000 00000000 00000000 00000000
029c0174: 00000000 00000000 00000000 00000000 00000000 00000000
029c018c: 00000000 00000000 00000000 00000000 00000000 00000000
029c01a4: 00000000 00000000 00000000 00000000 00000000 00000000
029c01bc: 00000000 00000000 00000000 00000000 00000000 00000000
029c01d4: 00000000 00000000 00000000 00000000 00000000 00000000
029c01ec: 00000000 00000000 00000000 00000000 00000000 00000000
029c0204: 00000000 00000000 00000000 00000000 00000000 00000000
029c021c: 00000000 00000000 00000000 00000000 00000000 00000000
029c0234: 00000000 00000000 00000000 00000000 00000000 00000000
029c024c: 00000000 00000000 00000000 00000000 00000000 00000000
029c0264: 00000000 00000000 00000000 00000000 00000000 00000000
029c027c: 00000000 00000000 00000000 00000000 00000000 00000000
029c0294: 00000000 00000000 00000000 00000000 00000000 00000000
029c02ac: 00000000 00000000 00000000 00000000 00000000 00000000
029c02c4: 00000000 00000000 00000000 00000000 00000000 00000000
029c02dc: 00000000 00000000 00000000 00000000 00000000 00000000
029c02f4: 00000000 00000000 00000000 00000000 00000000 00000000
029c030c: 00000000 00000000 00000000 00000000 00000000 00000000
029c0324: 00000000 00000000 00000000 00000000 00000000 00000000
029c033c: 00000000 00000000 00000000 00000000 00000000 00000000
029c0354: 00000000 00000000 00000000 00000000 00000000 00000000
029c036c: 00000000 00000000 00000000 00000000 00000000 00000000
029c0384: 00000000 00000000 00000000 00000000 00000000 00000000
029c039c: 00000000 00000000 00000000 00000000 00000000 00000000
029c03b4: 00000000 00000000 00000000 00000000 00000000 00000000
029c03cc: 00000000 00000000 00000000 00000000 00000000 00000000
029c03e4: 00000000 00000000 00000000 00000000 00000000 00000000
029c03fc: 00000000 00000000 00000000 00000000 00000000 00000000
029c0414: 00000000 00000000 00000000 00000000 00000000 00000000

Code:
(* marks the word pointed to by the instruction pointer)
01ae0f10: 00000000 00000000 00000000 00000000 00000000 00000000
01ae0f28: 00000000 00000000 00000000 00000000 00000000 00000000
01ae0f40: 00000000 00000000 00000000 00000000 00000000 00000000
01ae0f58: 00000000 00000000 00000000 00000000 00000000 00000000
01ae0f70: 00000000 00000000 00000000 00000000 00000000 00000000
01ae0f88: 00000000 00000000 00000000 00000000 00000000 00000000
01ae0fa0: 00000000 00000000 00000000 00000000 00000000 00000000
01ae0fb8: 00000000 00000000 00000000 00000000 00000000 00000000
01ae0fd0: 00000000 00000000 00000000 00000000 00000000 00000000
01ae0fe8: 00000000 00000000 00000000 00000000 00000000 00000000
01ae1000: 51ec8b55 00fc45c7 b8000000 00000001 fc7df799* 8bfc4589
01ae1018: 08c25de5 24448b00 01f88308 0088850f 15ff0000 01ae4004
01ae1030: 78a3016a e801ae53 00000a15 7459c085 5378a13c c93301ae
01ae1048: 53790d8a ff2501ae c1000000 ae53782d 80a31001 8901ae53
01ae1060: ae53840d 08e0c101 7ca3c103 e801ae53 00000270 0975c085
01ae1078: 000a10e8 ebc03300 0015ff72 a301ae40 01ae58d8 00088ee8
01ae1090: 5364a300 6ee801ae e8000003 00000632 000574e8 011ce800
01ae10a8: 05ff0000 01ae5360 c9333eeb 2c75c13b 53600d39 bd7e01ae
01ae10c0: 53600dff 0d3901ae 01ae53b0 31e80575 e8000001 000004ed
01ae10d8: 00025be8 09abe800 0ceb0000 7503f883 7ae85107 59000002
01ae10f0: c258016a 8b55000c 5d8b53ec 758b5608 7d8b570c 75f68510
01ae1108: 603d8309 0001ae53 fe8326eb 83057401 227502fe ae58dca1

Loaded modules:
(* denotes the module causing the exception)
00400000-0040ffff C:\bea\jrockit81sp6_142_10\bin\java.exe
7c920000-7c9b3fff C:\WINDOWS\system32\ntdll.dll
7c800000-7c91bfff C:\WINDOWS\system32\kernel32.dll
77da0000-77e48fff C:\WINDOWS\system32\ADVAPI32.dll
77e50000-77ee0fff C:\WINDOWS\system32\RPCRT4.dll
77be0000-77c37fff C:\WINDOWS\system32\MSVCRT.dll
00410000-0063bfff C:\bea\jrockit81sp6_142_10\jre\bin\jrockit\jvm.dll
76b10000-76b39fff C:\WINDOWS\system32\WINMM.dll
77d10000-77d9efff C:\WINDOWS\system32\USER32.dll
77ef0000-77f35fff C:\WINDOWS\system32\GDI32.dll
71a20000-71a36fff C:\WINDOWS\system32\WS2_32.dll
71a10000-71a17fff C:\WINDOWS\system32\WS2HELP.dll
7c340000-7c395fff C:\bea\jrockit81sp6_142_10\bin\MSVCR71.dll
76300000-7631cfff C:\WINDOWS\system32\IMM32.DLL
62c20000-62c28fff C:\WINDOWS\system32\LPK.DLL
73fa0000-7400afff C:\WINDOWS\system32\USP10.dll
77fc0000-77fd0fff C:\WINDOWS\system32\Secur32.dll
76cb0000-76ccffff C:\WINDOWS\system32\NTMARTA.DLL
76f30000-76f5bfff C:\WINDOWS\system32\WLDAP32.dll
76990000-76acbfff C:\WINDOWS\system32\ole32.dll
71b70000-71b82fff C:\WINDOWS\system32\SAMLIB.dll
10000000-10018fff C:\bea\jrockit81sp6_142_10\jre\bin\java.dll
00c20000-00c2dfff C:\bea\jrockit81sp6_142_10\jre\bin\verify.dll
00ca0000-00ca6fff C:\bea\jrockit81sp6_142_10\jre\bin\hpi.dll
00c80000-00c8efff C:\bea\jrockit81sp6_142_10\jre\bin\net.dll
719c0000-719fdfff C:\WINDOWS\system32\mswsock.dll
60fd0000-61024fff C:\WINDOWS\system32\hnetcfg.dll
71a00000-71a07fff C:\WINDOWS\System32\wshtcpip.dll
76ef0000-76f16fff C:\WINDOWS\system32\DNSAPI.dll
76f80000-76f87fff C:\WINDOWS\System32\winrnr.dll
76f90000-76f95fff C:\WINDOWS\system32\rasadhlp.dll
01ab0000-01abafff C:\WINDOWS\system32\psapi.dll
01d80000-01d87fff C:\bea\jrockit81sp6_142_10\jre\bin\nio.dll
02d30000-02d35fff C:\bea\jrockit81sp6_142_10\jre\bin\ioser12.dll
01ad0000-01ad4fff C:\bea\weblogic81\server\bin\wlntio.dll
01ae0000-01ae6fff *C:\student\course_wls_troubleshooting\bin\kboom.dll
045b0000-04670fff C:\bea\jrockit81sp6_142_10\jre\bin\dbghelp.dll
77bd0000-77bd7fff C:\WINDOWS\system32\VERSION.dll


"ExecuteThread: '14' for queue: " id=25 idx=0x34 tid=3572 lastJavaFrame=0x029BFC34

Stack 0: start=0x02980000, end=0x029C0000, guards=0x02983000 (ok), forbidden=0x02981000
Thread Stack Trace:
at _Java_troubleshooting_common_KBoom_killMeNow@8+17()@0x01AE1011
-- Java stack --
at troubleshooting/common/KBoom.killMeNow()V(Native Method)
at troubleshooting/common/KBoom.terminateWebLogic()V(KBoom.java:11)
at troubleshooting/servlets/WebLogicCoreDump.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V(WebLogicCoreDump.java:24)
at javax/servlet/http/HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(HttpServlet.java:853)
at weblogic/servlet/internal/ServletStubImpl$ServletInvocationAction.run()Ljava/lang/Object;(ServletStubImpl.java:1077)
at weblogic/servlet/internal/ServletStubImpl.invokeServlet(Ljavax/servlet
/ServletRequest;Ljavax/servlet/ServletResponse;Lweblogic/servlet/internal/FilterChainImpl;)V(ServletStubImpl.java:465)
at weblogic/servlet/internal/ServletStubImpl.invokeServlet(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(ServletStubImpl.java:348)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run()Ljava/lang/Object;(WebAppServletContext.java:7047)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(Lweblogic/security/subject/AbstractSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(AuthenticatedSubject.java:321)
at weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(SecurityManager.java:121)
at weblogic/servlet/internal/WebAppServletContext.invokeServlet(Lweblog
ic/servlet/internal/ServletRequestImpl;Lweblogic/servlet/internal/ServletResponseImpl;)V(WebAppServletContext.java:3902)
at weblogic/servlet/internal/ServletRequestImpl.execute(Lweblogic/kernel/ExecuteThread;)V(ServletRequestImpl.java:2773)
at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:224)
at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace

Additional information is available in:
C:\student\course_wls_troubleshooting\bin\jrockit.3420.dump
C:\student\course_wls_troubleshooting\bin\jrockit.3420.mdmp
If you see this dump, please open a support case with BEA and
supply as much information as you can on your system setup and
the program you were running. You can also search for solutions
to your problem at http://forums.bea.com in
the forum jrockit.developer.interest.general.
===== END DUMP
===============================================================

如果在启动Server的脚本里,加上-Djrockit.waitonerror,增加的位置在'java' 之后,'weblogic.Server' 之前。这样在Server Crash时,会弹出窗口:
不要选是或否,获取此时的Thread Dump。

没有评论: