GetLocalInt et al crash during garbage collection under stress conditions

I have been working on a crash for two months now and am preparing a bare-bones test case for SUN - I have pretty much ruled out any issue in my native code.

Before I do that, I am wondering if anyone has seen this problem before; in brief, I define a couple of breakpoints and stress-test with a number of threads executing the same Java code. Repeated calls to these JVMTI functions from multiple threads (even sequenced) in my callback function will cause a crash during garbage collection, with typically the info in the hs_* file as shown below. This affects any rev up to 1.5_09 and 1.6RC (haven't downloaded 1.6 FCS yet).

The current thread is either a VMThread or GCTaskThread, the memory read is usually "low", sometimes 0x9, or it can happen while writing an address typically in the 1.7GB range.

The only way to avoid the crash is to not call these JVMTI functions, and pretend the value is some canned one. It takes about a couple of hundred iterations before the crash occurs... I have tested with -XX:+UseSerialGC and w/o -server and the frequency of the crash is lower.

All in all, my symptoms do not appear to be different than what's described in http://forum.java.sun.com/thread.jspa?forumID=32&threadID=748873

It is reproducible on Win32, Solaris and Linux, and with the test case, it can occur within 15 seconds. Anyone have any suggestions?

#

# An unexpected error has been detected by HotSpot Virtual Machine:

#

# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d962d53, pid=5488, tid=6236

#

# Java VM: Java HotSpot(TM) Server VM (1.5.0_09-b01 mixed mode)

# Problematic frame:

# V [jvm.dll+0x92d53]

#

T H R E A D

Current thread (0x00ab53f8): VMThread [id=6236]

siginfo: ExceptionCode=0xc0000005, reading address 0x000006f8

Registers:

EAX=0x000006f8, EBX=0x00000000, ECX=0x0099fc58, EDX=0x6dbba520

ESP=0x0099fb94, EBP=0x0099fc58, ESI=0x2c12d230, EDI=0x0099fc58

EIP=0x6d962d53, EFLAGS=0x00010283

Top of Stack: (sp=0x0099fb94)

0x0099fb94:00ab5528 2c12d1fc 6da963d3 2c12d230

0x0099fba4:6db77751 0099fc58 00a490c8 00ab5528

0x0099fbb4:00000000 0099fc58 6db77814 00000000

0x0099fbc4:0099fc58 0099fc58 0099fc58 00a490c8

0x0099fbd4:6db78057 0099fc58 00000000 6db511fe

0x0099fbe4:0099fc58 0099fc58 00a49368 6d9956dc

0x0099fbf4:0099fc58 00ab4af8 00a49368 00000000

0x0099fc04:00a490c8 6d9638c2 00000000 00000001

Instructions: (pc=0x6d962d53)

0x6d962d43:24 08 57 8b f9 8b 06 85 c0 74 3e 3b 47 1c 73 39

0x6d962d53:8b 08 83 e1 03 80 f9 03 75 06 8b 00 24 fc eb 0a

Stack: [0x00960000,0x009a0000), sp=0x0099fb94, free space=254k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

V [jvm.dll+0x92d53]

VM_Operation (0x0007f990): generation collection for allocation, mode: safepoint

, requested by thread 0x00038880

P R O C E S S

Java Threads: ( => current thread )

0x00acbf68 JavaThread "breakpoint producer 3" daemon [_thread_blocked

, id=5456]

0x2bf08518 JavaThread "breakpoint producer 2" daemon [_thread_blocked

, id=6088]

0x00acbd50 JavaThread "breakpoint producer 1" daemon [_thread_blocked

, id=5800]

0x00ac4ce0 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=6644]

0x00ac39e0 JavaThread "CompilerThread1" daemon [_thread_blocked, id=2668]

0x00ac2b78 JavaThread "CompilerThread0" daemon [_thread_blocked, id=2888]

0x00ac1a70 JavaThread "AdapterThread" daemon [_thread_blocked, id=7900]

0x00ac0dd8 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=1364]

0x00ab8328 JavaThread "Finalizer" daemon [_thread_blocked, id=6940]

0x00ab76c0 JavaThread "Reference Handler" daemon [_thread_blocked, id=5816]

0x00038880 JavaThread "main" [_thread_blocked, id=5868]

Other Threads:

=>0x00ab53f8 VMThread [id=6236]

0x00ac5ec0 WatcherThread [id=8024]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])

[0x00037ec8/0x000006f8] Threads_lock - owner thread: 0x00ab53f8

[0x00038048/0x000006b8] Heap_lock - owner thread: 0x00038880

Heap

def new generationtotal 23616K, used 21188K [0x03ad0000, 0x05460000, 0x0a130

000)

eden space 21056K, 100% used [0x03ad0000, 0x04f60000, 0x04f60000)

from space 2560K,5% used [0x04f60000, 0x04f81090, 0x051e0000)

tospace 2560K,0% used [0x051e0000, 0x051e0720, 0x05460000)

tenured generationtotal 104896K, used 0K [0x0a130000, 0x107a0000, 0x23ad0000

)

the space 104896K,0% used [0x0a130000, 0x0a130000, 0x0a130200, 0x107a0000)

compacting perm gen total 16384K, used 1801K [0x23ad0000, 0x24ad0000, 0x2bad00

00)

the space 16384K, 10% used [0x23ad0000, 0x23c92440, 0x23c92600, 0x24ad0000)

No shared spaces configured.

Dynamic libraries:

<REMOVED>

VM Arguments:

jvm_args: -ea -Xms128m -Xmx512m -XX:MaxPermSize=128m -XX:NewRatio=4 -agentlib:Debugger

java_command: com.XXX.diagnostics.debugger.crash.Crash standalone

Launcher Type: SUN_STANDARD

Environment Variables:

JAVA_HOME=C:/java/1.5.x/jdk1.5.0_09

JAVA_COMPILER=NONE

PATH=<REMOVED>

USERNAME=<REMOVED>

LD_LIBRARY_PATH=<REMOVED>

OS=Windows_NT

PROCESSOR_IDENTIFIER=x86 Family 15 Model 4 Stepping 10, GenuineIntel

S Y S T E M

OS: Windows XP Build 2600 Service Pack 2

CPU:total 1 (cores per cpu 1, threads per core 2) family 15 model 4 stepping 10,

cmov, cx8, fxsr, mmx, sse, sse2, sse3, ht

Memory: 4k page, physical 2095192k(1104740k free), swap 4032856k(2998736k free)

vm_info: Java HotSpot(TM) Server VM (1.5.0_09-b01) for windows-x86, built on Sep

7 2006 13:40:20 by "java_re" with MS VC++ 6.0

[6013 byte] By [sanokistokaa] at [2007-11-26 13:42:30]
# 1
This may be related: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6497210
alan.batemana at 2007-7-8 0:00:23 > top of Java-index,Developer Tools,Debugging and Profiling Tool APIs...
# 2
Yes, this appears to be my problem. Any idea when the fix will be checked in, and in what release? Must go into the 1.5 tree, as far as I am concerned.thanks
sanokistokaa at 2007-7-8 0:00:23 > top of Java-index,Developer Tools,Debugging and Profiling Tool APIs...
# 3
I took out the calls to GetLocalObject() from my test case and it's still crashing with GetLocalInt et al, but seemingly only with -server, so I have filed it with SUN.
sanokistokaa at 2007-7-8 0:00:23 > top of Java-index,Developer Tools,Debugging and Profiling Tool APIs...
# 4
The fix for this bug should be in a jdk7 build soon. Once it has baked for a bit, we can look to make it available in an update release.
alan.batemana at 2007-7-8 0:00:23 > top of Java-index,Developer Tools,Debugging and Profiling Tool APIs...
# 5
Thanks. Please seriously consider making both the GetLocalObject and GetLocalInt et al fixes available in 1.5 - as you can imagine, they are cornerstone blocks for building debuggers (which is what I have done).Is the fix you are testing related to -server or JVMTI itself?
sanokistokaa at 2007-7-8 0:00:23 > top of Java-index,Developer Tools,Debugging and Profiling Tool APIs...
# 6
The fix in the jvmti implementation in the VM and it isn't specific to client or server. Once the fix is available in the jdk7 build then perhaps you could do some testing with it. Once it has baked we can look to get the fix into a 5.0 and 6.0 update.
alan.batemana at 2007-7-8 0:00:23 > top of Java-index,Developer Tools,Debugging and Profiling Tool APIs...
# 7
Yes, I will be happy to test for you. Let me know if you can't see my email address.
sanokistokaa at 2007-7-8 0:00:23 > top of Java-index,Developer Tools,Debugging and Profiling Tool APIs...
# 8
I tested it (JDK7, Build 10). The fix is right. Any idea when this could be implemented to v1.5 as well?
JTelefosa at 2007-7-8 0:00:23 > top of Java-index,Developer Tools,Debugging and Profiling Tool APIs...
# 9
I am dying to get this into 1.5... I have a full-fledged web-based debugger built into a major application in the financial services world and this is holding up providing the ultimate debugging tool.
sanokistokaa at 2007-7-8 0:00:23 > top of Java-index,Developer Tools,Debugging and Profiling Tool APIs...
# 10
This one is likely to be in 6.0 update 3 but I don't know about 5.0 yet. If this issue is a blocking issue for an important financial application then I would suggest logging a support call and escalate the bug.
alan.batemana at 2007-7-8 0:00:23 > top of Java-index,Developer Tools,Debugging and Profiling Tool APIs...