JVM issues after applying August Microsoft secuirty patch MS06-040
We applied Microsoft patch MS06-040 as part of our corporate patching policy. After the patch, java apps that were running fine started to give the following error.
java -Xms512M -Xmx1024M -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
(See Microsoft knowledge base for this patch)
www.microsoft.com/technet/security/Bulletin/MS06-040.mspx
Previously, this setting was fine. There is enoguh memory on the box for at least 1.4GB of heap. It seems the patch has affected the ability of JVM to allocate memory.
We are running Sun JVM 1.4.2_08 on Windows 2003 Advanced Server SP1 32 bit. We have logged a ticket with Microsoft but were trying to gather as much infomation on whether anybody else has experienced this.
[815 byte] By [
vhbsca] at [2007-10-3 2:30:13]

One of our clients is reporting this exact same problem after applying MS06-040.
We have found that by drastically decreasing the Java Max Heap size (-xmx), we are able to get around the problem. We have had to drop it from 1024 to about 400mb. Anything higher causes the Error occurred during initialization of VM message.
However, some of our other servers which have received the same MS patch have had no problems whatsoever, and are still able to allocate a full gigabyte to the heap.
We are unable to identify why some servers are having this problem and others are not. Some of these machines are virtually identical.
We would greatly appreciate any more information on this.
Thank you,
-Matthew
The issue arises from the patch introducing a new version of a DLL. DLLs (and executables) have a 'preferred' base address, if there isn't a clash during loading DLLs then they'll occupy the area their preferred address specifies.
In the case of this problem the DLL plants itself pretty squarely in the middle of the addressible 2GB region. Combine this with other system DLLs and the Java DLLs and you don't actually have a contiguous 1GB region left.
While I know there's a hotfix, I am uncertain why the heap memory is reserved the way it is. Is there are reason that the entire 1GB region must be contiguous when, internally, the heap is divided into a number of regions (perm, young, tenured,...) depending upon the GC type. Is there a technical reason these regions aren't reserved by the VM separately? Do the regions always grow in proportion to each other or do they resize independantly within the maximum size specified?
I noted in testing that Java 5.0 exhibits the same issue so they haven't changed the approach in newer releases. We've forced to provide a distributed application purely due to heap size limitations in windows (we haven't be able to reliably achieve a heap size above 1200MB for all customers).I guess the sooner the world moves to 64bit addressing the better.
We applied the Netapi32.dll and still getting the warning in event log "A desktop heap allocation failed". Please note that the windows 2003 is not R2.
Windows 2003 Advanced server SP1
Using JRE version 1.5.0_06
Has any one run into problems of memory leak or high handle count with windows 2003 SP1 and JRE version 1.5.0_06.