There might be some issue with your code like creating large arrays for some reason which don't get garbage collected or some of the other memory leak type things which can be disgnosed in Java 6 using various tools:
http://java.sun.com/developer/technicalArticles/J2SE/monitoring/index.html (or just by code inspection)
But if you've checked for that, you can always encourage Java to use more memory by adding a command-line flag like -mx512m (assuming you have that much available)
That setting might just delay an underlying memory issue, but it sounds like you just need that much memory.
Here are the options:
http://java.sun.com/j2se/1.5.0/docs/tooldocs/windows/java.html
Some additional "non-standard" command-line options can also help diagnose or get around memory issues:
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp