checking stack size
I have written a JNI app and it seems to work fine, but processing with very large files I run out of stack space.
I have solved this issue for the moment by changing the stack size with the -Xss param to java but this is not the ideal issue because with even bigger files I still will have issues.
So, I have 3 questions:
1. Is there any way (in the c++ dll) to query the total stack size ?
2. Is there any way to determine free (available) stack size ?
3. Is there any way to increase the stack size when the dll is already running (if it starts to get low)
I look forward to your thoughts
Ding
> I have written a JNI app and it seems to work fine,
> but processing with very large files I run out of
> stack space.
>
That suggests that you have a design problem.
As a guess you are using recursion. Start by unrolling the recursion code to produce a non-recursive solution.
>
> 1. Is there any way (in the c++ dll) to query the
> total stack size ?
Presumably you mean the java stack size. It doesn't exist until you start the VM. Once you start it you can't change it.
> 2. Is there any way to determine free (available)
> stack size ?
> 3. Is there any way to increase the stack size when
> the dll is already running (if it starts to get low)
>
No. The VM must be restarted.
> > I have written a JNI app and it seems to work
> fine,
> > but processing with very large files I run out of
> > stack space.
> >
>
> That suggests that you have a design problem.
>
> As a guess you are using recursion. Start by
> unrolling the recursion code to produce a
> non-recursive solution.
Good Guess but there is no recursion in my code :-)
While I am always trying to optimize my design, in this case I am constrained by 3rd party c libraries that are using very large multi dimensional arrays of doubles that are neccessary for lots of complicated math doing triangualtion of satellite data. So, I have no control over how much memory they want to consume.
> >
> > 1. Is there any way (in the c++ dll) to query the
> > total stack size ?
>
> Presumably you mean the java stack size. It doesn't
> exist until you start the VM. Once you start it you
> can't change it.
>
I was referring to the native C stack, used within the c++ dll, for which the java VM starts via the -Vss flag. I know the initial total as I am setting it at runtime. I just wanted to print the value from within the dll. #2 (below) is more important though.
> > 2. Is there any way to determine free (available)
> > stack size ?
This is the part I was mostly hoping for an answer to, as I need to communicate with the parent java app if the stack space gets too low, instead of just allowing the dll to crash when it runs out
> > 3. Is there any way to increase the stack size
> when
> > the dll is already running (if it starts to get
> low)
> >
>
> No. The VM must be restarted.
if that is true, It i a bummer :(
> > > I have written a JNI app and it seems to work
> > fine,
> > > but processing with very large files I run out
> of
> > > stack space.
> > >
> >
> > That suggests that you have a design problem.
> >
> > As a guess you are using recursion. Start by
> > unrolling the recursion code to produce a
> > non-recursive solution.
>
> ood Guess but there is no recursion in my code :-)
>
> While I am always trying to optimize my design, in
> this case I am constrained by 3rd party c libraries
> that are using very large multi dimensional arrays of
> doubles that are neccessary for lots of complicated
> math doing triangualtion of satellite data. So, I
> have no control over how much memory they want to
> consume.
>
> ood Guess but there is no recursion in my code :-)
>
Then it is unclear as to what you think a good solution would entail.
Do you intend to just to keep retrying the processing step with ever increasing stack sizes? Will that not impact the processing time? Why not just start with a larger stack size? And if the stack size just continues to grow because the file sizes are continuing to grow then what?
> While I am always trying to optimize my design, in
> this case I am constrained by 3rd party c libraries
> that are using very large multi dimensional arrays of
> doubles that are neccessary for lots of complicated
> math doing triangualtion of satellite data. So, I
> have no control over how much memory they want to
> consume.
>
One solution would be to create a C app, not a java one. Find the maximum amount of stack space that you can allocate via a C app and still operate (this depends on what happens with the file itself.) The output goes to a file.
Your java app just starts that in a separate process space.
> >
>
> I was referring to the native C stack, used within
> the c++ dll, for which the java VM starts via the
> -Vss flag. I know the initial total as I am setting
> it at runtime. I just wanted to print the value from
> within the dll. #2 (below) is more important
> though.
>
That would depend on your OS and compiler.
> > > 2. Is there any way to determine free
> (available)
> > > stack size ?
>
> This is the part I was mostly hoping for an answer
> to, as I need to communicate with the parent java app
> if the stack space gets too low, instead of just
> allowing the dll to crash when it runs out
>
That is a two part question. You want to know the size and you want to know if it gets 'too low'.
You suggested that this is all stack based processsing. So unless you have C methods that are called by the library and in addition it is recursive in nature then this is not possible. There is no point where you could detect it.
You can analyze the file yourself and compute a size before you start processing. But if that is what you are doing then you should do it first and let the java app know the result.
Other than that most OSes allow you to catch 'system exceptions' (which might or might not be actual C++ exceptions.) One of these would be generated when the stack overflows.
You then convert into something that java understands.
For example in Windows there is a function that allows you to set up system exceptions so that they cause a C++ exception to be thrown. You would then catch this exception and convert it into a java exception.
That lets you know that it didn't work but then what?
For example in windows you can create a custom stack for the dll but only when the dll loads (so far as I know.) So to restart this you would have to unload the dll and that means that you will have to have everything wrapped in a custom class loader (which is the only way to cause a dll to be unloaded.)
Not to mention that in my experience the windows function that I mentioned above doesn't necessarily always catch everything.
> > >
> >
> > No. The VM must be restarted.
>
> if that is true, It i a bummer :(
You can however, on most OSes, increase the stack size for the shared library. This is done on start up of the shared library. To resize it however means that the shared library must be reloaded. And the only way you can do that in the Sun VM is with a class loader.
