getRuntime.exec spawning multiple processes
I am having a very weird issue, which I have been trying to debug for a while. I am executing a command on a unix box using:
Process proc = Runtime.getRuntime().exec(command, null, workDir);
proc.waitFor();
Where the command is the "zip" program which zips several files up selected by the user. The reason I am using the OS to zip instead of doing it in memory, is because the files are rather large, and when multiple users start using this web app it will bring the application server down to its knees.
The problem seems to only occur when it involves a large zip file, what will happen is that it will zip up the file, and when its close to being done (or done), it will spawn another identical zip process. This can go on forever unless the process is terminated on the unix box.
I have debugged using getInputStream and getErrorStream, and have not received any errors at all... until I terminate the processes.
String dbug;
BufferedReader input =new BufferedReader(new InputStreamReader(proc.getInputStream()));
BufferedReader err =new BufferedReader(new InputStreamReader(proc.getErrorStream()));
System.out.println(">> START DEBUG OUTPUT stdout --");
while ((dbug = input.readLine()) !=null)
{
System.out.println(">>"+dbug);
}
System.out.println(">> END DEBUG OUTPUT stdout --");
input.close();
System.err.println(">> START DEBUG OUTPUT stderr --");
while ((dbug = err.readLine()) !=null)
{
System.err.println(">>"+dbug);
}
System.err.println(">> START DEBUG OUTPUT stderr --");
err.close();
I have run the exact same process on unix with the exact same arguments and have absolutely no problems.
Any thoughts? Anything will help.
Thank you in advance.
[2377 byte] By [
d0gmatica] at [2007-11-26 12:20:18]
# 1
I don't know why you get this problem but you do not follow 'best practice' for dealing with stdout and stderr. It would be worth you reading http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html .
# 2
Thanks for the link. I actually did take a look at the document and implemented much of the code suggested previously, but with no luck have been trying other things I have found online.PS> I am fairly new to java.
# 3
> Thanks for the link. I actually did take a look at
> the document and implemented much of the code
> suggested previously, but with no luck have been
> trying other things I have found online.
You have not implemented what I feel is the most important part i.e. the use of threads to read stdout and stderr.
# 4
Will try that, hopefully I will see something different.
# 5
> You have not implemented what I feel is the most
> important part i.e. the use of threads to read stdout
> and stderr.
I implemented this, and did not really make any difference. No unexpected output, and no errors. Although, while doing this and testing a few things I left out:
proc.waitfor();
and I did not get the spawning of processes! I am going to test a bunch more times with and without this to be sure that is the problem.
Thanks,
# 6
Well... it seems like there might be a possible bug with waitFor(). In my case, as mentioned earlier, it will spawn the same process again which can go on in an endless loop.
Instead of using waitFor() I resorted to test if the file exists, since the zip process creates a temporary file name until its done this is possible:
File zipFile = new File(zipDir+pathSeparato+zipFileName+".zip");
while(!zipFile.exists()){
try{
Thread.sleep(5000);
}
catch(InterruptedException ie)
{
ie.printStackTrace();
}
}
# 7
> Well... it seems like there might be a possible bug
> with waitFor(). In my case, as mentioned earlier, it
> will spawn the same process again which can go on in
> an endless loop.
I doubt it. I use waitFor() all the time and have never had a problem like this. Have you tried checking the return code from the waitFor()? Are you looping without realizing it?
I have just done a grep on all my source code and I have over 50 waitFor() and they all work properly on Windows XP/2000 and Linux FC3/FC4/FC5.
Message was edited by:
sabre150
# 8
You are right, while what I previously mentioned worked twice, after testing various times it produced the same problem. I was wrong to say that it is waitFor().
This time I can say I know exactly what the problem is...
This is a webapp using a servlet, apparently the servlet is being called again by the server which is causing the additional spawning of processes. Since the zipping of the file can take a while... several minutes, it seems that the wait is so long that the servlet is called again... producing the unwanted behavior.
Is there anyway that you know of that will stop this from occurring? I am currently looking for possible solutions online.
Thanks for your help
# 9
> Where the command is the "zip" program which zips
> several files up selected by the user. The reason I
> am using the OS to zip instead of doing it in memory,
> is because the files are rather large, and when
> multiple users start using this web app it will bring
> the application server down to its knees.
So you've implemented a "solution" where you create extra processes (more work) and transmits data between those processes (more work) to solve a non-existent problem. If you're writing data to disk and reading it back, that's even more unnecessary work.
You can write your own code to read files and write a zip archive of those files directly to the servlet's response. And unless you screw things up with bad coding, the size of the files doesn't affect the memory usage at all. The ZIP format was designed to make streaming data like that a simple thing to do.
# 10
> This is a webapp using a servlet, apparently the
> servlet is being called again by the server which is
> causing the additional spawning of processes. Since
> the zipping of the file can take a while... several
> minutes, it seems that the wait is so long that the
> servlet is called again... producing the unwanted
> behavior.
What you need to do is build anti-replay code into your servlet. Store a MRU cache list if IDs in the session. During the post check to see if the ID is in your cache. If it is, then ignore the request. If it isn't, put it in your cache and continue. Be sure to use a new ID on each page that you produce and pass it to your servlet.
# 11
> > Where the command is the "zip" program which zips
> > several files up selected by the user. The reason
> I
> > am using the OS to zip instead of doing it in
> memory,
> > is because the files are rather large, and when
> > multiple users start using this web app it will
> bring
> > the application server down to its knees.
>
> So you've implemented a "solution" where you create
> extra processes (more work) and transmits data
> between those processes (more work) to solve a
> non-existent problem. If you're writing data to disk
> and reading it back, that's even more unnecessary
> work.
>
> You can write your own code to read files and write a
> zip archive of those files directly to the servlet's
> response. And unless you screw things up with bad
> coding, the size of the files doesn't affect the
> memory usage at all. The ZIP format was designed to
> make streaming data like that a simple thing to do.
This would be better, but the OP will also have to use PipedInputStream and PipeOutputStream and create an extra thread to read and write between them. This just might be too much for the OP.
# 12
> > You can write your own code to read files and write
> a
> > zip archive of those files directly to the
> servlet's
> > response. And unless you screw things up with bad
> > coding, the size of the files doesn't affect the
> > memory usage at all. The ZIP format was designed
> to
> > make streaming data like that a simple thing to
> do.
>
>
> This would be better, but the OP will also have to
> use PipedInputStream and PipeOutputStream and create
> an extra thread to read and write between them. This
> just might be too much for the OP.
That just shows that you don't know how to do it either. No wonder the world is full of rubbish code. Let me give you the outline.ZipOutputStream zos = new ZipOutputStream(response.getOutputStream());
ZipEntry ze = new ZipEntry("abc.txt");
// call methods of ze to set other information
zos.putNextEntry(ze);
// insert code here to copy data from the file to the ZipOutputStream
zos.closeEntry();
zos.finish();
Like I said, it's easy. And you don't have to muck about with pipes or processes or anything like that. Just plain old Java with plain old file copying.
# 13
Dr.Clap said:
> So you've implemented a "solution" ....
Is there really need for such sarcasm?
> That just shows that you don't know how to do it
> either.
Well, thank god you are here.
I had implemented that code before even posting on this form, which as I mentioned before brought the app server down to its knees. that is why the "solution" was chosen, which by the way solved that problem... which by the way... exists.
All I am doing is creating a zip file which needs to be written to the file system... that is it... nothing else.
The file is never being read back, nor is anything else being done with the zip file within java, once the zip file is generated the only other step is for a user to download it.
The non-existent problem was pretty existent on our servers, and using the OS zip command solved it, thanks for your great feedback.
> What you need to do is build anti-replay code into
> your servlet. Store a MRU cache list if IDs in the
> session. During the post check to see if the ID is
> in your cache. If it is, then ignore the request. If
> it isn't, put it in your cache and continue. Be sure
> to use a new ID on each page that you produce and
> pass it to your servlet.
Thanks Caffeine0001 that sounds good, and I did something similar which stopped the spawning of processes. Though brought another issue. See this is done also with XSLT, which is why the waiting for the process to finish is important, so the user can download the zip file when its done.
So I have solve the problem of the spawning processes, but now the page is transformed and shows the "download" button when the file is not ready. Though I think that if I store the waitFor() in a session attribute, it should be able to pick up where it left off, and wait till its done.
Thanks again.
# 14
> Though brought another issue. See this is
> done also with XSLT, which is why the waiting for the
> process to finish is important, so the user can
> download the zip file when its done.
There is no need to create a file with the output of an XSLT. It can be piped straight into a zip stream and the result can be piped directly to the user.
Unless I am missing something, there is no need to generate any intermediate file.
# 15
> This would be better, but the OP will also have to
> use PipedInputStream and PipeOutputStream and create
> an extra thread to read and write between them. This
> just might be too much for the OP.
Must of been a brain ****. I was up to 3am last night doing a code push.
