Best approach to read a large amount of data and write it to a file
What would be the best way to accelerate the process of:
1)Reading from a db2 about hundred of thousands of record
2)Writing the data to a txt file (approx. size is 400MB)
The current application is getting the result set as follows:
paramStmt = con.createStatement();
paramRS = paramStmt .executeQuery(query);
While looping the record set, writing to a txt file is performed as follows:
StringBuffer sb = new StringBuffer();
sb.append ...........
sb.append ...........
FileWriter fw = new FileWriter("results.txt");
BufferedWriter bw = new BufferedWriter (fw);
bw.write(sb);
bw.close;
fw.close;
There are no errors in the actual application, however I would like to enhance the overall performance with any better approach
Thanks
> What would be the best way to accelerate the process
> of:
> 1)Reading from a db2 about hundred of thousands of record
> 2)Writing the data to a txt file (approx. size is
> 400MB)
Use the tools (export) that come with the database and don't use java at all.
Perfect i will check the JProfiler to assess the performance. Thanks.
On the other side, is there any other Known approach that will enhance the performance while getting the result set... for example, if I get the records by chunks of 1000.
And if I to use the FileOutputStream and write the data as objects, will it enhance the performance.
My understanding is that the FileOutputStream is a bridge from Byte to Char mode. So it will read the object members values then write it as ASCII
In fact with this way, there is one more step in programming:
1-Read the record
2-Store the values in the object .................the extra step
3-Write it
Although there is one additional step, would this approach accelerate the file writting process?
> My understanding is that the FileOutputStream is a
> bridge from Byte to Char mode. So it will read the
> object members values then write it as ASCII
> In fact with this way, there is one more step in
> programming:
> 1-Read the record
> 2-Store the values in the object .................the
> extra step
> -Write it
>
FileOutputStream writes bytes and nothing else.
My presumption is that you thought that you would doing object output (like via ObjectOutputStream.)
And as I said I doubt that will meet your business needs.
> Although there is one additional step, would this
> approach accelerate the file writting process?
I doubt that if you profiled the application you would find that the file writing is the bottleneck. Not unless you are writing to a remote file on a busy network (and the solution to that is to not do it.)
