Application architecture for high volume data transaction
Hello,
I came across a performance issue in one of the application developed for our customer. The application is n tier application involving a rich client, EJB application server, DAO layer and data base.
The data is transferred within the layers from DAO to client using value objects. DAO returns the model in case of select query and inserts or updates the database using the value objects. I believe this is the standard practice for OO development and works fine when the data volume is low.
However, when the same architecture is used for query where thousands of records are retrieved and for mass updates where thousands of records are updated in database, the performance degrades drastically.
I would like to have your expertise on what is the best architecture in this case where huge data is processed and moved back and forth from server to client. Is there any standard framework?
Zubs
[940 byte] By [
Zubsza] at [2007-10-3 5:59:25]
> Hello,
>
> I came across a performance issue in one of the
> application developed for our customer. The
> application is n tier application involving a rich
> client, EJB application server, DAO layer and data
> base.
OK.
> The data is transferred within the layers from DAO to
> client using value objects. DAO returns the model in
> case of select query and inserts or updates the
> database using the value objects. I believe this is
> the standard practice for OO development and works
> fine when the data volume is low.
> However, when the same architecture is used for query
> where thousands of records are retrieved
What user needs 1000s of records to update at once? Why do you have to retrieve all the records? Of course it doesn't scale - it's an O(n) problem, where n is the number of rows in the database.
It's worse with updates, especially if you don't batch your SQL statements. If you do one network roundtrip per INSERT, and there are n of them, of course your performance will degrade. You need to figure out how to batch your statements into more manageable chunks to cut down on the network latency.
> and for mass
> updates where thousands of records are updated in
> database, the performance degrades drastically.
You probably shouldn't do mass batch updates this way. Better to use ETL tools in a batch mode.
> I would like to have your expertise on what is the
> best architecture in this case where huge data is
> processed and moved back and forth from server to
> client. Is there any standard framework?
Check the batching issue. Measure the # of roundtrips to confirm this and then figure out how to reduce them.
Don't give users access to thousands of rows at once. Google gives me billions of pages, but only 25 at a time. use WHERE clauses to let clients pare down the result set that they operate on.
%
Well I forgot to mention in the first mail.
The mass update is the customer's requirement. They have provided special button for mass update.
My idea is to keep the existing functionality as described as it is and only for mass update, as a special case, make specific changes so that the value objects are not involved.
This not very much OO ways.. however currently I see only option.
> Well I forgot to mention in the first mail.
>
> The mass update is the customer's requirement. They
> have provided special button for mass update.
>
> My idea is to keep the existing functionality as
> described as it is and only for mass update, as a
> special case, make specific changes so that the value
> objects are not involved.
That's fine.
Profile the app. I'm confident that network traffic is what's killing you. Figure out how to batch INSERT/UPDATE requests and your performance will improve.
%
