Large memory usage versus DB overhead.
Hey
I have (or will have) a web application based on Spring, Hibernate, Quartz, and other technologies, with backend as Postgres. There will be quite many concurrent users (approximately 10000). Each of such a user has relatively big amount of data stored in database. Now the design is based on paradigm: user logs on -> whole data that belong to that user is loaded from DB and stored in some static HashMap ([user_id, UserBean]) where "UserBean" is representing all user data and it's quite a fine grained object (very big). The reason for that is: it's an AJAX-based app, and user think-time on a page is quite short, so many XMLHttpRequests come in short time, and as big percent of those requests involve database activity, i did as i did to not overload database. But when the amount of users will grow rapidly (let's say mentioned 10000) i fear that there will be problem with memory usage. So, my question is: What strategy should i take to both: 1) don't cause memory issues, and 2) don't cause too big performance overhead issued by often DB activity ?Thanks in advance :) . Tom
[1110 byte] By [
rojeka] at [2007-10-2 2:07:17]
On thing I have to question is your assumption that you will overload the DB. What DB are your using? It may very well be that you are right and that you will push the DB past it's limit but my feeling is that there are a lot of DBs that can easily handle the loads you mention.
I do agree that you shouldn't keep going back to the DB for the same info. However, I'm not sure that I agree that retrieving all the info for that user upfront is a good strategy. Do you need all that info on every log in? You can implement a hybrid solution that stores things that you have retrieved without retrieving everything.You can also implement memory sensitive caches and similar approaches to mitigate these issues.
If you are using a connection pool, it's extremely unlikely you will overload the DB. What you may run into is contention for the connections. This will be exacerbated by executing big long running queries.
I concur with the prior remark, that a database is a solid work horse.
There are some issues though. Fine grained and database do not go well together; databases are sparse data and not well suited for interrelated information.
You could store your data in an compressed XML blob in the database.
In that way you can SAX parse the data (for instance by a GZipInputStream.
Should ever the DB performance prove a bottleneck, switch to holding the blob in RAM.
A HashMap seems terrible, maybe not somuch for the RAM amount, but
for the persistence during restarts, cleanliness and such.
Consider load balancers, for the web app, and the DB server.
Cache anything that is read-only or read-mostly and is accessed often. Lazy load is another good option, though whether it actually increases or decreases the load on your database will depend on how your data is accessed in the application. But normally, it does help.- Saish
Saisha at 2007-7-15 19:48:43 >
Hi,
I am considering to build a Java compatible In memory database to speed up the application of JAVA. We have now the C++ version of the database, named MTDB, which is hundreds of times faster than phisical database. The in memory database runs on solaris, linux and windows system. MTDB supports load sharing, data distribution, and realtime data synchronizing. MTDB shows large capacity, great performance, stability, and scalability in telecom applications. Add JAVA access interface to MTDB is very valuable. Please send me email if you have interest in this.
Thanks.
