Architecting multi-threaded applications in Java
Please let me apologize up front for the length of this post. I feel that if someone is willing to take their time to respond (and I hope someone will) I need to respect their time by providing a reasonably complete posting. If this is too much for you or you are time constrained, please feel free to ignore me and this post; and don't worry, I'm married and I have teenagers, so I'm used to being ignored on a regular basis!
Also, this looked like the correct Forum "OO Design" but if it is not, please let me know and I will attempt to move it to the correct forum ASAP.
I am tasked with re-architecting an existing Java (batch) application. I am interested in hearing what others think would be the best architecture for this application, given the basic requirements listed below.
This application accesses 100 different databases simultaneously, some of databases are LAN accessible but the majority are accessed over WAN. This application does not currently use (explicit) Java Threads. The current architecture starts a separate JVM for each of the 100 different databases (all 100 run concurrently). These Java programs take full advantage of the JDBC API including but not limited to RowSet and CLOB processing. The current processes are invoked via multiple command lines (through a script actually), and look something like this:
java Launch program'type1' coll'DB1' trace'NO' ifile ?repository.ini?java Launch program'type2' coll'DB1' trace'NO' ifile ?repository.ini?java Launch program'type1' coll'DB2' trace'NO' ifile ?repository.ini?java Launch program'type2' coll'DB2' trace'NO' ifile ?repository.ini?java Launch program'type1' coll'DB3' trace'NO' ifile ?repository.ini?java Launch program'type2' coll'DB3' trace'NO' ifile ?repository.ini?
Using this architecture we are starting 100 concurrent JVM抯 on our server, each JVM taking about 40 megabytes of memory for a total of 4 gigabytes of system memory.
In the example provided (above):
"type1" averages about 10 seconds of execution time, maximum 30 seconds, and runs every 05 minutes
"type2" averages about 03 minutes of execution time, maximum 14 minutes, and runs every 60 minutes
The type1 program issues several hundred database calls
The type2 program issues several thousand database calls.
When these programs (type1 and type2) start, they use 70% of the CPU on the server (an E15000 quad processor), then drops to about 20% after 10 seconds or so. The initial heavy CPU usage is attributed to "Java and program" startup, initial calls, etc. Most CPU usage and I/O after the first 10 seconds can be attributed to the work Oracle is doing in response to the JDBC calls from the programs.
Last year realizing a deficiency in the current 1 for 1 architecture, we designed a new program type, we call "type3", using explicit Java Threads. Type3 is actually a different process then type1 and type2 in that it was not timing sensitive (see below). Type3 was designed and developed using multiple thread queues that are controlled by parameters in an .ini file. The parameters are used to control thread concurrency, limiting the total count of concurrent threads and allowing prioritization of specific types of threads. In addition, we implemented several consumer/producer classes / algorithms to enable robust message passing between the Threads (logging, sending email, program metadata ...).
We have been very pleased with the overall performance of the "type3" program that uses Threads, but have found that we must limit the concurrency of Threads or we run the risk of degrading performance apparently due to heavy context switching and/or the blocking (waiting) of Threads. In testing of the type3 program, we found that exceeding 16 and 32 total threads in a single JVM resulted in significant performance degradation. The blocking of Threads we saw was attributed specifically to the Oracle JDBC driver.
I would like to implement a similar architecture used by the "type3" program for both the "type1" and "type2" programs. There are several issues that I must address to successfully re-architect this application.
1) The application process for type1 and type 2 is timing sensitive. All (100-300) Threads must start concurrently with initial processing completed within 1 minute of startup. Initial startup includes connecting to two separate databases (1 unique, 1 common) and issuing and completing queries on both (not the program logic, just the ability to start processing the ResultSet(s)) within 1 minute of program startup. If slowdowns cause this initial logic to extend past 1 minute the data is considered corrupt and the information is lost forever.
2) The count of concurrent tasks is expected to rise from 100 to 300 total tasks (equates to 150 type1 threads, 150 type 2 threads.).
THE PROBLEM AS I SEE IT (YOU MAY SEE OTHER PROBLEMS...)
I don't believe that a Java program using JDBC will scale to allow 300 total Threads to run concurrently and efficiently. As mentioned, once we moved above 16 and then 32 threads in testing we experienced performance degradation. Based on our observations of the type3 program (now in production) I have serious questions about the ability of the Oracle JDBC Driver to scale in this environment.
Is there any reason to believe that a single Java/JDBC program architecture spawning 300 Java Threads will scale as needed / per our requirements?
What are the alternatives for a robust Java architecture that will be flexible enough to support 300 time sensitive concurrent tasks?
The Type1 and Type2 programs process a lot of data, but hold very little data at any one time. Still, what are the chances that scalability related to internal JVM memory usage will become an issue when executing XX concurrent Threads in a single JVM?
One of the thoughts I've been playing with is to start a process from the command line that is only used to spawn discrete JVM抯, each of them working on X number of type1 processes and X number of type2 processes (i.e. 10 external (spawned) processes each doing 15 type1's, 10 external (spawned) processes each doing 15 type2's), with the counts being controlled by external parameters. This would reduce the total concurrent JVM's from an impractical 300 (using 12gig system memory) down to a more practical 30 (uses 1.2gig system memory) or perhaps even less (almost a 75% reduction in system memory while increasing programs/threads by 200%). This does not address the concern of running out of memory within the JVM itself because I don抰 know how much of a problem that may be. This solution is complex in that there is much code that must be written that is not related to the actual business logic, it is only written to support the architecture. Because of the architectural overhead in implementing this solution, it is definitely not my first choice, but I抦 struggling to come up with something else, which is of course, why I抦 asking for help!
This is not a trivial implementation. I'm expecting implementation of this new architecture along with several other enhancements to take 6+ months. If the architecture I implement doesn't scale, I'm in big, big trouble (like "the paths crossing" in 揋hostbusters?kind of trouble). As a result, I'm asking (maybe I'm pleading, I'm not sure) for any suggestions on how to go about making this right the first time.
The first true test of scalability will be the first time the application runs in production.
Boy, if I had a dime for every time I抳e heard that?br>
We do not have the ability to run a full system test using full data volumes. The best we can do is to implement something that theoretically will scale to our needs.
Anything you want to say, I'm happy to hear. I cannot change the requirements, but I am open to hearing anything that will help guide me towards creating a truly scalable application architecture.
Thank you!
WorkForFood, it's an impressive description of a difficult problem. Having seen so many of your fine posts in the past I'm not sure that I'm going to help you much here, but I'll offer the few thoughts that I have for your consideration.
I'll assume as a given that you've done all that's possible to optimize the SQL and the tables, indexes are in place as needed, network roundtrips are minimized as much as possible, and memory usage in the app has been scrubbed as much as possible. You've done all due diligence with profiling your type1 and type2 to understand where all those resources are going. I'll assume that you're taking full advantage of connection and thread pools, etc.
Since app servers are designed and optimized to handle this sort of thing, I'm wondering if type 1 and type 2 as stateless session EJB would be suitable. It sounds like you're making a big effort to duplicate the kinds of things that app servers are supposed to handle for you. It can have the connection pools ready to go and be waiting for requests from type 1 and type 2 clients as they come in. It'll manage all the EJB lifecycle issues and pooling for you. Throw in an F5 for load balancing and maybe you can manage it with clustered app servers.
None of this addresses your concern about the Oracle driver and its scalability, though. If that truly is the bottleneck, none of what I said will help the situation.
I've got a Web app running in production now that load tested up to 3300 simultaneous users without crashing or stalling. The WebLogic instances were using out of the box configuration, without much tuning. (A sad story.) We connected to Oracle 9i using their thin driver. The connection pool had something like 10 connections in the pool.
Admittedly, the simulated load was a mix of read and write operations, and none of them were of the duration that you're talking about. My point is that the connection pool is able to support more users than we would if they were all responsible for acquiring their own resources.
I'd strongly urge that you profile the situation with an OptimizeIt to determine where the resources are being consumed. You're an experienced developer, but data can never hurt.
Great problem. I wish I could help more.
%
> I'll assume as a given that you've done all that's
> possible to optimize the SQL and the tables, indexes
> are in place as needed, network roundtrips are
> minimized as much as possible, and memory usage in
> the app has been scrubbed as much as possible.
> You've done all due diligence with profiling your
> r type1 and type2 to understand where all those
> resources are going. I'll assume that you're taking
> full advantage of connection and thread pools, etc.
I am confident that the SQL is optimized and that the Oracle schema and configuration have been tuned to provide the very best performance possible from Oracle and JDBC. We have profiled the application using several different tools including your recommended OptimizeIt and found that a great percentage (>80%) of the wall clock and CPU time are spent in the JDBC calls with most of the time spent in row retrieval from the ResultSet and while not limited to CLOB processing, CLOB processing is a factor. Most of the 100 different databases are on WAN where network latency is measurable so great care has been taken to minimize network round trips, ensure only required data is fetched, use RowSet to cache small lookup tables that are repeatedly read and 揻etchsize?is tuned optimally for the environment. A special effort was made to 揷lean up?the code to specifically minimize overall memory consumption and optimize all String processing. A recent move to Oracle10 showed demonstrable performance improvement in all JDBC activity and specifically where CLOB processing occurred.
> Since app servers are designed and optimized to
> handle this sort of thing, I'm wondering if type 1
> and type 2 as stateless session EJB would be
> suitable. It sounds like you're making a big effort
> to duplicate the kinds of things that app servers are
> supposed to handle for you. It can have the
> connection pools ready to go and be waiting for
> requests from type 1 and type 2 clients as they come
> in. It'll manage all the EJB lifecycle issues and
> pooling for you. Throw in an F5 for load balancing
> and maybe you can manage it with clustered app
> servers.
I抳e thought about application servers but felt that the model that they work under doesn抰 truly fit the unique requirements of this application. Right or wrong, this is my basic understanding of the application server model today:
An application server allows best case sharing of limited resources including network & database connections, memory and CPU. The application server can also be used to artificially constrain resource usage by an application and allow multiple applications to share a server without significant resource conflicts. Scalability of the application servers is enhanced through clustering of loosely coupled application server instances. Using an application server within your architecture minimizes the time and money spent on developing your own (similar) proprietary architecture. The ROI of using an application server consistently within your architecture is multiplied by the number of applications that you write to that standard.
The 搇imited?resources that I抦 currently struggling with are not specifically database connections; at least they are not my major issue (yet). In my usage scenario I want an architecture that will take advantage of all available resources (i.e. database connections, CPU and I/O) and *not* artificially constrain them. My primary goal is best performance, but my architectural concerns are more about total system memory usage, internal JVM memory constraints and of course the scalability of the Java Threading model and the Oracle JDBC driver running within that model.
I have two very different profiles for transactions that I抦 running, described earlier as type1 and type2. From a transaction perspective, I would consider type1 a medium duration transaction (10+ seconds) and type2 a long duration transaction (3+ minutes). What they have in common is that they both must complete some initial processing within 1 minute of startup. An architecture that scales adequately for type1 may not scale adequately for type2, but I feel that it is imperative that they share an architecture and code base, so for now I will focus on type2.
The type2 transaction executes once an hour connecting concurrently150(x2) times to one database and150 times to 150 different databases. If the application server was robust enough to allow creation of a connection pool that would allow sharing of connections to multiple databases there may be some benefit in the connection profile of the 150 unique connections. I don抰 know if that functionality is available but I have a feeling that it is not, that is, I would think a single connection pool would be limited to a single database (but I could be wrong).
I also have reservations on the 150 local connections because the type2 transaction opens 150 (x2, see digress below) connections with each connection taking a minimum of 4mb on the Oracle side. Leaving these connections open and active when not in use (57 minutes of every hour) wouldn抰 be well tolerated by the other applications sharing this server, and would be difficult to explain to others from a technical perspective.
In this same vein, Oracle has a fairly robust transaction manager that could also be deployed without adding any significant complexity to the overall application architecture (it can be enabled with a simple configuration change in Oracle). It is in effect very similar to using application server connection pooling. Oracle will limit the total number of active connections to a user defined maximum, will keep the connections active for a set period of time and will queue up requests until a connection is available. I have considered using the Oracle transaction manager on the common (local) database if and when I start experiencing failures caused by attempting too many active connections to Oracle.
<DIGRESS>
The application actually must open 150x2 connections to local (common) database. The 2nd connection is opened and inserts a row with a known key into a table (i.e. type2, DB3) effectively locking the application and preventing someone from accidentally running the same application against the same database concurrently. When the application completes it rolls back the insert, unlocking the application. All application work is done through the 1st connection.
</DIGRESS>
> None of this addresses your concern about the Oracle
> driver and its scalability, though. If that truly is
> the bottleneck, none of what I said will help the
> situation.
Yes, while I expanded on that (above), you are correct and we are on the same page on this. However your thoughts on taking advantage of the application server, connection pooling and related concepts are definitely worthy of exploration.
> I've got a Web app running in production now that
> load tested up to 3300 simultaneous users without
> crashing or stalling. The WebLogic instances were
> using out of the box configuration, without much
> tuning. (A sad story.) We connected to Oracle 9i
> using their thin driver. The connection pool had
> something like 10 connections in the pool.
Interesting; I admit I抦 no expert in application servers, and since you brought it up, I will do research on WebLogic and it抯 applicability to my particular situation. Things change over time and it抯 possible that these products have matured beyond my current perceptions of them.
> Admittedly, the simulated load was a mix of read and
> write operations, and none of them were of the
> duration that you're talking about. My point is that
> the connection pool is able to support more users
> than we would if they were all responsible for
> acquiring their own resources.
I understand what you are saying to be true. I抳e always thought of an application server as essential for enabling scalability of multi-user environments that require real-time response. To be honest, I have not strongly considered use of an application server for enabling batch programs which as you know have quite a different performance profile.
<REMINISCE>
At some point in a past life I was considered a performance expert in IMS on the IBM mainframe. Around that time, IBM introduced incredible technology that allowed you to execute batch style programs called Batch Message Processing (BMP) easily within the IMS/DC & CICS (application server) environment. It was hailed as the beginning of a new era providing batch programs with highly tuned buffers, transaction rollback, automatic restart and more! It did usher in a new era, but it was an era of incredibly poorly performing applications and booming hardware sales (probably what IBM had in mind all along). Running these batch programs alongside the online transactions with their uniquely different performance profiles could absolutely devastate online response times which were commonly measured in 10th(s) of a second. Soon businesses were running out to purchase millions of dollars of hardware upgrades and eventually ended up purchasing additional multi-million dollar mainframes in order to successfully prioritize performance between the real-time transactions and batch transactions. Developers and management were hooked by the additional functionality and it was a step forward, but at an enormous dollar cost in hardware and lot of blood, sweat and tears by the technicians. I know we are not talking about the same thing here, but having lived through that experience, talking about enabling batch processing within an application server does give me pause..
</REMINSCE>
I have definitely considered connection pooling sans application server within this batch environment, and I implemented a poor mans approach to that in the multi-threaded type3 application described earlier. This poor man抯 connection pooling is quite different from a traditional connection pool in that it will continue to open a new connection each time a request comes in until Oracle responds that no more connections are available. At that point Exception processing code will wait a user defined number of seconds then retry and this will cycle like that until a connection becomes available. It抯 a brute force method, and I抦 certain that someone will have issues with it, but it has worked very well within our environment.
> I'd strongly urge that you profile the situation with
> an OptimizeIt to determine where the resources are
> being consumed. You're an experienced developer, but
> data can never hurt.
We have profiled the application and I抦 reasonable certain that the Java code is performing optimally under the constraints of the current architecture.
> Great problem. I wish I could help more.
I sincerely appreciate your input on what for me is a difficult and truly pivotal architectural decision. Thank you very much!
> I am confident that the SQL is optimized and that the
> Oracle schema and configuration have been tuned to
> provide the very best performance possible from
> Oracle and JDBC. We have profiled the application
> using several different tools including your
> recommended OptimizeIt and found that a great
> percentage (>80%) of the wall clock and CPU time are
> spent in the JDBC calls with most of the time spent
> in row retrieval from the ResultSet and while not
> limited to CLOB processing, CLOB processing is a
> factor. Most of the 100 different databases are on
> WAN where network latency is measurable so great care
> has been taken to minimize network round trips,
> ensure only required data is fetched, use RowSet to
> cache small lookup tables that are repeatedly read
> and 揻etchsize?is tuned optimally for the
> environment. A special effort was made to 揷lean up?br>> the code to specifically minimize overall memory
> consumption and optimize all String processing. A
> recent move to Oracle10 showed demonstrable
> performance improvement in all JDBC activity and
> specifically where CLOB processing occurred.
Very good. This does not surprise me.The quality of your past posts is so high that any other response was unlikely. I wrote what I did to make it clear that I need not worry about that stuff with you. It seems that I was correct. It might be the ONLY thing I have correct in this post. 8)
> I抳e thought about application servers but felt that
> the model that they work under doesn抰 truly fit the
> unique requirements of this application. Right or
> wrong, this is my basic understanding of the
> application server model today:
OK, I'll buy that.
> An application server allows best case sharing of
> limited resources including network & database
> connections, memory and CPU. The application server
> can also be used to artificially constrain resource
> usage by an application and allow multiple
> applications to share a server without significant
> resource conflicts. Scalability of the application
> servers is enhanced through clustering of loosely
> coupled application server instances. Using an
> application server within your architecture minimizes
> the time and money spent on developing your own
> (similar) proprietary architecture. The ROI of using
> an application server consistently within your
> architecture is multiplied by the number of
> applications that you write to that standard.
>
> The 搇imited?resources that I抦 currently struggling
> with are not specifically database connections; at
> least they are not my major issue (yet). In my usage
> scenario I want an architecture that will take
> advantage of all available resources (i.e. database
> connections, CPU and I/O) and *not* artificially
> constrain them. My primary goal is best performance,
> but my architectural concerns are more about total
> system memory usage, internal JVM memory constraints
> and of course the scalability of the Java Threading
> model and the Oracle JDBC driver running within that
> model.
OK. (Note to self: You're not helping here.)
> I have two very different profiles for transactions
> that I抦 running, described earlier as type1 and
> type2. From a transaction perspective, I would
> consider type1 a medium duration transaction (10+
> seconds) and type2 a long duration transaction (3+
> minutes). What they have in common is that they both
> must complete some initial processing within 1 minute
> of startup. An architecture that scales adequately
> for type1 may not scale adequately for type2, but I
> feel that it is imperative that they share an
> architecture and code base, so for now I will focus
> on type2.
>
> The type2 transaction executes once an hour
> connecting concurrently150(x2) times to one database
> and150 times to 150 different databases. If the
> application server was robust enough to allow
> creation of a connection pool that would allow
> sharing of connections to multiple databases there
> may be some benefit in the connection profile of the
> 150 unique connections. I don抰 know if that
> functionality is available but I have a feeling that
> it is not, that is, I would think a single connection
> pool would be limited to a single database (but I
> could be wrong).
Correct - one DB per pool. You'd need 150 pools.
But what's the alternative? To ignore the pool and let each type1 and type2 thread create its own connection? At least the pools can have them warm and waiting. I've always read that the startup time for acquiring a connection was long. A pool spreads it out over all apps.
The argument against this is that even if it takes several seconds to get that connection, your runtime is on the order of minutes so it's a wash. Right? So strike the pool idea.
> I also have reservations on the 150 local connections
> because the type2 transaction opens 150 (x2, see
> digress below) connections with each connection
> taking a minimum of 4mb on the Oracle side. Leaving
> these connections open and active when not in use (57
> minutes of every hour) wouldn抰 be well tolerated by
> the other applications sharing this server, and would
> be difficult to explain to others from a technical
> perspective.
Agreed.
> In this same vein, Oracle has a fairly robust
> transaction manager that could also be deployed
> without adding any significant complexity to the
> overall application architecture (it can be enabled
> with a simple configuration change in Oracle).
You mean the XA driver now?
> It is
> in effect very similar to using application server
> connection pooling. Oracle will limit the total
> number of active connections to a user defined
> maximum, will keep the connections active for a set
> period of time and will queue up requests until a
> connection is available. I have considered using the
> Oracle transaction manager on the common (local)
> database if and when I start experiencing failures
> caused by attempting too many active connections to
> Oracle.
>
> <DIGRESS>
> The application actually must open 150x2 connections
> to local (common) database. The 2nd connection is
> opened and inserts a row with a known key into a
> table (i.e. type2, DB3) effectively locking the
> application and preventing someone from accidentally
> running the same application against the same
> database concurrently. When the application
> completes it rolls back the insert, unlocking the
> application. All application work is done through
> the 1st connection.
> </DIGRESS>
> Yes, while I expanded on that (above), you are
> correct and we are on the same page on this. However
> your thoughts on taking advantage of the application
> server, connection pooling and related concepts are
> definitely worthy of exploration.
>
> Interesting; I admit I抦 no expert in application
> servers, and since you brought it up, I will do
> research on WebLogic and it抯 applicability to my
> particular situation. Things change over time and
> it抯 possible that these products have matured beyond
> my current perceptions of them.
Worth checking out. If WebLogic is too rich for your blood, Oracle has a Java EE app server built in. (It used to be Orion, if I remember right.) You might have access to it already, depending on your license.
> I understand what you are saying to be true. I抳e
> always thought of an application server as essential
> for enabling scalability of multi-user environments
> that require real-time response. To be honest, I
> have not strongly considered use of an application
> server for enabling batch programs which as you know
> have quite a different performance profile.
>
> <REMINISCE>
> At some point in a past life I was considered a
> performance expert in IMS on the IBM mainframe.
> Around that time, IBM introduced incredible
> e technology that allowed you to execute batch style
> programs called Batch Message Processing (BMP) easily
> within the IMS/DC & CICS (application server)
> environment. It was hailed as the beginning of a new
> era providing batch programs with highly tuned
> buffers, transaction rollback, automatic restart and
> more! It did usher in a new era, but it was an era
> of incredibly poorly performing applications and
> booming hardware sales (probably what IBM had in mind
> all along). Running these batch programs alongside
> the online transactions with their uniquely different
> performance profiles could absolutely devastate
> online response times which were commonly measured in
> 10th(s) of a second. Soon businesses were running
> out to purchase millions of dollars of hardware
> upgrades and eventually ended up purchasing
> additional multi-million dollar mainframes in order
> to successfully prioritize performance between the
> real-time transactions and batch transactions.
> Developers and management were hooked by the
> e additional functionality and it was a step forward,
> but at an enormous dollar cost in hardware and lot of
> blood, sweat and tears by the technicians. I know we
> are not talking about the same thing here, but having
> lived through that experience, talking about enabling
> batch processing within an application server does
> give me pause..
> </REMINSCE>
Great story. That's the problem with guys like me who are late to this party. We don't have the history that you do. I'll be you have deja vu moments all day long: "Aren't web services just CORBA all over again?"8)
>
> I have definitely considered connection pooling sans
> application server within this batch environment, and
> I implemented a poor mans approach to that in the
> multi-threaded type3 application described earlier.
> This poor man抯 connection pooling is quite
> e different from a traditional connection pool in
> that it will continue to open a new connection each
> time a request comes in until Oracle responds that no
> more connections are available. At that point
> Exception processing code will wait a user defined
> number of seconds then retry and this will cycle like
> that until a connection becomes available. It抯 a
> brute force method, and I抦 certain that someone will
> have issues with it, but it has worked very well
> within our environment.
>
> We have profiled the application and I抦 reasonable
> certain that the Java code is performing optimally
> under the constraints of the current architecture.
>
> > Great problem. I wish I could help more.
>
> I sincerely appreciate your input on what for me is a
> difficult and truly pivotal architectural decision.
> Thank you very much!
I'm sorry that I'm not more helpful. I'm certainly interested.
I can't help but wonder how places like Google and eBay and Yahoo! are handling high-transaction, high-volume situations. They probably have fewer hardware constraints than you do. My understanding of Google is that they run on a cluster of commodity Linux blade servers that they treat like light bulbs: when one goes south, they remove it from the cluster and plug in another one.
But their transactions sound very different from yours. Your problem sounds database-bound. Theirs are more request/response, read-mostly HTTP requests. Not like your batch processes.
The thing that would keep me up at night would be the possibility of data corruption. The timing has to be just right. I'd worry about that. I'll bet you do, too.
The problem with batch scheduling like this is that any process that's late or runs overly wrong screws up the whole thing. An event-driven process would be safer, because that would keep everyone waiting for latecomers.
What about taking advantage of the build-in event listening features built into Java Beans to avoid this? Maybe a combination of a Java scheduler like Quartz and an event model would help.
%
> 1) The application process for type1 and type 2 is
> timing sensitive. All (100-300) Threads must start
> concurrently with initial processing completed within
> 1 minute of startup. Initial startup includes
> connecting to two separate databases (1 unique, 1
> common) and issuing and completing queries on both
> (not the program logic, just the ability to start
> processing the ResultSet(s)) within 1 minute of
> program startup. If slowdowns cause this initial
> logic to extend past 1 minute the data is considered
> corrupt and the information is lost forever.
>
I am rather curious about this requirement and as it relates to the overall architecture particularily given that it appears that all of the databases are Oracle and are routinely updated.
Why doesn't each server just dump the relevent data?
For example in the above case a scheduled process runs on the database server itself and does a timestamp dump of the data (either to storage tables or files.)
And the other databases could do the same given that there is some time sensitive material there.
The dumped data, not the original data, is moved. Files in this case, rather than database access would probably be ideal. I wouldn't be surprised with WANs connectivity that moving files and parsing would probably be faster than JDBC calls (particularily if a ftp apps were used rather than using a java api and tuning was done.)
This should make it much harder to loose data as well.
So what prevents the system from doing this?
> I am rather curious about this requirement and as it
> relates to the overall architecture particularily
> given that it appears that all of the databases are
> Oracle and are routinely updated.
>
> Why doesn't each server just dump the relevent data?
>
> So what prevents the system from doing this?
Good questions, and obviously technically it is possible to do what you are saying, but to be successful using an integrated suite of discrete technologies (as you are suggesting) I think there needs to be a controlled environment, and that is not the case for this application. I agree that there are runtime efficiencies to be had taking your suggested route, but to be straightforward with this, the tight integration of discrete technologies required to make that architecture work is very difficult to roll out and maintain on a grand scale, especially when this application is installed in a variety of different businesses using all flavors and versions of UNIX, Linux and Windows along with accessing different versions of Oracle (7 to 10g).It comes down to the oft asked question, is it better to push the data to a centralized server from the source or have the centralized server pull the data from the source. I抳e found that the answer to that question is always; 搃t depends? In this instance, the decision, for what I believe are very good reasons, was to pull the data from the source. I cannot see the application mandate changing, and so it frames the architecture that must be deployed:
A mandate of this application is that no program agents be required on the WAN database server. In addition this application requires that no object creation be required in the WAN databases.
You may notice the words 揵e required?is used in a couple of places, and the fact is this application抯 performance can be enhanced if you choose to run components (Java programs) on the application database server where the Java programs will then act as agents. Doing so minimizes network traffic significantly (close to 75%) and while it is still possible, you are much less unlikely to 憁iss?the data window. Still, not one company has chosen to roll it out that way (as agents), the choice is always to roll it out using 憂o agent?technology, which minimizes our concerns about keeping agents current but I believe it抯 done that way because it is much easier to initially install and maintain operationally.
There are very good reasons for both mandates and I won't go into too much detail here except to say that we believe that agent technology in general has demonstrated itself to be severely flawed. Specifically that over time the changes in the underlying environments including the O/S version and patches, Database version and patches and other supporting utilities corrupt the agents making them unreliable and making it increasingly difficult to patch and make agents immediately available for all environments. The 搉o agent?architecture is the #1 reason why this application gets chosen for deployment. In addition, the 搉o agent?architecture allows for very quick installation within a large scale environment and has the added bonus of allowing centralized (easy) administration of all related processes.
It is interesting that your suggestions very closely mimicked our original beta architecture for this application. The beta architecture was surprisingly quick to deploy and was very successfully in our first installation, which only looked at a couple of databases. This was our proof of concept, not for the architecture but for the application itself. The architecture had the benefit of being made up of existing discrete components and proved itself to be quite nimble and was the correct architecture for the tightly controlled environment that we were in at the time.
We enabled our current 搉o agent?architecture after about 8 months of running beta and dubbed it 1.0 (circa 1996-1997). The original 1.0 architecture was conceived with the assumption of processing against a maximum of 8-12 databases and 1000-3000 rows per DB. We further enhanced the architecture with an eye towards scalability which became our 2.0 release with the assumption of processing a maximum of 40 databases and 5000 rows.
The 2.0 architecture has scaled surprisingly well up to our current levels of 100 databases processing 12,000 rows per DB (apparently we are not very good at estimating volume metrics). However it is my belief that we have maxed out our 2.0 architecture, it has run its course, and something else is required, but we still need to stay true to some of the original design goals which were used to sell the concept behind the application. I believe that this overarching framework of 搉on agent?technology is still the very best choice for this particular application.
I like to think that once this application is re-architected I will still be as firmly entrenched in that same belief. :)
>
> This should make it much harder to loose data as well.
>
I had hoped that the timing issue would just be accepted, but I had gut feeling that I would end up clarifying the timing / corruption statement, so I抣l give it a try because if you asked about it, then you get the issue and further information can only help.
I realize that I stated something just a bit wrong, and hopefully I did not lead you astray, where I said:
揑f slowdowns cause this initial logic to extend past 1 minute the data is considered corrupt and the information is lost forever.?/i>
Should have read as:
揑f slowdowns cause this initial logic to extend past 1 minute the data is considered corrupt and that data is lost forever.?/i>
One goal of the architecture is to never miss the one minute processing window unless it is caused by an external event. That is, the architecture should never be the root cause of a failure in performance or function.
The timing requirements are as stated, but we couldn抰 in good conscious write the application knowing that a network glitch or extended database outage (on either end) could ruin the information collection process. So, we developed an application that can 憇kip?executions and not completely lose the information (but the detail level data is gone forever). Don抰 get me wrong, the detail data / information is very, very important but we have mitigated the possible (and probable) loss of detail by ensuring that we can always 慶atch up?when necessary.
What is the end result of this 揷atch up?process? Reports that roll up anomalous data provide more accurate reporting then queries that work with anomalous detail. Unless something serious and systemic happens, the highest level reports are close to 100% accurate even if we are dealing with anomalous data (we could miss our time critical window every other time we executed the program and reporting at higher levels would still be solid). What would be lacking when reporting with anomalous data would be the ability to drill down into that information and see an accurate recreation of what actually went on to create the summary. These data anomalies are smoothed over by the application interface that creates the reports, and the user rarely has to worry about it, but can view the actual anomalies if they so choose (but they never do).
Thanks for the opportunity to think about a different way of doing things. Your post gave me some things to think about and helped me clarify in my mind why we made some of the original primary architectural decisions, and some of the more recent ones. I hope I was able to respond in a way that helps you understand a little more about why I抦 looking for help on this. If you have any more thoughts or ideas, I am wide open to hear them and will do my best to respond in kind. I抳e been doing this long enough to know that no one is going to hand me a working architecture, but it is fantastic to be able to see things from different perspectives and to hear from people with very different experiences then mine respond with intelligent and thoughtful ideas. Many thanks for the post, it is appreciated.
I'm going to attempt a comment even though I haven't really read everything here (there's alot) so forgive me if I look a little stupid.
Is it right that your application runs on one server accessing many? or runs on many accessing many?
I guess I'm asking whether you can stick a "middle" server somewhere that they java apps talk to. This server could have an application written in a more thread-friendly and effecient language then java, and do processing from there? You could perhaps have this middle server do some load-balancing based on the requests it gets and send requests to sub-servers. I'm not really sure if this is at all appropriate for your situation, so sorry if it's a really dumb idea :)
> I'm going to attempt a comment even though I haven't
> really read everything here (there's alot)
>
Yeah, sorry about that?I抦 kind of the Stephen King of programmers, once I start writing about technical issues I never want to stop! Plus, I抦 working under a pseudo name just like he did; perhaps I should change WorkForFood to Richard Bachman to be more appropriate! :)
> Is it right that your application runs on one server
> accessing many? or runs on many accessing many?
>
Because the application is written in Java, the application can run on any platform that supports Java. We have run the application from a PC, a standalone server and on any of the database servers that is being accessed. That ability to run anywhere is the primary reason we chose Java, not just because it was really cool, which it is, but because we didn't have to worry about targeting a specific platform.
We also chose to use JDBC / SQL as an exclusive API which allows us to access data on any platform that supports Oracle. This is, in a sense a warehouse application where we collect information from 100抯 of servers and place it into a single server. Because we chose the JDBC / SQL API, that single server (the warehouse) is *not* limited to the Oracle platform, and we have in a pinch installed and used MS Access (this was a little touchy), MS SQL Server (much better) and Cloudscape (now known as Derby) when it was required.
Between the wonders of Java and the use of the platform and database neutral JDBC / SQL API we were able to truly realize the lofty and sometimes scoffed at goal of "write once run anywhere".
So, in your vernacular, it runs on one server and accesses many.
> I guess I'm asking whether you can stick a "middle"
> server somewhere that they java apps talk to. This
> server could have an application written in a more
> thread-friendly and effecient language then java, and
> do processing from there? You could perhaps have this
> middle server do some load-balancing based on the
> requests it gets and send requests to sub-servers.
> I'm not really sure if this is at all appropriate for
> your situation, so sorry if it's a really dumb idea :)
Nope, not dumb at all. I do have some significant reservations regarding the use of a middleware server for this particular (batch) application but concede that I am no expert and there may be more available in a middleware server then my initial perceptions would have me believe. Duffymo had some similar suggestions and I am looking into several that he mentioned as possible options.
Thanks for the post, and thank you for the suggestion.
> Very good. This does not surprise me.The quality
> of your past posts is so high that any other response
> was unlikely. I wrote what I did to make it clear
> that I need not worry about that stuff with you. It
> seems that I was correct. It might be the ONLY thing
> I have correct in this post. 8)
Nonsense, your spelling has been excellent! :)
> > In this same vein, Oracle has a fairly robust
> > transaction manager that could also be deployed
> > without adding any significant complexity to the
> > overall application architecture (it can be
> enabled
> > with a simple configuration change in Oracle).
>
> You mean the XA driver now?
No, I was describing Oracle抯 multithreaded shared servers (MTS); I hadn抰 really considered XA in my scenario.
> Worth checking out. If WebLogic is too rich for your
> blood, Oracle has a Java EE app server built in. (It
> used to be Orion, if I remember right.) You might
> have access to it already, depending on your
> license.
Thanks for the suggestions!
>"Aren't web services just CORBA all over again?"8)
God I hope not, but yes.
> I'm sorry that I'm not more helpful. I'm certainly interested.
To the contrary, you have been extremely helpful, and I really appreciate the interest and the insights.
> I can't help but wonder how places like Google and
> eBay and Yahoo! are handling high-transaction,
> high-volume situations. They probably have fewer
> hardware constraints than you do. My understanding
> of Google is that they run on a cluster of commodity
> Linux blade servers that they treat like light bulbs:
> when one goes south, they remove it from the cluster
> and plug in another one.
I think there are great lessons to be learned at Google and eBay. eBay had some real struggles early on that they were able to put behind them. However, they have a near perfect business model in that they carry no inventory; require no warehouses, no distributors and no sales channels. They are a virtual company in the purest sense and that allows them the luxury of putting an incredible percentage of their revenue dollars back into R&D. If they were going to fail, the finger would have pointed directly to availability and performance, and I think they came close to failing at one point, but they didn抰. I would love to hear the technical stories behind that success.
> But their transactions sound very different from
> yours. Your problem sounds database-bound. Theirs
> are more request/response, read-mostly HTTP requests.
> Not like your batch processes.
I agree, there transaction models are different, but there is nothing wrong with looking at success and seeing if there is something there for you.
> The thing that would keep me up at night would be the
> possibility of data corruption. The timing has to be
> just right. I'd worry about that. I'll bet you do,
> too.
>
> The problem with batch scheduling like this is that
> any process that's late or runs overly wrong screws
> up the whole thing. An event-driven process would be
> safer, because that would keep everyone waiting for
> latecomers.
>
> What about taking advantage of the build-in event
> listening features built into Java Beans to avoid
> this? Maybe a combination of a Java scheduler like
> Quartz and an event model would help.
I tried to re-address this timing issue (which I tried to simplify in my original post, but obviously failed) in a post back to jschell. Hopefully you will get a chance to read that post and let me know if it changes any of your thoughts.
Thanks again.
I haven抰 finished my research yet, and I would still be very interested in hearing anybody抯 thoughts on any part of this post, but I have one specific thought regarding one of the multi-threading issues. One of the concerns I mentioned was the issue of memory management and the fear of running out of memory within the JVM when creating 100抯 of threads within my application code.
One of the features available in Java 5 is called Java Management Extensions or JMX. I believe JMX can be used to monitor the current state of the JVM, providing a variety of getXXX methods to peer into the inner workings of the JVM. This sounds interesting to me, and let me put together a possible scenario and a couple of questions and see if anyone has any thoughts.
My original thought for controlling threads within a single JVM was passing a startup parameter to the program that would limit the total count of concurrent threads, and define ahead of time how many JVM抯 I would be running concurrently. It抯 an inelegant solution to this problem because the default number passed and the overall JVM count would always be arbitrary and tuning of that parameter would have to take place over time. Using JMX it may be possible (I haven抰 looked at it in detail) to hinge dynamic Thread creation based on having an adequate amount (a threshold) of available heap memory.
Because my application has a startup time constraint, I might consider taking this a step further and once I have created enough Threads to exceed the defined heap threshold (as viewed through JMX) I could 憇pawn?a new JVM through the O/S that can now create as many threads as possible in the 2nd JVM until the 2nd JVM heap threshold is exceeded (again through JMX) and it could 憇pawn?a new (3rd) JVM through the O/S, and so on and so on.
This solution is not complete in that it does not address the 慴locking?issue that I have seen with the Oracle driver, and it might be necessary to limit the thread counts to avoid blocking even if heap thresholds are never exceeded. However, I like to think that the Oracle driver will at some point correct the blocking issue, or I抣l figure out how to work around it. If either happens, then the heap mechanism will be in place and the application will actually get better at taking advantage of available resources over time.
This isn抰 my proposed architecture; I抦 just looking possibilities. At first glance, this sounds interesting but I抦 curious if anyone else has any thoughts on the concept. Has anyone ever done something like this? Any chance it would remain stable and not start popping up JVM抯 like popcorn in hot oil?
Here are a couple of related questions:
Are there any issues or concerns about one JVM 憇pawning?a 2nd JVM?
I have never 憇pawned?a 2nd JVM from a JVM. Would it be appropriate to 憇pawn?the script which in turn starts the 2nd JVM, or to 憇pawn?the 2nd JVM directly knowing that I must pass dynamic parameters during the 憇pawn?(For example, JVM 1 runs 5 threads against DB1-DB5 then spawns JVM 2 which needs to know to start with DB6)?
When spawning a script to start a new JVM, I am concerned under UNIX / LINUX that the spawned JVM will not inherit the same environment as the original JVM so CLASSPATH and other environment variables would be missing from the environment.
I don抰 know the history behind JMX, but many times debugging facilities are not highly optimized and when you attempt to use them as part of your normal program logic they can do more harm then good. Should JMX be used within a production application for this purpose?
I have never coded anything JMX, and will probably do some more reading, but if you are aware of anything I would appreciate you sharing it. Are there known unique or important considerations for using JMX in setup or usage?
Thank you.
I think the spawn issue has problems.
1. For a given application there is a limited number of threads due to memory and/or resources.
2. Everything runs on one box.
3. Physical memory is limited to one box.
4. There are time constraints.
Combining the above I think it is likely that the due to swap outs that you will loose if you try multiple VMs. That would be the case if memory itself is the problem. It might not be the case if there is another resource problem. However given the number of threads that you are suggesting it doesn't seem likely that anything but memory is the problem.
If you are not constrained by the above then why create a solution that is? That would involve one of the following.....
1. Use more than one box.
2. Use much more memory (on a box that supports it) and get a VM that supports that memory directly.
Some interesting thoughts, thank you.
> I think the spawn issue has problems.
> 1. For a given application there is a limited number
> of threads due to memory and/or resources.
> 2. Everything runs on one box.
> 3. Physical memory is limited to one box.
> 4. There are time constraints.
>
> Combining the above I think it is likely that the due
> to swap outs that you will loose if you try multiple
> VMs. That would be the case if memory itself is the
> problem. It might not be the case if there is
> another resource problem. However given the number
> of threads that you are suggesting it doesn't seem
> likely that anything but memory is the problem.
Right now, I am starting 100 separate JVM's simultaneously using an O/S scheduler. There is enough memory for that to happen on this server, which is currently configured with 16 gigabytes of memory, swapping rarely if ever occurs even though this application is sharing the server with a few other applications. I already consider the memory usage of this application excessive because I know that two of these processes could easily be done per JVM therefore I know that the application is using at least 50% too many resources to get this work done, and the cost of running the application is 50% more then it needs to be. The chances are good that I can do more then 2 per JVM, perhaps as many as 10 per JVM for the type2 process (3min), and perhaps as many as 100 in the type 1 (10sec) process which could reduce system resource usage (and specifically system memory usage) by more then 90%.
A new requirement has been given (which I mentioned earlier) to start 200 additional processes which in the current architecture will equate to 300 JVM's starting simultaneously. I am looking for economy of scale here, if I was only running 5 processes I wouldn抰 care, but at 100, 200 or 300 the inefficient use of system resources is a problem that should be corrected.
The whole point of the architectural change is to curtail the excessive resource usage of the current architecture, specifically system memory. If I can do the same processing against the same data volumes, but cut down the total count of JVM's from 100 to 10 (and the each JVM footprint doesn抰 expand 10 fold) then I have accomplished my goal of enhancing the architecture to make this application more efficient and as a result far more scalable.
Using Java Threads on a different but similar process I was able execute a single JVM, kicking off a maximum of 15 concurrent Threads, and the performance of the application was very good (but did not have the 1 minute window to worry about). If I had simply "added" these similar processes into the current architecture then I would have to start an additional 50 JVM's to do the same amount of work (in about the same amount of time). It is an order of magnitude more efficient in using system resources then the older architecture, and in practice is performing the same or better.
One of the assumptions that I am making here in setting the goals and discussing projected benefits is that the memory footprint of the JVM won抰 double when I run two processes (Threaded) within a single JVM. I don抰 expect the memory footprint to expand much at all even if I were to run 5 of these processes. That is the economy of scale that I抦 hoping for. In your comments I got the impression that you were not expecting that same economy of scale. Do I have that correct, and if I do can you give me a little more information on why economy of scale may not occur or not occur to the levels I am hoping for?
I need to re-architect to make this application more efficient (cheaper to run) and more scalable. That抯 why I抦 doing it.
> If you are not constrained by the above then why
> create a solution that is? That would involve one of
> the following.....
> 1. Use more than one box.
This application will probably always be limited to running from a single server, the capacity of the servers are always different, sometimes markedly so, but all current installations run these processes on a single server. Just so you know, there isn抰 any reason why this Java application抯 processes cannot be split out to 100 different servers running 100 different processes, but no one wants the administrative overhead of keeping that configuration maintained. I don抰 want to lose the flexibility of being able run processes from any server and adding in Threading or the ability to spawn additional JVM抯 doesn抰 have to take that capability away.
> 2. Use much more memory (on a box that supports it)
> and get a VM that supports that memory directly.
Are you implying that I can get another vendors Java VM or that the SUN JVM will scale to the capacity available on today抯 larger servers? For example, the server I抦 currently running on has 4 dedicated processors and 16 gigabytes of system memory. Is there a VM that will take advantage of all 4 processors when using threads (I have heard rumblings that there is a practical ratio of Java threads to available CPU抯) while allowing those threads access to 16 gig of memory? I抦 not being facetious here, I am really wondering, because if that is the case, and short of any other significant limitations, then the overhead of designing and developing spawning architecture is overkill (as I think you may have been implying).
It seems like your running all these seperate VM's because Oracle's driver blocks in a non-optimal way?I might look into classloading schemes that allow me to load several drivers in isolation, potentially bypassing any locks in the driver.
> It seems like your running all these seperate VM's
> because Oracle's driver blocks in a non-optimal way?
>
> I might look into classloading schemes that allow me
> to load several drivers in isolation, potentially
> bypassing any locks in the driver.
Yes, one of the reasons is due to the Oracle driver blocking in a non-optimal way.
Your idea sounds like it has merit; I certainly have never considered the possibility. I have done basic reading on the Java class loader, but never thought I would have a need to override or extend it. Please allow me to paraphrase to make sure I understand your suggestion, and please pardon me as I struggle with the terminology here:
Within a single JVM, using a proprietary class loader code, I could load multiple instances of the Oracle driver, each instance residing in its own address space (within the JVM) sharing nothing with the other Oracle driver instances? The reason they do not share is that they are in effect instances of different classes?
If I have that close to correct, this appears to require advanced programming techniques and a unique knowledge of the underpinnings?of Java. Since I read your post I've browsed a couple of articles on class loaders and to be honest, at first glance it scares me to think about writing code that works so intimately with the core of Java.
Are there any concerns (besides mine) about negatively affecting portability, performance or most importantly the stability of an application that implements one or more proprietary class loaders (using the existing API's mentioned in articles)?
Are there configuration issues with deploying code that includes proprietary class loaders?
Thanks for the suggestion!
> >
> > I might look into classloading schemes that allow me
> > to load several drivers in isolation, potentially
> > bypassing any locks in the driver.
>
> Yes, one of the reasons is due to the Oracle driver
> blocking in a non-optimal way.
I am rather certain that won't work with the OCI driver.
it would with the thin driver.
> Within a single JVM, using a proprietary class
> loader code, I could load multiple instances of the
> Oracle driver, each instance residing in its own
> address space (within the JVM) sharing nothing with
> the other Oracle driver instances? The reason they
> do not share is that they are in effect instances of
> different classes?
Looks about right, someone could probably quibble with the term "address space"... "name space" might be a little closer.
> Are there configuration issues with deploying code
> that includes proprietary class loaders?
You don't really need to implement your own class loader, java.net.URLClassloader will work just fine, all you need is to create several of them.
This method may then be of some use when you load your JDBC driver:
http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Class.html#forName(java.lang.String,%20boolean,%20java.lang.ClassLoader)
The only issue I can see is making sure that each classloader loads its own private java.sql.DriverManager class.
(Of course, this is all theory I haven't tried it myself...)
> Some interesting thoughts, thank you.
>
> > I think the spawn issue has problems.
> > 1. For a given application there is a limited
> number
> > of threads due to memory and/or resources.
> > 2. Everything runs on one box.
> > 3. Physical memory is limited to one box.
> > 4. There are time constraints.
> >
> > Combining the above I think it is likely that the
> due
> > to swap outs that you will loose if you try
> multiple
> > VMs. That would be the case if memory itself is the
> > problem. It might not be the case if there is
> > another resource problem. However given the number
> > of threads that you are suggesting it doesn't seem
> > likely that anything but memory is the problem.
>
> Right now, I am starting 100 separate JVM's
> simultaneously using an O/S scheduler. There is
> enough memory for that to happen on this server,
> which is currently configured with 16 gigabytes of
> memory, swapping rarely if ever occurs even though
> this application is sharing the server with a few
> other applications. I already consider the memory
> usage of this application excessive because I know
> that two of these processes could easily be done per
> JVM therefore I know that the application is using at
> least 50% too many resources to get this work done,
> and the cost of running the application is 50% more
> then it needs to be. The chances are good that I can
> do more then 2 per JVM, perhaps as many as 10 per JVM
> for the type2 process (3min), and perhaps as many as
> 100 in the type 1 (10sec) process which could reduce
> system resource usage (and specifically system memory
> usage) by more then 90%.
There is a resource problem of some sort.
Running two per JVM is either going to use more, less or the same as running in two separate VMs.
Given your server I doubt that the VM overhead itself is adding to the resource problem so that suggests that running 2 per VM is not going to reduce the load due to the VM (to a measurable degree.)
So if memory was a problem (note this is an if) then running 2 in the same VM is not going to reduce the memory usage. Both run independently and both still need to use the same amount of memory as before.
Both still run in a thread (or threads) and those are still used.
Both still use sockets and those are still used.
So those are all resources whose usage will not be reduced by running two per VM.
Given that you have demonstrated (via experimentation) that some resource is not duplicated when in one VM I would like to know what resource you are conserving or which one you think you are conserving?
Basically right now it sounds like the only resource at all that is in contention is the Oracle driver. And if that is the case the vast majority of this thread is irrelevant. And the solution for that is as follows.....
1. Determine either statically or dynamically the threshold for the Oracle driver.
2. Create a main application that spins processes and tasks (threaded) based on the threshold limits. It spins up enough processes to cover everything you need.
I suspect for 1 above that you would have to build a history table and allow the application to experimentally determine the best size over time. Rather complicated but doable.
Alternatively if you are using the thin driver and locking is the issue then use the class loader. There is at least one thread in the JDBC forum that discusses the correct way to do this. It isn't as easy as loading your own class.
>
> Are you implying that I can get another vendors Java
> VM or that the SUN JVM will scale to the capacity
> available on todays larger servers? For example,
> the server Im currently running on has 4 dedicated
> processors and 16 gigabytes of system memory. Is
> there a VM that will take advantage of all 4
> processors when using threads (I have heard rumblings
> that there is a practical ratio of Java threads to
> available CPUs) while allowing those threads access
> to 16 gig of memory? Im not being facetious here, I
> am really wondering, because if that is the case, and
> short of any other significant limitations, then the
> overhead of designing and developing spawning
> architecture is overkill (as I think you may have
> been implying).
There are 64 bit VMs that will use somewhere in the 10-16 gig range in the VM itself. Could be higher but those are the reports I have seen here.
> There is a resource problem of some sort.
>
> Running two per JVM is either going to use more, less
> or the same as running in two separate VMs.
>
> Given your server I doubt that the VM overhead itself
> is adding to the resource problem so that suggests
> that running 2 per VM is not going to reduce the load
> due to the VM (to a measurable degree.)
>
> So if memory was a problem (note this is an if) then
> running 2 in the same VM is not going to reduce the
> memory usage. Both run independently and both still
> need to use the same amount of memory as before.
>
> Both still run in a thread (or threads) and those are
> still used.
>
> Both still use sockets and those are still used.
>
> So those are all resources whose usage will not be
> reduced by running two per VM.
>
> Given that you have demonstrated (via
> experimentation) that some resource is not duplicated
> when in one VM I would like to know what resource you
> are conserving or which one you think you are
> conserving?
>
> Basically right now it sounds like the only resource
> at all that is in contention is the Oracle driver.
> And if that is the case the vast majority of this
> s thread is irrelevant. And the solution for that is
> as follows.....
>
> 1. Determine either statically or dynamically the
> threshold for the Oracle driver.
> 2. Create a main application that spins processes and
> tasks (threaded) based on the threshold limits. It
> spins up enough processes to cover everything you
> need.
>
> I suspect for 1 above that you would have to build a
> history table and allow the application to
> experimentally determine the best size over time.
> Rather complicated but doable.
>
> Alternatively if you are using the thin driver and
> locking is the issue then use the class loader.
> There is at least one thread in the JDBC forum that
> t discusses the correct way to do this. It isn't as
> easy as loading your own class.
>
> >
> > Are you implying that I can get another vendors
> Java
> > VM or that the SUN JVM will scale to the capacity
> > available on todays larger servers? For example,
> > the server Im currently running on has 4
> dedicated
> > processors and 16 gigabytes of system memory. Is
> > there a VM that will take advantage of all 4
> > processors when using threads (I have heard
> rumblings
> > that there is a practical ratio of Java threads to
> > available CPUs) while allowing those threads
> access
> > to 16 gig of memory? Im not being facetious here,
> I
> > am really wondering, because if that is the case,
> and
> > short of any other significant limitations, then
> the
> > overhead of designing and developing spawning
> > architecture is overkill (as I think you may have
> > been implying).
>
> There are 64 bit VMs that will use somewhere in the
> 10-16 gig range in the VM itself. Could be higher but
> those are the reports I have seen here.
Well, you got me completely confused now. I'm not sure why we can't get on the same page here. It's not that we are in disagreement on the solution, Im wide open on that, but we seem to disagree on what the problem is. I'll try again, but I feel that I must be missing something that is obvious to you and as a result I am not seeing your answers in the correct context.
My problem as I see it is that my 100 concurrent JVM's are taking 4 gigabytes of system memory every time they run. Startup (the first few seconds) is also taxing CPU, but not enough for the system administrators to yell at me.
A secondary problem that I am aware of but does not impact the current architecture is the Oracle driver can block multi-threaded applications (in testing).
I'm not experiencing any other significant resource issues. But the system memory issue must be resolved (by reducing total system memory usage) before I will be allowed to take on the additional 200 databases as part of this process.
Each JVM that I bring up uses a minimum of 40megabytes when it starts, but does not appear to take any additional system memory during execution. The JVM is started with no arguments, it uses the default JVM memory configuration for the O/S it is running on.
If I start a Thread in a JVM, it will take significantly less then 40 megabytes (this is what I believe to be true).
Let's pretend that starting a Thread takes/requires 1m additional system memory because I have completely exhausted the 40m I already have (which in reality I have not).
In a perfect world, I could start one JVM, it uses 40m, I start 100 Threads and now I am using 140m total system memory.
Now I know that the values I'm using here are not correct, but the point is why would I use 4 gigabytes of memory when I could in theory use as little as 140 megabytes and get the same amount of work done?
It sounds like you are suggesting that every time I start a Thread it will take 40m (or something similar to the JVM), and I just don't think that is the case. Even though Threads have their own execution path, it would be darn poor design for the not to share any code or libraries. I think it takes significantly less then 40mb, and that is where I see enhanced scalability when using Threads. If I am wrong about this, then Im just staring down the wrong road, which is fine because Im just searching for answers right now.
Where am I going wrong here? If Im a lost cause on this Ill understand
I did understand your comments on Class Loader, thank you for that. I thought Class Loaders were used more often in middle ware applications to help enable multi-user environments. I guess I am doing a form of that, so maybe it is applicable.
I was aware of the 64 bit JVM but have never implemented it. This application runs under Solaris in its largest incarnation, but also runs in Windows as well where the 64 bit JVM would not be available in most cases and probably not useful anyway since those servers are much smaller configurations using either 2g or 4g of total system memory.
> (Of course, this is all theory I haven't tried it
> myself...)
Outstanding, thank you for the detailed information. I have done much more on less theory, so that is not a problem for me. I'm not usually afraid of technology, but when my paycheck for the next several years rides on my decisions I sometimes at least pause to take a breath before barreling down my road.
Very good, thanks so much!
> I am rather certain that won't work with the OCI
> driver.
>
> it would with the thin driver.
That is very important to know, since I currently allow either one to be used with the application. I will have to think about the issues, but since 10g I really don't have any reservations about the thin driver, at least for the work that I do.
>
> Well, you got me completely confused now. I'm not
> sure why we can't get on the same page here. It's
> not that we are in disagreement on the solution, Im
> wide open on that, but we seem to disagree on what
> the problem is. I'll try again, but I feel that I
> must be missing something that is obvious to you and
> as a result I am not seeing your answers in the
> correct context.
>
> My problem as I see it is that my 100 concurrent
> JVM's are taking 4 gigabytes of system memory every
> time they run. Startup (the first few seconds) is
> also taxing CPU, but not enough for the system
> administrators to yell at me.
>
> A secondary problem that I am aware of but does not
> impact the current architecture is the Oracle driver
> can block multi-threaded applications (in testing).
>
Ok I see it now.
So back to your question....
What are the alternatives for a robust Java architecture that will be flexible enough to support 300 time sensitive concurrent tasks?
I suggested the general solution before. So changing it slightly...
1. Determine either statically or dynamically the threshold for each single process (probably based on 'type' as well.)
2. Create a main application that spins processes based on the threshold limits. It spins up enough processes to cover everything you need.
3. Each process is started with the necessary information (like command line or command line file) to allow it to determine which tasks to run.
Initially the static derivation is going to be easiest. If you collect statistics then you can manually adjust (carefully) to achieve a better solution over time.
By the way, based on 300 processes on a single box you might want to scale the network load. It won't matter much if your solution works if the network can't handle it.
> By the way, based on 300 processes on a single box
> you might want to scale the network load. It won't
> matter much if your solution works if the network
> can't handle it.
Right your are. I've talked to the sysadmins about the network issue and they assured me that it should be no problem, but they would monitor it and if it became a problem they have the ability to increase the network capacity on this server to whatever levels were required. Now, I don't know exactly how they plan on doing that, but they were young, confident and I've got it in writing, so I'm covered.
Well, I'm going to wind this thread down for now, but if you happen to think of anything that I should know I will definitely continue to monitor this thread, and will of course respond as appropriate (it just won't be the priority it has been the last couple of days).
I cannot tell you how much I appreciate everybody's thoughts and ideas. Now, I'm going to go hunker down in my bat cave and start testing out some of the ideas to make sure they work as I expect, and that I'm capable of actually coding them. After that I will finalize the architecture and the heavy coding will begin, which means I won't see the sun for another 24 weeks!
I've got a real good idea of the direction(s) I want to go to create a robust and scalable architecture that wreaks of performance thanks to you guys, but don't be surprised if I stop back and try and pick your brains one or twice more.
I am glad to say that whatever the chosen architecture turns out to be, it would not have been as good without all of your help, so once again, I thank you all individually and collectively.
What a pleasurable post to read from beginning to end.~D
If you start several JVMs each will compile the byte code into machine code separately. For hundreds of processes Hotspot takes a non-trivial amount of memory to do that. With one process and hundreds of threads Hotspot only compiles once; I'd expect memory savings in the hundreds of megabytes. There are some papers on the Barcelona project with data: http://research.sun.com/projects/barcelona/index.html
I wrote a little test that accesses memory-based databases (Derby, HSQLDB) using lots of threads in a tight loop. Even on a Windows PC hundreds of threads seem to work fine. The program gets slower when I start more threads but I don't see a catastrophic drop. So a few hundred threads as such shouldn't be a problem. The Oracle JDBC driver could be a problem if it really manages to have separate connections interfere with each other.
> > By the way, based on 300 processes on a single box
> > you might want to scale the network load. It
> won't
> > matter much if your solution works if the network
> > can't handle it.
>
> Right your are. I've talked to the sysadmins about
> the network issue and they assured me that it should
> be no problem, but they would monitor it and if it
> became a problem they have the ability to increase
> the network capacity on this server to whatever
> levels were required. Now, I don't know exactly how
> they plan on doing that, but they were young,
> confident and I've got it in writing, so I'm covered.
No problem then. :)
> If you start several JVMs each will compile the byte
> code into machine code separately. For hundreds of
> processes Hotspot takes a non-trivial amount of
> memory to do that. With one process and hundreds of
> threads Hotspot only compiles once; I'd expect memory
> savings in the hundreds of megabytes. There are some
> papers on the Barcelona project with data:
> http://research.sun.com/projects/barcelona/index.html
>
That's what my assumption is, but I appreciate hearing someone else say it with confidence. Ill check out the Barcelona project papers; thanks for the resource.
> I wrote a little test that accesses memory-based
> databases (Derby, HSQLDB) using lots of threads in a
> tight loop. Even on a Windows PC hundreds of threads
> seem to work fine. The program gets slower when I
> start more threads but I don't see a catastrophic
> drop. So a few hundred threads as such shouldn't be a
> problem. The Oracle JDBC driver could be a problem if
> it really manages to have separate connections
> interfere with each other.
That sounds great. I am concerned about the Oracle driver, and I think I have to architect with the assumption that it has limitations but I'm also guessing I will be much more intimate with it by the time I complete this project. If I do find that the problem is my code and not the Oracle driver, or there is a work around for the Oracle driver concurrency issue I will post that information.
Thanks for the post!
Maybe this is totally naive but is it possible to write your own Oracle driver, if that turns out to be the issue?If you could you could certainly optimise it to meet your specific applications requirements. It might also be easier then changing your entire architecture ... ?
> Maybe this is totally naive but is it possible to
> write your own Oracle driver, if that turns out to be
> the issue?
>
> If you could you could certainly optimise it to meet
> your specific applications requirements.
I don't think it would be wise to take on writing an Oracle driver myself. My brain would be willing, maybe even excited by the idea, but my background would be lacking and I think it would take too much time to complete. That doesn't mean it's a dumb or naive idea, it just means that for good or bad I don't have the skills necessary to accomplish it.
I've had similar out of the box thoughts myself regarding the Oracle concurrency issue. One of the things I considered was moving the application closer to the "common" LAN database and actually run the processes within the Oracle internal JVM. The Oracle internal JVM is a proprietary implementation designed to work well within a multi-tasking / multi-threaded environment. The internal JVM was designed by Oracle to support hundreds and perhaps thousands of concurrent users (Java processes). Oracle actually recommends moving poorly performing data intensive applications to the internal JVM for enhanced performance. Oracle suggests that the move is trivial, but with the internal JVMs significant limitations in file I/O I think most applications would require significant code changes to successfully make the transaction. Still, I have written several Oracle functions in Java that connect to and use the internal JVM and from my experience; Oracle seems to have done this internal JVM thing right. Depending on the amount of coding overhead that is required to implement my own non-blocking / multi-tasking environment it is not impossible that I will take another look at the Oracle internal JVM.
> It might also be easier then changing your entire architecture
> ... ?
Well, that is a pretty good point and I should try and remember that my goal isn't a new architecture. A new architecture is just one solution to my real goal of minimizing resources and increasing capacity of my application.
> What a pleasurable post to read from beginning to
> end.
>
> ~D
Indeed! And not a Duke dollar in sight.
This was altruism in it's purest and brightest form. As the recipient I hope to repay the gift by helping others in this same forum. I've already tried several times and despite limited success, I will continue my efforts as time permits. I'm glad you enjoyed the post and I hope that you can take as much from it as I have.
Looks like to me your only real issue is your lack of confidence in the Oracle Driver. it looks like the Oracle Driver is the only underperforming part of the system, not Java itself.
I would ask Oracle about that performance degradation when reacing 32 threads.
However, personally I would not spawn so many threads if they have to access the same reasource. There is really no point. Sure its a easy way to not have to manage the resource so finely, but your not gaining anything by it.
Perhaps a thread pool, and tune the number of threads based on performance. That would scale better than a hard thread for each accessor.
J2EE can indeed handle this. You can use j2EE without using EJBs if you wish. In any event the whole point of J2EE is scalability. You are not going to beat J2EE on your own without spending a considerable amount of time duplicating what they have solved in many cases.
im also concerned with how you start your sessions.if you are not passing data with the start command i would prefer if the jvm could handle timing and starting itself. Or create a program that runs all the time that can signal when it wants to start a session. Gotta stop creating a jvm just to start a session, even if it is immediately destroyed.
> Looks like to me your only real issue is your lack of
> confidence in the Oracle Driver. it looks like the
> Oracle Driver is the only underperforming part of the
> system, not Java itself.
>
> I would ask Oracle about that performance degradation
> when reacing 32 threads.
This isn't 100% the case; the Oracle locking is a problem, and Threads wait on some Oracle resource; which is a huge problem for my timing sensitive application. However, the performance degradation is not necessarily related to the Oracle driver, I believe it is due to context switching and the reason that I mentioned 16 and 32 Threads is I hit a different Threshold depending on how many actual processors are on the server, in my case it was 2 and 4 respectively. But the testing was not exhaustive and further testing and research would have to be to done to identify the root cause(s).
> However, personally I would not spawn so many threads
> if they have to access the same reasource. There is
> really no point. Sure its a easy way to not have to
> manage the resource so finely, but your not gaining
> anything by it.
Im not really clear on your point; one thought I described was that each Thread would be accessing both a common (the same) and a different database. What they have in common is the use of the Oracle driver itself, but three DB connections (2 to the common and 1 to the unique database) would be unique to the Thread. Each Thread is creating a separate instance of the primary working class (call it the business logic), but may share logging classes instances, etc. Im not sure if that clarifies anything, but again I wasnt certain what your point was.
> Perhaps a thread pool, and tune the number of threads
> based on performance. That would scale better than a
> hard thread for each accessor.
Yes, that is my thought too. I have already implemented just that in one of our more recent programs, and the Thread count is currently limited by processor count and using Thread groups it will also prioritize certain types of Threads (they get to start first).
> J2EE can indeed handle this. You can use j2EE
> without using EJBs if you wish. In any event the
> whole point of J2EE is scalability. You are not
> going to beat J2EE on your own without spending a
> considerable amount of time duplicating what they
> have solved in many cases.
If you are talking about an application server I am still quite leery (see an earlier post in this thread) of implementing long running batch transactions in an environment designed and optimized for online fast running transactions. Sorry, if I got your point wrong here. I also mentioned in one of the posts that I was looking at the possibility of implementing some of this using the Oracle internal JVM which provides much of what you have discussed in your post. The Oracle internal JVM may also be optimized for short running transactions, but I dont know that. Im also just not sure I want to get that proprietary with the code.
> im also concerned with how you start your sessions.if
> you are not passing data with the start command i
> would prefer if the jvm could handle timing and
> starting itself. Or create a program that runs all
> the time that can signal when it wants to start a
> session. Gotta stop creating a jvm just to start a
> session, even if it is immediately destroyed.
I am currently passing data through script files to start each and every JVM with a unique set of parameters. Everything is run off cron or something like cron under Windows. If I was going to use a program that runs all the time I would definitely look first at an application server of some kind, as you mentioned earlier, they have built and tested that functionality so I dont have to. Im not sure what your last sentence is trying to tell me, although I agree that I want to stop creating a JVM for every batch transaction, and is in fact one of the primary drivers behind a look at re-architecting my application.
Your comments are appreciated.
>
> If you are talking about an application server I am
> still quite leery (see an earlier post in this
> thread) of implementing long running batch
> transactions in an environment designed and optimized
> for online fast running transactions.
I would suppose that most (like 99%) of systems that do something called a 'transaction' would require batch processing.
That is certainly true of the payments industry.
Even in a web content only network one should probably still be running something on the server logs to look for problems. That would be a batch process.
I would guess that the batch processing is fundamental to the business as well. It certainly is in the payments industry.
Perhaps you are just referring to the possible impact on performance. If so the system needs to be architected to handle that.
