Bug in Oracle JDBC Pooling Classes - Deadlock
We are utilizing Oracle's connection caching (drivers 10.2.0.1) and have found a deadlock situation. I reviewed the code for the (drivers 10.2.0.3) and I see the same problem could happen.
I searched and have not found this problem identified anywhere. Is this something I should post to Oracle in some way (i.e. Metalink?) or is there a better forum to get this resolved?
We are utilizing an OCI driver with the following setup in the server.xml
<ResourceParams name="cmf_toolbox">
<parameter>
<name>factory</name>
<value>oracle.jdbc.pool.OracleDataSourceFactory</value>
</parameter>
<parameter>
<name>driverClassName</name>
<value>oracle.jdbc.driver.OracleDriver</value>
</parameter>
<parameter>
<name>user</name>
<value>hidden</value>
</parameter>
<parameter>
<name>password</name>
<value>hidden</value>
</parameter>
<parameter>
<name>url</name>
<value>jdbc:oracle:oci:@PTB2</value>
</parameter>
<parameter>
<name>connectionCachingEnabled</name>
<value>true</value>
</parameter>
<parameter>
<name>connectionCacheProperties</name>
<value>(InitialLimit=5,MinLimit=15,MaxLimit=75,ConnectionWaitTimeout=30,InactivityTimeout=300,AbandonedConnectionTimeout=300,ValidateConnection=false)</value>
</parameter>
</ResourceParams>
We get a deadlock situation between two threads and the exact steps are this:
1) thread1 - The OracleImplicitConnectionClassThread class is executing the runAbandonedTimeout method which will lock the OracleImplicitConnectionCache class with a synchronized block. It will then go thru additional steps and finally try to call the LogicalConnection.close method which is already locked by thread2
2) thread2 - This thread is doing a standard .close() on the Logical Connection and when it does this it obtains a lock on the LogicalConnection class. This thread then goes through additional steps till it gets to a point in the OracleImplicitConnectionCache class where it executes the reusePooledConnection method. This method is synchronized.
Actual steps that cause deadlock:
1) thread1 locks OracleImplicitConnectionClass in runAbandonedTimeout method
2) thread2 locks LogicalConnection class in close function.
3) thread1 tries to lock the LogicalConnection and is unable to do this, waits for lock
4) thread2 tries to lock the OracleImplicitConnectionClass and waits for lock.
***DEADLOCK***
Thread Dumps from two threads listed above
thread1
Thread Name : Thread-1 State : Deadlock/Waiting on monitor Owns Monitor Lock on 0x30267fe8 Waiting for Monitor Lock on 0x509190d8 Java Stack at oracle.jdbc.driver.LogicalConnection.close(LogicalConnection.java:214) - waiting to lock 0x509190d8> (a oracle.jdbc.driver.LogicalConnection) at oracle.jdbc.pool.OracleImplicitConnectionCache.closeCheckedOutConnection(OracleImplicitConnectionCache.java:1330) at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.runAbandonedTimeout(OracleImplicitConnectionCacheThread.java:261) - locked 0x30267fe8> (a oracle.jdbc.pool.OracleImplicitConnectionCache) at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:81)
thread2
Thread Name : http-7320-Processor83 State : Deadlock/Waiting on monitor Owns Monitor Lock on 0x509190d8 Waiting for Monitor Lock on 0x30267fe8 Java Stack at oracle.jdbc.pool.OracleImplicitConnectionCache.reusePooledConnection(OracleImplicitConnectionCache.java:1608) - waiting to lock 0x30267fe8> (a oracle.jdbc.pool.OracleImplicitConnectionCache) at oracle.jdbc.pool.OracleConnectionCacheEventListener.connectionClosed(OracleConnectionCacheEventListener.java:71) - locked 0x34d514f8> (a oracle.jdbc.pool.OracleConnectionCacheEventListener) at oracle.jdbc.pool.OraclePooledConnection.callImplicitCacheListener(OraclePooledConnection.java:544) at oracle.jdbc.pool.OraclePooledConnection.logicalCloseForImplicitConnectionCache(OraclePooledConnection.java:459) at oracle.jdbc.pool.OraclePooledConnection.logicalClose(OraclePooledConnection.java:475) at oracle.jdbc.driver.LogicalConnection.closeInternal(LogicalConnection.java:243) at oracle.jdbc.driver.LogicalConnection.close(LogicalConnection.java:214) - locked 0x509190d8> (a oracle.jdbc.driver.LogicalConnection) at com.schoolspecialty.cmf.yantra.OrderDB.updateOrder(OrderDB.java:2022) at com.schoolspecialty.cmf.yantra.OrderFactoryImpl.saveOrder(OrderFactoryImpl.java:119) at com.schoolspecialty.cmf.yantra.OrderFactoryImpl.saveOrder(OrderFactoryImpl.java:67) at com.schoolspecialty.ecommerce.beans.ECommerceUtil.saveOrder(Unknown Source) at com.schoolspecialty.ecommerce.beans.ECommerceUtil.saveOrder(Unknown Source) at com.schoolspecialty.ecommerce.beans.UpdateCartAction.perform(Unknown Source) at com.schoolspecialty.mvc2.ActionServlet.doPost(ActionServlet.java:112) at com.schoolspecialty.ecommerce.servlets.ECServlet.doPostOrGet(Unknown Source) at com.schoolspecialty.ecommerce.servlets.ECServlet.doPost(Unknown Source) at javax.servlet.http.HttpServlet.service(HttpServlet.java:709) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157) at com.schoolspecialty.ecommerce.servlets.filters.EcommerceURLFilter.doFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:186) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683) at java.lang.Thread.run(Thread.java:534)
