白云岛资源网

dbcp 连接池不合理的锁导致连接耗尽解决方案
站长资源 2024/12/23 佚名
18 1538
白云岛资源网 Design By www.pvray.com

dbcp 连接池不合理的锁导致连接耗尽解决方案

应用报错,表象来看是连接池爆满了。

org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:241) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]
    at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82) [dubbo-2.5.3.jar:2.5.3]
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_33]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_33]
    at java.lang.Thread.run(Thread.java:662) [na:1.6.0_33]
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:103) ~[commons-dbcp.jar:1.2.1]
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540) ~[commons-dbcp.jar:1.2.1]
    at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74) ~[crmpub-jdbc-1.0.jar:1.0]
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]
    ... 32 common frames omitted
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174) ~[commons-pool-1.6.jar:1.6]
    at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) ~[commons-dbcp.jar:1.2.1]
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) ~[commons-dbcp.jar:1.2.1]
    ... 35 common frames omitted

应用堆栈，几十条线程都阻塞到dbcp池的PoolableConnectionFactory.makeObject方法了，此方法在等待0x0000000709a638a0对象锁，但0x0000000709a638a0对象一直被其中一条线程执行PoolableConnectionFactory.makeObject时锁了，而且这条线程获取到锁后还阻塞住了，所以导致后面几十条线程都BLOCKED了。

"DubboServerHandler-132.121.95.136:28101-thread-300" daemon prio=10 tid=0x00007f02b4136800 nid=0x4954 waiting for monitor entry [0x00007f0286018000]
  java.lang.Thread.State: BLOCKED (on object monitor)
  at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
  - waiting to lock <0x0000000709a638a0> (a org.apache.commons.dbcp.PoolableConnectionFactory)
  at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
  at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)
  at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
  at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
  at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74)
  at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)
  at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
  at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
  at com.eshore.crm.service.common.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74)
  at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
  at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
  at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
  at com.eshore.crm.service.sysmgr.sca.impl.SysMgrServiceImpl$$EnhancerByCGLIB$$bfd71326.qryBulletinList(<generated>)
  at com.alibaba.dubbo.common.bytecode.Wrapper101.invokeMethod(Wrapper101.java)
  at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)
  at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)
  at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
  at com.eshore.crmpub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90)
  at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
  at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)
  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
  at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
  at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  at java.lang.Thread.run(Thread.java:662)

  Locked ownable synchronizers:
  - <0x000000070a120f00> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)


"DubboServerHandler-132.121.95.136:28101-thread-290" daemon prio=10 tid=0x00007f03203da800 nid=0x4948 runnable [0x00007f0286a22000]
  java.lang.Thread.State: RUNNABLE
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.read(SocketInputStream.java:129)
  at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
  at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
  at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
  - locked <0x0000000715a9ce70> (a com.mysql.jdbc.util.ReadAheadInputStream)
  at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3140)
  at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:597)
  at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1085)
  at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2494)
  at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2527)
  at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2309)
  - locked <0x00000007159d1898> (a com.mysql.jdbc.JDBC4Connection)
  at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
  at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
  at sun.reflect.GeneratedConstructorAccessor78.newInstance(Unknown Source)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
  at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:419)
  at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
  at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
  at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
  - locked <0x0000000709a638a0> (a org.apache.commons.dbcp.PoolableConnectionFactory)
  at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
  at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)
  at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
  at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
  at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74)
  at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)
  at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
  at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
  at com.eshore.crm.service.common.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74)
  at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
  at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
  at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
  at com.eshore.crm.service.cachemgr.sca.common.DictCacheServiceImpl$$EnhancerByCGLIB$$e27e8e25.getDict(<generated>)
  at com.alibaba.dubbo.common.bytecode.Wrapper93.invokeMethod(Wrapper93.java)
  at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)
  at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)
  at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
  at com.eshore.crmpub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90)
  at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
  at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)
  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
  at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
  at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  at java.lang.Thread.run(Thread.java:662)

  Locked ownable synchronizers:
  - <0x000000070a0f9218> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

可以看到其实获取锁的线程是阻塞在这里了，做mysql握手包的时候一直在等待读，阻塞在网络IO了。

 java.lang.Thread.State: RUNNABLE
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.read(SocketInputStream.java:129)
  at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
  at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
  at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
  - locked <0x0000000715a9ce70> (a com.mysql.jdbc.util.ReadAheadInputStream)
  at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3140)
  at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:597)
  at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1085)
  at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2494)
  at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2527)
  at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2309)

这个是dbcp1.x版本在创建新的连接对象时会把整个连接工厂类对象锁了，一旦创建过程中发生阻塞就会导致整个池都死掉。这个问题很大。

解决方案，升级dbcp2.x版本或tomcat jdbc或druid连接池，推荐druid连接池，功能强大自带监控。

感谢阅读，希望能帮助到大家，谢谢大家对本站的支持！

白云岛资源网 Design By www.pvray.com
广告合作：本站广告合作请联系QQ：858582 申请时备注：广告合作（否则不回）
免责声明：本站资源来自互联网收集,仅供用于学习和交流,请遵循相关法律法规,本站一切资源不代表本站立场,如有侵权、后门、不妥请联系本站删除！
白云岛资源网 Design By www.pvray.com
www.pvray.com 白云岛资源网
39,976影音资源
44,792技术资源
21,817软件资源
651,128站长资源

友情链接

杰晶网络 DDR爱好者之家 桃源资源网 杰网资源 富贵资源网 南强小屋 铁雪资源网 幽灵资源网 万梅资源网 狼山资源网 白云岛资源网 昆仑资源网 相思资源网 明霞山资源网 内蒙古资源网 黑松山资源网 茶园资源网 饿虎岗资源网 大旗谷资源网 常春岛资源网 岱庙资源网 兴国资源网 快活林资源网 蝙蝠岛资源网 帝王谷资源网 白云城资源网 伏龙阁资源网 清风细雨楼 天枫庄资源网 圆月山庄资源网 无争山庄资源网 神水资源网 移花宫资源网 神剑山庄资源网 无为清净楼资源网 金钱帮资源网 丐帮资源网 华山资源网 极乐门资源网 小李飞刀资源网 凤求凰客栈 风云阁资源网 金狮镖局 鸳鸯亭资源网 千金楼资源网 更多链接
白云岛资源网 Design By www.pvray.com