dbcp 连接池不合理的锁导致连接耗尽解决方案

dbcp 连接池不合理的锁导致连接耗尽解决方案

应用报错,表象来看是连接池爆满了。

org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:241) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]
    at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82) [dubbo-2.5.3.jar:2.5.3]
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_33]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_33]
    at java.lang.Thread.run(Thread.java:662) [na:1.6.0_33]
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:103) ~[commons-dbcp.jar:1.2.1]
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540) ~[commons-dbcp.jar:1.2.1]
    at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74) ~[crmpub-jdbc-1.0.jar:1.0]
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]
    ... 32 common frames omitted
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174) ~[commons-pool-1.6.jar:1.6]
    at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) ~[commons-dbcp.jar:1.2.1]
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) ~[commons-dbcp.jar:1.2.1]
    ... 35 common frames omitted

应用堆栈，几十条线程都阻塞到dbcp池的PoolableConnectionFactory.makeObject方法了，此方法在等待0x0000000709a638a0对象锁，但0x0000000709a638a0对象一直被其中一条线程执行PoolableConnectionFactory.makeObject时锁了，而且这条线程获取到锁后还阻塞住了，所以导致后面几十条线程都BLOCKED了。

"DubboServerHandler-132.121.95.136:28101-thread-300" daemon prio=10 tid=0x00007f02b4136800 nid=0x4954 waiting for monitor entry [0x00007f0286018000]
  java.lang.Thread.State: BLOCKED (on object monitor)
  at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
  - waiting to lock <0x0000000709a638a0> (a org.apache.commons.dbcp.PoolableConnectionFactory)
  at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
  at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)
  at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
  at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
  at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74)
  at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)
  at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
  at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
  at com.eshore.crm.service.common.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74)
  at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
  at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
  at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
  at com.eshore.crm.service.sysmgr.sca.impl.SysMgrServiceImpl$$EnhancerByCGLIB$$bfd71326.qryBulletinList(<generated>)
  at com.alibaba.dubbo.common.bytecode.Wrapper101.invokeMethod(Wrapper101.java)
  at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)
  at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)
  at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
  at com.eshore.crmpub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90)
  at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
  at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)
  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
  at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
  at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  at java.lang.Thread.run(Thread.java:662)

  Locked ownable synchronizers:
  - <0x000000070a120f00> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"DubboServerHandler-132.121.95.136:28101-thread-290" daemon prio=10 tid=0x00007f03203da800 nid=0x4948 runnable [0x00007f0286a22000]
  java.lang.Thread.State: RUNNABLE
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.read(SocketInputStream.java:129)
  at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
  at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
  at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
  - locked <0x0000000715a9ce70> (a com.mysql.jdbc.util.ReadAheadInputStream)
  at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3140)
  at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:597)
  at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1085)
  at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2494)
  at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2527)
  at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2309)
  - locked <0x00000007159d1898> (a com.mysql.jdbc.JDBC4Connection)
  at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
  at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
  at sun.reflect.GeneratedConstructorAccessor78.newInstance(Unknown Source)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
  at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:419)
  at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
  at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
  at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
  - locked <0x0000000709a638a0> (a org.apache.commons.dbcp.PoolableConnectionFactory)
  at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
  at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)
  at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
  at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
  at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74)
  at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)
  at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
  at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
  at com.eshore.crm.service.common.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74)
  at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
  at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
  at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
  at com.eshore.crm.service.cachemgr.sca.common.DictCacheServiceImpl$$EnhancerByCGLIB$$e27e8e25.getDict(<generated>)
  at com.alibaba.dubbo.common.bytecode.Wrapper93.invokeMethod(Wrapper93.java)
  at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)
  at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)
  at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
  at com.eshore.crmpub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90)
  at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
  at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)
  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
  at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
  at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  at java.lang.Thread.run(Thread.java:662)

  Locked ownable synchronizers:
  - <0x000000070a0f9218> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

可以看到其实获取锁的线程是阻塞在这里了，做mysql握手包的时候一直在等待读，阻塞在网络IO了。

 java.lang.Thread.State: RUNNABLE
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.read(SocketInputStream.java:129)
  at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
  at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
  at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
  - locked <0x0000000715a9ce70> (a com.mysql.jdbc.util.ReadAheadInputStream)
  at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3140)
  at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:597)
  at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1085)
  at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2494)
  at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2527)
  at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2309)

这个是dbcp1.x版本在创建新的连接对象时会把整个连接工厂类对象锁了，一旦创建过程中发生阻塞就会导致整个池都死掉。这个问题很大。

解决方案，升级dbcp2.x版本或tomcat jdbc或druid连接池，推荐druid连接池，功能强大自带监控。

感谢阅读，希望能帮助到大家，谢谢大家对本站的支持！

# dbcp  # 连接池不合理的锁导致连接耗尽  # 连接池锁导致连接耗尽  # 浅谈在Spring中如何使用数据源(DBCP、C3P0、JNDI)  # Java数据库连接池之DBCP浅析_动力节点Java学院整理  # Tomcat 7-dbcp配置数据库连接池详解  # java使用dbcp2数据库连接池  # java配置dbcp连接池(数据库连接池)示例分享  # Java dbcp连接池基本使用方法详解  # 连接池  # 在等待  # 十条  # 就会  # 在这  # 这个问题  # 住了  # 这条  # 希望能  # 可以看到  # 自带  # 报错  # 谢谢大家  # 里了  # 死掉  # 过程中  # prio  # 一直被  # DubboServerHandler  # makeObject 

相关文章： 详解一款开源免费的.NET文档操作组件DocX（.NET组件介绍之一）  c# 在ASP.NET Core中管理和取消后台任务  佛山企业网站制作公司有哪些,沟通100网上服务官网？  宝盒自助建站智能生成技巧：SEO优化与关键词设置指南  制作网站软件推荐手机版,如何制作属于自己的手机网站app应用？  北京营销型网站制作公司,可以用python做一个营销推广网站吗？  建站之星备案是否影响网站上线时间？  如何快速搭建FTP站点实现文件共享？  建站主机如何安装配置？新手必看操作指南  如何快速搭建自助建站会员专属系统？  建站之星导航如何优化提升用户体验？  小型网站制作HTML,*游戏网站怎么搭建？  网站制作外包价格怎么算,招聘网站上写的“外包”是什么意思？  如何通过主机屋免费建站教程十分钟搭建网站？  制作网站的公司有哪些,做一个公司网站要多少钱？  香港服务器WordPress建站指南：SEO优化与高效部署策略  如何获取上海专业网站定制建站电话？  教学论文网站制作软件有哪些,写论文用什么软件 ？  微信小程序 input输入框控件详解及实例（多种示例）  济南企业网站制作公司,济南社保单位网上缴费步骤？  外贸公司网站制作哪家好,maersk船公司官网？  广州顶尖建站服务：企业官网建设与SEO优化一体化方案  如何配置FTP站点权限与安全设置？  h5在线制作网站电脑版下载,h5网页制作软件？  如何制作网站标识牌,动态网站如何制作（教程）？  广州网站制作的公司,现在专门做网站的公司有没有哪几家是比较好的，性价比高，模板也多的？  网站制作怎么样才能赚钱,用自己的电脑做服务器架设网站有什么利弊，能赚钱吗？  建站之星在线客服如何快速接入解答？  C#如何在一个XML文件中查找并替换文本内容  高防服务器租用如何选择配置与防御等级？  如何通过PHP快速构建高效问答网站功能？  详解jQuery中基本的动画方法  公司门户网站制作流程,华为官网怎么做？  建站之星如何修改网站生成路径？  建站之星展会模版如何一键下载生成？  建站之星安装步骤有哪些常见问题？  如何制作算命网站,怎么注册算命网站？  建站之星后台密码遗忘如何找回？  如何用AWS免费套餐快速搭建高效网站？  建站为何优先选择香港服务器？  金*站制作公司有哪些,金华教育集团官网？  C++中引用和指针有什么区别？（代码说明）  兔展官网 在线制作,怎样制作微信请帖？  建站VPS选购需注意哪些关键参数？  SQL查询语句优化的实用方法总结  ,网页ppt怎么弄成自己的ppt？  如何快速搭建安全的FTP站点？  如何在云指建站中生成FTP站点？  存储型VPS适合搭建中小型网站吗？  建站主机SSH密钥生成步骤及常见问题解答？