记一次事务管理@Transactional导致的BUG
背景:uat和dev2个环境,进行实时数据双向同步。且高并发。单无人机600毫秒一次。
dev向uat同步无人机实时轨迹的时候, dev不存在卡顿,uat每隔20多秒就会卡顿,一次卡顿就20秒。但是不影响使用。
现在有时间进行问题排查,做出记录。
正确排查过程:
1.,每一行打事件日志,定位问题代码,根据数据日志找到对应的线程。
2.通过arthas跟踪对应线程。(arthas不可全信, 卡顿的时候输出所有线程找不到卡顿的线程,必须单独跟踪。)
3.堆栈信息:
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74f2d65a
at java.base@17.0.0.1/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74f2d65a
at java.base@17.0.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
at java.base@17.0.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1757)
at com.alibaba.druid.pool.DruidDataSource.pollLast(DruidDataSource.java:2368)
at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1799)
at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1502)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5074)
at com.alibaba.druid.filter.logging.LogFilter.dataSource_getConnection(LogFilter.java:915)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5070)
at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:724)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5070)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1477)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1463)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:83)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:269)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:532)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:405)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:617)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:386)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720)
at com.airdance.core.service.BufferService$$SpringCGLIB$$0.getMessageBuffer(<generated>)
at com.airdance.com.DataProcessingService.httpData(DataProcessingService.java:166)
at com.airdance.api.sync.out.TcpTrajectoryDataSynchronizationService.handleMessage(TcpTrajectoryDataSynchronizationService.java:118)
at com.airdance.core.route.AbstractRouteFlow.handleMessage(AbstractRouteFlow.java:46)
at jdk.internal.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
at java.base@17.0.0.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base@17.0.0.1/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:169)
at org.springframework.integration.handler.support.IntegrationInvocableHandlerMethod.doInvoke(IntegrationInvocableHandlerMethod.java:45)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119)
at org.springframework.integration.handler.support.MessagingMethodInvokerHelper$HandlerMethod.invoke(MessagingMethodInvokerHelper.java:1086)
at org.springframework.integration.handler.support.MessagingMethodInvokerHelper.invokeHandlerMethod(MessagingMethodInvokerHelper.java:569)
at org.springframework.integration.handler.support.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:482)
at org.springframework.integration.handler.support.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:360)
at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:114)
at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:95)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:145)
at org.springframework.integration.handler.AbstractMessageHandler.doHandleMessage(AbstractMessageHandler.java:105)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:132)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
at org.springframework.integration.channel.AbstractMessageChannel.sendInternal(AbstractMessageChannel.java:378)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:332)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:302)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
at org.springframework.integration.endpoint.MessageProducerSupport.lambda$sendMessage$1(MessageProducerSupport.java:262)
at org.springframework.integration.endpoint.MessageProducerSupport$$Lambda$1734/0x000000080161d7f8.run(Unknown Source)
at io.micrometer.observation.Observation.observe(Observation.java:499)
at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:262)
at org.springframework.integration.ip.tcp.TcpReceivingChannelAdapter.onMessage(TcpReceivingChannelAdapter.java:88)
at org.springframework.integration.ip.tcp.connection.TcpNetConnection.receiveAndProcessMessage(TcpNetConnection.java:233)
at org.springframework.integration.ip.tcp.connection.TcpNetConnection.run(TcpNetConnection.java:206)
at java.base@17.0.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base@17.0.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base@17.0.0.1/java.lang.Thread.run(Thread.java:833)
4.产生思考;一个内存缓存数组,为什么会出现德鲁伊的数据库连堆栈?


结论:前面的架构居然在整个类上加了事务。导致我哪怕内存缓存获取都能卡住。
代码没有问题的时候其实是不会去看这些地方的。

排查的时候AI没有作用。
但是排出原因后,AI还是能很快的得出结论:
## 为什么"没有查数据库却走到DruidDataSource"?
因为`@Transactional`注解会:
1. 在方法开始时开启事务(需要数据库连接)
2. 即使方法内部没有数据库操作,事务框架仍需获取连接来管理事务状态
3. 高并发下连接池竞争导致线程阻塞
AtomGit 是由开放原子开源基金会联合 CSDN 等生态伙伴共同推出的新一代开源与人工智能协作平台。平台坚持“开放、中立、公益”的理念,把代码托管、模型共享、数据集托管、智能体开发体验和算力服务整合在一起,为开发者提供从开发、训练到部署的一站式体验。
更多推荐



所有评论(0)