1 现象
今天在uat环境压力测试的时候报如下错误
java.lang.RuntimeException: Timed out waiting for operation at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:135) at org.jasig.cas.ticket.registry.MemCacheTicketRegistry.updateTicket(MemCacheTicketRegistry.java:109) at org.jasig.cas.ticket.registry.AbstractDistributedTicketRegistry$TicketDelegator.updateTicket(AbstractDistributedTicketRegistry.java:73) at org.jasig.cas.ticket.registry.AbstractDistributedTicketRegistry$TicketGrantingTicketDelegator.grantServiceTicket_aroundBody2(AbstractDistributedTicketRegistry.java:167) at org.jasig.cas.ticket.registry.AbstractDistributedTicketRegistry$TicketGrantingTicketDelegator.grantServiceTicket_aroundBody3$advice(AbstractDistributedTicketRegistry.java:57) at org.jasig.cas.ticket.registry.AbstractDistributedTicketRegistry$TicketGrantingTicketDelegator.grantServiceTicket(AbstractDistributedTicketRegistry.java:1) at org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket_aroundBody2(CentralAuthenticationServiceImpl.java:243) at org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket_aroundBody3$advice(CentralAuthenticationServiceImpl.java:57) at org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:1) at org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket_aroundBody4(CentralAuthenticationServiceImpl.java:276) at org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket_aroundBody5$advice(CentralAuthenticationServiceImpl.java:57) at org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:1) at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) at org.perf4j.aop.AbstractTimingAspect$1.proceed(AbstractTimingAspect.java:47) at org.perf4j.aop.AgnosticTimingAspect.runProfiledMethod(AgnosticTimingAspect.java:53) at org.perf4j.aop.AbstractTimingAspect.doPerfLogging(AbstractTimingAspect.java:45) at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) at com.github.inspektr.audit.AuditTrailManagementAspect.handleAuditTrail(AuditTrailManagementAspect.java:126) at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at com.sun.proxy.$Proxy25.grantServiceTicket(Unknown Source) at org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:52) at org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:188) at org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51) at org.springframework.webflow.action.EvaluateAction.doExecute(EvaluateAction.java:77) at org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:188) at org.springframework.webflow.execution.AnnotatedAction.execute(AnnotatedAction.java:145) at org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51) at org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:101) at org.springframework.webflow.engine.State.enter(State.java:194) at org.springframework.webflow.engine.Transition.execute(Transition.java:227) at org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:51) at org.springframework.webflow.engine.State.enter(State.java:194) at org.springframework.webflow.engine.Transition.execute(Transition.java:227) at org.springframework.webflow.engine.impl.FlowExecutionImpl.execute(FlowExecutionImpl.java:393) at org.springframework.webflow.engine.impl.RequestControlContextImpl.execute(RequestControlContextImpl.java:214) at org.springframework.webflow.engine.TransitionableState.handleEvent(TransitionableState.java:119) at org.springframework.webflow.engine.Flow.handleEvent(Flow.java:555) at org.springframework.webflow.engine.impl.FlowExecutionImpl.handleEvent(FlowExecutionImpl.java:388) at org.springframework.webflow.engine.impl.RequestControlContextImpl.handleEvent(RequestControlContextImpl.java:210) at org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:105) at org.springframework.webflow.engine.State.enter(State.java:194) at org.springframework.webflow.engine.Transition.execute(Transition.java:227) at org.springframework.webflow.engine.impl.FlowExecutionImpl.execute(FlowExecutionImpl.java:393) at org.springframework.webflow.engine.impl.RequestControlContextImpl.execute(RequestControlContextImpl.java:214) at org.springframework.webflow.engine.TransitionableState.handleEvent(TransitionableState.java:119) at org.springframework.webflow.engine.Flow.handleEvent(Flow.java:555) at org.springframework.webflow.engine.impl.FlowExecutionImpl.handleEvent(FlowExecutionImpl.java:388) at org.springframework.webflow.engine.impl.RequestControlContextImpl.handleEvent(RequestControlContextImpl.java:210) at org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:105) at org.springframework.webflow.engine.State.enter(State.java:194) at org.springframework.webflow.engine.Transition.execute(Transition.java:227) at org.springframework.webflow.engine.impl.FlowExecutionImpl.execute(FlowExecutionImpl.java:393) at org.springframework.webflow.engine.impl.RequestControlContextImpl.execute(RequestControlContextImpl.java:214) at org.springframework.webflow.engine.TransitionableState.handleEvent(TransitionableState.java:119) at org.springframework.webflow.engine.Flow.handleEvent(Flow.java:555) at org.springframework.webflow.engine.impl.FlowExecutionImpl.handleEvent(FlowExecutionImpl.java:388) at org.springframework.webflow.engine.impl.RequestControlContextImpl.handleEvent(RequestControlContextImpl.java:210) at org.springframework.webflow.engine.ViewState.handleEvent(ViewState.java:232) at org.springframework.webflow.engine.ViewState.resume(ViewState.java:196) at org.springframework.webflow.engine.Flow.resume(Flow.java:545) at org.springframework.webflow.engine.impl.FlowExecutionImpl.resume(FlowExecutionImpl.java:261) at org.springframework.webflow.executor.FlowExecutorImpl.resumeExecution(FlowExecutorImpl.java:169) at org.springframework.webflow.mvc.servlet.FlowHandlerAdapter.handle(FlowHandlerAdapter.java:183) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.jasig.cas.web.init.SafeDispatcherServlet.service_aroundBody2(SafeDispatcherServlet.java:128) at org.jasig.cas.web.init.SafeDispatcherServlet.service_aroundBody3$advice(SafeDispatcherServlet.java:57) at org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:1) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at com.github.inspektr.common.web.ClientInfoThreadLocalFilter.doFilter(ClientInfoThreadLocalFilter.java:63) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:498) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: /10.10.137.107:11211 at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:160) at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:133) ... 123 more
2，故障分析
网上搜索了半天，之前压力测试的时候也没有报这样的错误，但是今天出现了。看错误信息,发现是get操作的时候报time out，有可能原因是大量数据并发的时候，有进程一直等待，或者就是并发数量不够
3，处理过程
通过ps -ef|grep memcached 查看memcached进程
app_use+ 12162 1 0 15:07 ? 00:00:00 ./memcached -d -u memcached -p 11211 -m 1024 -c 1024
发现并发连接数是1024是默认值
故调大这个参数为2048试试，结果不再报类似错误了。
4，后续措施
针对此问题，可以调大并发连接数，或者可以考虑使用集群，增加memcached的可用性
相关推荐
1. **监控**：收集memcached实例的性能指标，如命中率、内存使用情况、连接数等，帮助运维人员实时了解集群状态。 2. **动态调整**：允许在运行时添加、删除或更新memcached实例，无需停机或重启服务。 3. **故障...
4. **监控和分析**：`memadmin`提供的监控功能可以帮助用户了解`memcached`的运行状况，包括但不限于查看当前连接数、内存使用情况、命令统计等，以便于调整缓存策略和优化系统性能。 5. **安装与配置**：`memadmin...
12. **`Memcache::getStats()`** - 获取服务器的一些基本统计信息，如命中率、连接数等。 13. **`Memcache::getVersion()`** - 获取当前连接的Memcached服务器版本信息。 14. **`Memcache::increment()`** - 增加...
- `-c`：设定最大打开连接数，默认为1024。根据预期的并发连接数进行调整。 - `-v`：输出警告和错误信息，方便监控和排错。 - `-vv`：打印详细的客户端请求和响应信息，有助于调试。 - `-h`：显示帮助信息，列出所有...
- `-c`：最大运行的并发连接数，默认为 1024 - `-P`：设置保存 Memcached 的 pid 文件 #### 五、停止 Memcached 停止 Memcached 服务： ```bash kill `cat /tmp/memcached.pid` ``` 以上就是 Linux 环境下 ...
2. **监控**：可以使用`stats`命令查看memcached的状态，包括命中率、内存使用情况、连接数等。还可以借助第三方工具如`memcached-top`或监控系统集成进行实时监控。 **六、应用场景** 1. **Web应用缓存**：减少对...
- `-c` 设定最大并发连接数，默认为1024。 - `-f` 块大小增长因子，默认为1.25。 - `-n` 最小分配空间，包括key、value和flags，默认为48。 - `-h` 显示帮助信息。 通过理解并运用这些参数，你可以根据实际需求调整...
2. **性能优化**: 调整内存分配、限制并发连接数、合理设置缓存策略等都可以提升Memcached的性能。 3. **安全性**: 虽然Memcached默认仅监听本地，但在生产环境中，应确保防火墙配置正确，防止未授权访问。考虑使用...
如果将错误的版本与PHP环境匹配，可能会导致无法加载扩展或者运行异常。 安装PHP_Memcache扩展时，必须确保与您的PHP版本和操作系统位数相匹配。例如，如果你的PHP是5.6版本并且运行在Windows 64位系统上，你需要...
- 配置Memcached服务器，包括设置监听端口、最大内存使用量、最大连接数等。 - 在PHP代码中，使用`new Memcache()`创建一个实例，然后通过`connect()`或`pconnect()`方法连接到Memcached服务器。 - 使用`set()`、...
饱和度涉及资源利用率，如CPU、内存和连接数的使用情况。 另外，RED指标体系（请求量、错误、响应时间）是从四个黄金信号中简化出的一种通用监控框架。请求量（Request rate）跟踪单位时间内的请求次数，错误...
同时，滴滴建立了基于nagios的监控系统，对CPU、内存、I/O、网卡带宽、进程存活、Nginx流量、HTTP状态码、Fast-CGI（php-fpm）、MySQL连接数、主从延迟、Memcached连接数和QPS、MongoDB连接数和QPS等关键指标进行...
5. **调整Web服务器配置**：优化Apache或Nginx的配置，例如开启Gzip压缩，调整连接数限制等。 6. **代码审查**：审查并修复可能引起性能下降的代码，或者禁用不必要的插件。 7. **负载均衡**：如果局域网内用户...
- **开源工具**：如`mcmon`、`memcached-top`、`mcacheb`等，它们提供了丰富的统计信息，包括命中率、缓存大小、内存使用、连接数等。 - **图形化界面工具**：如`Cacti`、`Zabbix`、`Ganglia`等，它们能以图表形式...
由于网站的流量巨大，连接到所有数据库变得不可能，代码库庞大导致多方面的资源消耗，包括内存消耗和连接数限制。此外，缓存检索是大多数页面CPU使用率的约10%。缓存的好处显而易见，它能帮助减少对数据库的直接访问...
当连接数达到`max_connections`限制时，新的连接请求会被拒绝。临时解决方案是增大`max_connections`，但这可能导致服务器资源过度消耗，加剧问题。更优的做法是尽量复用已建立的连接，使用连接池来管理连接，减少...
- **连接管理**：合理设置最大连接数，避免过多连接导致资源耗尽。 - **缓存策略**：考虑使用缓存工具如Varnish或Memcached来减少数据库压力。 4. **性能基准测试**： - 使用基准测试工具（如`pgbench`）模拟...