`
mryufeng
  • 浏览: 981751 次
  • 性别: Icon_minigender_1
  • 来自: 广州
社区版块
存档分类
最新评论

Memory matters - even in Erlang (再次说明了了解内存如何工作的必要性)

阅读更多
原文地址:http://www.lshift.net/blog/2010/02/28/memory-matters-even-in-erlang

作者解决问题的思路非常敬佩! 真没想到hibernation后, 由于对象的移动, 使得内存访问的不连续, 导致内存cahche的失效, 速度可以慢这么多!


Some time ago we got an interesting bug report for RabbitMQ. Surprisingly, unlike other complex bugs, this one is easy to describe:

At some point basic.get suddenly starts being very slow - about 9 times slower!

Basic.get doesn’t do anything complex - it just pops a message from a queue. This behaviour was quite unexpected. Our initial tests confirmed that we have a problem when a queue contains thousands of elements:

queue_length: 90001  basic_get 3333 times took: 1421.250ms
queue_length: 83335  basic_get 3333 times took: 1576.664ms
queue_length: 60004  basic_get 3333 times took: 1403.086ms
queue_length: 53338  basic_get 3333 times took: 9659.434ms [ look at that! ]
queue_length: 50005  basic_get 3333 times took: 9885.598ms
queue_length: 46672  basic_get 3333 times took: 8562.136ms

Let me repeat that. Usually popping a message from a queue takes Xms. At some point, it slows down to 9*Xms.

It turned out that the problem is with the queue:len() function, which is executed during the basic.get. Actually, queue:len() calls only erlang:length() builtin. At some point it switches to the “slow” mode.

Erlang:length() is a builtin that iterates through a linked list and counts it’s length. It’s complexity is O(N), where N is the length of the list. This function is implemented in the VM so it’s expected to be very, very fast.

The problem is not with erlang:length() being slow. It’s about being unpredictably slow. Let’s take a look at Erlang interpreter source code (erl_bif_guard.c:erts_gc_length_1). Here’s the main loop for erlang:length():

i=0
while (is_list(list)) {
    i++;
    list = CDR(list_val(list));
}

It does nothing unusual - it just iterates through list elements. However, recompiling Erlang with some debugging information confirms that the problem is indeed here:

clock_gettime(CLOCK_REALTIME, &t0);
while (is_list(list)) {
    i++;
    list = CDR(list_val(list));
}
clock_gettime(CLOCK_REALTIME, &t1);
td_ms = TIMESPEC_NSEC_SUBTRACT(t1, t0) / 1000000.0;
if (i > 200000 || td_ms > 2.0) {
    fprintf(stderr, "gc_length_1(%p)=%i %.3fms\n\r", reg[live], i, td_ms);
}

gc_length_1(0x7f4dbfa7fc19)=499999 2.221ms
gc_length_1(0x7f4dbfa7fc19)=499999 2.197ms
gc_length_1(0x7f4dbfa7fc19)=499999 2.208ms
(hibernation)
gc_length_1(0x7f4db0572049)=499999 13.793ms
gc_length_1(0x7f4db0572049)=499999 12.806ms
gc_length_1(0x7f4db0572049)=499999 12.531ms

This confirms Matthias’ initial guess - the slowdown starts after Erlang process hibernation.

For those who aren’t Erlang experts: Hibernation is an operation that compacts an Erlang process. It does aggressive garbage collection and reduces the memory footprint of a process to absolute minimum.

The intended result of hibernation is recovering free memory from the process. However its side effect is a new memory layout of objects allocated on the heap.

Ah, how could I have forgotten! The memory is nowadays slow! What happens, is that before hibernation list elements are aligned differently, more dense. Whereas after hibernation they are sparse. It’s easy to test it - let’s count the average distance between pointers to list elements:

gc_length_1(0x7f5c626fbc19)=499999 2.229ms avg=16.000 dev=0.023
gc_length_1(0x7f5c626fbc19)=499999 3.349ms avg=16.000 dev=0.023
gc_length_1(0x7f5c626fbc19)=499999 3.345ms avg=16.000 dev=0.023
(hibernation)
gc_length_1(0x7f5c61f7d049)=499999 13.800ms avg=136.000 dev=0.266
gc_length_1(0x7f5c61f7d049)=499999 12.726ms avg=136.000 dev=0.266
gc_length_1(0x7f5c61f7d049)=499999 12.367ms avg=136.000 dev=0.266

Confirmed! Standard deviation is surprisingly small, so we can read the numbers as:

    * Before hibernation list elements are aligned exactly one after another, values are somewhere else.
    * After hibernation list elements are interleaved with values.

This behavior does make sense. In most cases when you traverse the list, you actually do something with the values. After hibernation, when you access list item, the value will be already loaded to the CPU cache.

Knowing the mechanism, it’s easy to write a test case that reproduces the problem.

The average distance between pointers in my case is constant - the standard deviation is negligible. This information has a practical implication - we can “predict” where the next pointer will be. Let’s use that information to “fix” the Erlang VM by prefetching memory!

while (is_list(list)) {
    i++;
    list2 = CDR(list_val(list));
    __builtin_prefetch((char*)list2 + 128*((long)list2-(long)list));
    list = list2;
}

Test script running on original Erlang VM:

length: 300001  avg:0.888792ms dev:0.061587ms
length: 300001  avg:0.881030ms dev:0.040961ms
length: 300001  avg:0.875158ms dev:0.019436ms
hibernate
length: 300001  avg:14.861762ms dev:0.150635ms
length: 300001  avg:14.833733ms dev:0.017405ms
length: 300001  avg:14.884861ms dev:0.220119ms

Patched Erlang VM:

length: 300001  avg:0.742822ms dev:0.029322ms
length: 300001  avg:0.739149ms dev:0.012897ms
length: 300001  avg:0.739465ms dev:0.014417ms
hibernate
length: 300001  avg:7.543693ms dev:0.284355ms
length: 300001  avg:7.342802ms dev:0.330158ms
length: 300001  avg:7.265960ms dev:0.053176ms

The test runs only a tiny bit faster for the “fast” case (dense conses) and twice as fast for the “slow” case (sparse conses).

Should this patch be merged into mainline Erlang? Not really. I have set the prefetch multiplier value to 128 and I don’t even know if it’s optimal. This was only an experiment. But it was fun to see how low-level system architecture can affect high-level applications.

分享到:
评论
5 楼 pizigou 2010-04-20  
hack 需要达到这个级别 太深入了。。
4 楼 litaocheng 2010-03-11  
mryufeng 写道
litaocheng 写道
呵呵,不了解内部结构,遇到问题,只能大呼诡异啊。。
不过rabbitmq,为什么不用一个变量保存queue的长度啊,每次reply的时候都调用queue:len/1 确实有点不必要。O(N)的复杂度啊..

reply({ok, queue:len(BufferTail), Msg},
State#q{message_buffer = BufferTail,
next_msg_id = NextId + 1});

和这个命令的调用频度有关?


它这个basic.get命令调用应该很频繁哦。应答会返回对应的msg和剩余msg的数量。
3 楼 mryufeng 2010-03-10  
litaocheng 写道
呵呵,不了解内部结构,遇到问题,只能大呼诡异啊。。
不过rabbitmq,为什么不用一个变量保存queue的长度啊,每次reply的时候都调用queue:len/1 确实有点不必要。O(N)的复杂度啊..

reply({ok, queue:len(BufferTail), Msg},
State#q{message_buffer = BufferTail,
next_msg_id = NextId + 1});

和这个命令的调用频度有关?
2 楼 litaocheng 2010-03-10  
呵呵,不了解内部结构,遇到问题,只能大呼诡异啊。。
不过rabbitmq,为什么不用一个变量保存queue的长度啊,每次reply的时候都调用queue:len/1 确实有点不必要。O(N)的复杂度啊..

reply({ok, queue:len(BufferTail), Msg},
State#q{message_buffer = BufferTail,
next_msg_id = NextId + 1});
1 楼 iso1600 2010-03-10  
除了erlang:length() 其他地方应该也有类似现象,patch这个地方治标不治本。
是不是hibernate本身优化解决比较好。

相关推荐

    Pseudo-Mask Matters in Weakly-Supervised Semantic.pdf

    【伪掩模在弱监督语义分割中的重要性】\n\n在计算机视觉领域,语义分割是一项基础任务,它需要耗时的像素级手动注释。为了减轻标注负担,弱监督语义分割方法应运而生,这些方法利用了如涂鸦注释、边界框、点或图像级...

    吉他谱_Nothing Else Matters - Metallica.pdf

    初级入门吉他谱 guitar tab

    Matters Computational-ideas, algorithms, source code

    ### Matters Computational: Key Insights and Algorithms #### Overview "Matters Computational: Ideas, Algorithms, Source Code" is a comprehensive resource aimed at computationalists and programmers ...

    Data Transfer Matters for GPU Computing

    <Data Transfer Matters for GPU Computing> Abstract—Graphics processing units (GPUs) embrace manycore compute devices where massively parallel compute threads are offloaded from CPUs. This ...

    matters-rest-spring-jersey-tomcat-mybatis:事项休息春天泽西岛tomcat mybatis

    标题 "matters-rest-spring-jersey-tomcat-mybatis" 暗示这是一个关于构建RESTful服务的项目,使用了Spring、Jersey、Tomcat和MyBatis这些技术。描述中的“事项-休息-春天-球衣-tomcat-mybatis 1”可能是项目名的另...

    Matters Computational

    - **In-Place Methods to Apply Permutations to Data**: Techniques for applying permutations directly to data without additional memory. - **Random Permutations**: Generating random permutations ...

    Donetica - Your search matters-crx插件

    语言:Deutsch,English,Français,español,italiano Donetica-您的搜索很重要! 当Donation和Etica在一起时-Donetica。 通过搜索改变世界,搜索很重要! 仅需单击几下,Donetica就会捐赠给慈善,生态,健康与福利...

    react-native-size-matters:一个React-Native实用程序带,用于在不同大小的设备上缩放应用程序UI的大小

    :triangular_ruler: React本机大小问题 一个React-Native实用程序带,... 那是一件乏味的工作。 react-native-size-matters提供了一些简单的工具,使您的缩放变得更加容易。 这个想法是在标准的约5英寸屏幕移动设备

    Closed-loop Matters: DualRegression Networks for Single Image Super-Resolution

    loop Matters: Dual Regression Networks for Single Image Super-Resolution"一文深入探讨了超分辨率技术中的关键挑战,并提出了一种创新的双回归网络架构,旨在解决从LR到HR图像转换中的病态问题和真实世界数据...

    Docker-in-Action.pdf

    To make matters worse, I had altered my environment in a way that was incompatible with other software that I use regularly. Reverting those changes took an embarrassingly long time. I distinctly ...

    Every Search Matters-crx插件

    语言:English 有机会改变世界,成为支持慈善机构的大社区的一部分。 我们的主要目标是为想要改变世界的人们提供更好的机会来实现这一目标。 当然,您可以通过直接向想要支持的慈善组织捐款来做到这一点,但是如果您...

    Infrastructure_Matters_POWER8_vs_XEON_x86-IBM官网引导的报告-中文

    ### 基础架构的重要性:POWER8与XEON x86对比分析 #### 概述 随着企业数字化转型的步伐不断加快,服务器基础架构的选择成为决定业务成功与否的关键因素之一。本报告聚焦于IBM的POWER8架构与Intel XEON x86架构之间...

    performance-matters-open-assignment:Tuts +公开作业的练习文件

    Tuts +开放式作业: 授课教师:Adi Purdila 这是我最近进行的有关优化网页性能的课程的一项公开作业。 查看课程,观看作业视频,下载工作文件,然后让我们知道您的学习情况! 可在Tuts + 2015年9月10日使用

    Matters Computational(pdf)

    - **矩阵转置**:在不使用额外内存的情况下,实现矩阵的转置操作。 - **三重反转旋转**:通过三次反转操作来实现数组或列表的旋转。 - **XOR 排列**:一种基于异或操作的特殊排列。 - **格雷码排列**:基于格雷...

    performance-matters-checklist:@ cmda-minor-web的性能检查表·2017-2018

    绩效检查表 目录 图片 CSS ... JavaScript 字型 后端优化 各种各样的 提示:Chrome中没有扩展名的其他个人资料/用户 使用正确的lang ... 尝试避免重定向,这些重定向可能会带来较高的延迟开销 ...视觉进度的优化非常糟糕...

    Matters Computational ideas, algorithms, source code

    ### Matters Computational: Key Insights, Algorithms, and Source Code #### Overview "Matters Computational" is a comprehensive resource designed for computationalists, including professional ...

    Why Functional Programming Matters

    In this paper we show that two fea- tures of functional languages in particular, higher-order functions and lazy evaluation, can contribute significantly to modularity. As examples, we manipulate ...

Global site tag (gtag.js) - Google Analytics