`
netcome
  • 浏览: 482084 次
  • 性别: Icon_minigender_1
  • 来自: 北京
社区版块
存档分类
最新评论

Memory usage analysis

阅读更多

Memory usage analysis

System-wide memory analysis

Top/Free

These two command-line tools are the two most well-known and (especially with the default columns shown) very useless and misleading.

Free

In free the used swap shows the amount of used swap and used -/+ buffers/cache is supposed to show the amount of really used RAM. The latter is computed as all memory used minus various caches and buffers, however since sizes reported for caches and buffers don't reflect reality very well (they include actually really used memory), the numbers are useless for anything but getting a very rough picture.

Top

The totals in top are the same case like with free. The values reported for processes that are shown by default are not of much use either - their meaning is unclear or they are of no practical use. It is possible to change the columns shown using the f key to get some more useful values. However top is still only useful for rough analysis as it doesn't provide any details and does not account at all for sharing memory between processes.

Explanation of some of the columns (some of which may be wrong though because of their confusing meaning and values):

  • VIRT - the size of allocated address space. Not memory, but address space. This value in practice means next to nothing. When a process requests a large memory block from the system but uses only a small part of it, real usage will be low, VIRT will be high.
  • RES - resident memory size, i.e. the memory that the process uses in RAM. While this value has some practical value it is not that useful because it includes all memory this process possibly shares with other processes (e.g. KDE applications share large KDE libraries and the size of these libraries is includes in RES of every such application). Swap is not included which additionally decreases the usefulness of this value.
  • SHR - the amount of memory that could be potentially shared with other processes. The meaning is somewhat unclear and this value is of little practical value and it is not useful on its own.
  • SWAP - the amount of swapped out memory - not only memory moved to the swap but also for example memory-mapped files, which again makes this value not very useful.
  • CODE - memory taken by executable code. Various tables used by the code (e.g. virtual tables) don't seem to be included though - not very useful.
  • DATA - it clearly does not mean what the manual page says and the actual meaning of the value is unclear.

See the manual page for top for more details. Note however that the manual page (as of procps version 2.3.7 and Linux version 2.6.18) is either obsolete or appears to be wrong in description of some of the fields (e.g. DATA).

/proc

Various files in the /proc filesystem can provide information about memory. However they're are poorly documented (linux/Documentation/filesystems/proc.txt) and many of them are broken (and since top and free get their values the same way, they are equally broken). In /proc/PID/ there are several files:

  • status - human-readable data about the process, includes some data shown by top
  • stat/statm - similar, not in human-readable format
  • maps - lists memory mappings of the process (TODO if somebody feels like documenting, feel free to do so)
  • smaps - lists memory mappings of the process with more details (TODO if somebody feels like documenting, feel free to do so)

 

Exmap

Exmap (http://www.berthels.co.uk/exmap/) is probably the best currently available tool for system-wide memory analysis. Advantages include detailed information about memory, accounting for shared memory and, last but not least, reporting values that make sense. Since exmap is a relatively new tool it's not in very wide-spread use, so you will possibly have to download and build it yourself (it requires gtkmm and boost libraries). At http://ktown.kde.org/~seli/download/exmap/ is my lousy attempt at SUSE packages (unsupported).

Exmap has documentation that explains its use and the meaning of the various values. In short:

  • Run exmap. It may be better to run exmap as root in order to get access to all memory.
  • The 'Processes' tab shows memory information about each process in the first listview. The values are:
    • VM - VIRT from top - not really useful.
    • Mapped size - the total of memory actually used by the process, both in RAM and swap. Note however that e.g. libraries are not swapped out to swap but simply discarded (and read again from the files if needed). This value includes even shared memory.
    • Resident size - mapped size, but only in RAM (without swap).
    • Sole mapped - memory used only by this process. For example if a process uses a shared library that no other process uses at the moment it is included here.
    • Writable - memory with the processes' private data. It is part of sole mapped that the process has already written to.
    • Effective - the effective values are an attempt to compute how much memory a process uses in practice. Effective mapped/resident are mapped/resident values adjusted for shared memory. Memory that is shared by more processes is equally divided among them, i.e. if 10 processes use a 10MB large shared library (and each of them really uses the whole library), the library is counted as 10MB in mapped/resident values but only as 1MB in effective values.
  • The second listview shows memory mappings for the selected process. There are memory-mapped files and binaries and also several special mappings:
    • [heap] - dynamically allocated memory (i.e. malloc etc.)
    • [stack] - the stack of the process
    • [anon] - anonymous mapping from the mmap() system call - they should usually be viewed the same like [heap] although with multithreaded applications some [anon] mappings may be another [stack] mappings
  • The remaining two listviews provide details about the mappings. For experts (see exmap documentation).

Exmap in practice: Run exmap (possibly as root, if you get messages about failures to open files and you need them). Sort processes by effective mapped size. Higher values are worse. Use the second listview to find out which file is possibly responsible or if the high memory usage comes from the process data ([heap], [anon] and [stack]). It usually cannot be changed which libraries are used, so the values that should be actually checked are writable and sole mapped columns - they are the memory actually used by the application itself (although a portion may come from calling library functions of course).

X resources analysis

Applications using the X window system allocate some resources (such as pixmaps and windows) in the X server process and refer to them only using their handles (ID numbers). The memory for these resources is allocated in the X server process. The xrestop tool shows resource usage of applications in a way similar to top (without being seriously broken). The two most important columns are 'Pxms mem', which is memory taken by pixmaps (QPixmap), and 'Other', which is memory taken by other resources (should be usually low, high value may indicate a leak in code using directly Xlib, possibly in some library). TODO: there should be a tool to help detect pixmap leaks

Application memory analysis

Intro

Applications use memory in several ways that can be seen in exmap. There is memory taken by binaries, there are data segments from binaries (such as global variables but also e.g. relocation tables) and there is dynamic memory used. Dynamic memory is stack and mainly heap. Heap is part of memory for allocations using malloc(), operator new and similar functions.

There are two ways memory is allocated in the heap, as far as memory usage is concerned. There is one main heap area (shown as [heap] in exmap) that has one end fixed and other end is moved using the brk() system call. So when an allocation needs to be done a part of this area is reserved (recorded in heap internal structures) and its address is returned. If there's no more contiguous free space available the area is enlarged by moving the upper end using the brk() call. This has several consequences:

  • Every allocation has a certain overhead - when allocation many very small blocks the overhead caused by heap internal structures can significantly increase the actual memory usage. Various additional implementations of memory allocation like pools, arenas or obstacks can be used to reduce this problem (they use a special area of memory and the whole area must be freed at once, so there's no need to keep track of each allocation).
  • Memory can get easily fragmented - allocating blocks of different sizes can lead to heap having holes that are not big enough for following allocations. Heap implementations try to group small sizes together to reduce this problem and again additional memory allocation implementations like obstacks can reduce this problem as well (they use special area of memory that is freed at once so it cannot fragment the heap).
  • The main heap area is one contiguous memory area that cannot be split into smaller parts - when a lot of memory is allocated, then one permanent allocation is made and previous allocations are freed, the permanent allocation keeps the movable end fixed and prevents shrinking (TODO: does malloc actually ever use brk() to shrink?). This problem again can be reduced by using a special allocation implementation with its own memory area. Heap implementations also try to reduce this problem by allocating separate memory for large allocations (usually 128KB and larger) using mmap() system call instead of brk() - these show as [anon] in exmap.
Note that using additional memory allocation implementations is usually difficult with Qt/KDE as the developer cannot control how classes like QString, QList etc. allocate memory for their data.

 

Memprof

Memprof (http://www.gnome.org/projects/memprof/) tracks malloc() calls and can therefore be used for analysing heap usage. Application needs to be launched using memprof (pay attention avoid forking using --nofork or similar) and they should have debug information available to get full details. Top bar shows memory usage (yellow - used, blue - peak usage, red - leaks, after pressing the Leaks), number of allocations (at the moment, does not include already freed memory) and bytes allocated (heap overhead is not included). After pressing the Profile button it can be examined where the allocations come from.

Memprof in practice: Compile application with debug info, run it using memprof. Press the Profile button when you want memory analysis. In the listbox select the item with the maximum total (should be __libc_start_main or main), in the listview on the right there will be a tree of allocations from this function and the functions called from it. Open the tree, follow the high values, find problems.

kmtrace

Kmtrace (from kdesdk/kmtrace in KDE SVN) has similar purpose to memprof but with different usage and different way of presenting results (the output format is text file and the usage needs more manual intervention).

Installation (e.g. to /tmp/kmtrace):

svn co svn://anonsvn.kde.org/home/kde/branches/KDE/3.5/kdesdk -N
cd kdesdk
svn co svn://anonsvn.kde.org/home/kde/branches/KDE/3.5/kdesdk/kmtrace
svn co svn://anonsvn.kde.org/home/kde/branches/KDE/3.5/kde-common/admin
make -f Makefile.cvs
./configure --prefix=/tmp/kmtrace
make && make install

 

Usage:

LD_PRELOAD=/tmp/kmtrace/lib/libktrace.so MALLOC_TRACE=kmtrace.out xterm
/tmp/kmtrace/bin/kmtrace kmtrace.out --tree kmtrace.tree >kmtrace.txt
Starting and finishing of tracking is done using functions ktrace() and kuntrace() from libktrace.so, so it is possible to finish tracking even before the application exits by calling kuntrace() manually (TODO kmtrace could be perhaps hacked to react on a signal or something):
gdb
attach [pid]
call kutrace()
quit

File kmtrace.out is an internal file that needs to be post-processed and can be deleted afterwards. File kmtrace.txtcontains summary information and all allocations sorted by size - they're actually referred to as leaks, since tracking usually is finished after program exit, but when stopping tracking while the application is still running they represent all currently existing allocations. File kmtrace.tree provides all the allocations in a tree (first number is number of bytes, second is number of allocations). It is possible to limit the depth of the tree and to ignore subtrees with small sizes, see 'kmtrace --help'.

Valgrind

Valgrind's skin Massif ('valgrind --tool=massif application') traces memory usage of the application. After it exits it generates a .ps file showing allocations progress for main places and a text file with details (TODO: the text file does not seem very usable).

Valgrind can also detect memory leaks ('valgrind --tool=memcheck --leak-check=yes application', additionally also '--show-reachable=yes' may be used to show memory still allocated at application exit even if it's not unreachable). After application exit valgrind will print out backtrace of all leaked allocations (use '--num-callers=50' to get deeper backtraces).

HOWTO (AKA right to the point)

If you want to analyse memory usage of the whole system or to thoroughly analyse memory usage of one application (not just its heap usage), use exmap. For whole system analysis, find processes with the highest effective usage, they take the most memory in practice, find processes with the highest writable usage, they create the most data (and therefore possibly leak or are very ineffective in their data usage). Select such application and analyse its mappings in the second listview. See exmap section for more details. Also use xrestop to check high usage of X resources, especially if the process of the X server takes a lot of memory. See xrestop section for details.

If you want to detect leaks, use valgrind or possibly kmtrace (TODO memprof doesn't work for leaks for me). See their sections for more details.

If you want to analyse heap (malloc etc.) usage of an application, either run it in memprof or with kmtrace, profile the application and search the function call tree for biggest allocations. See their sections for more details.

 

http://ktown.kde.org/~seli/memory/analysis.html

分享到:
评论

相关推荐

    Learning Linux Binary Analysis(PACKT,2016)

    Finally, you will be walked through complex userspace memory infection analysis. This book will lead you into territory that is uncharted even by some experts; right into the world of the computer ...

    A Compressed Suffix Tree Based Implementation with Low Peak Memory Usage (2014)-计算机科学

    A Compressed Suffix Tree Based Implementation With Low Peak MemoryUsage 1Daniel Saad Nogueira Nunes2 Mauricio Ayala-Rincón3Instituto de Ciências Exatas Departamentos de Ciência da Computação...

    Learning.Linux.Binary.Analysis.1782

    Finally, you will be walked through complex userspace memory infection analysis. This book will lead you into territory that is uncharted even by some experts; right into the world of the computer ...

    学习linux二进制分析英文版

    Finally, you will be walked through complex userspace memory infection analysis., This book will lead you into territory that is uncharted even by some experts; right into the world of the computer ...

    Object-Oriented Analysis and Design 第六章

    - Flyweight pattern reduces memory usage by sharing common objects in a large number of similar objects. - Proxy pattern provides a surrogate or placeholder for another object to control access, ...

    微软内部资料-SQL性能优化2

     Describe how configuration options affect memory usage.  Describe the effect on the I/O subsystem when memory runs low.  List at least two memory myths and why they are not true. Recommended ...

    JProfiler Helper

    It provides developers with a detailed analysis of their Java programs' performance, enabling them to identify bottlenecks, optimize memory usage, and enhance overall application efficiency....

    graph-tool-2.27.tar.bz2

    Graph-tool is an efficient Python module for manipulation and ... This confers it a level of performance that is comparable (both in memory usage and computation time) to that of a pure C/C++ library.

    ICC8051 Micro Series 8051 C-Compiler V4.10A/DOS

    Usage: icc8051 {<options>} <sourcefile> {<options>} Sourcefile: 'C' source file with default extension: .c Environment: QCC8051 Options (specified order is of no importance): -o file Put object on: ...

    通过Key前缀分析Redis的内存占用按内存大小排序导出结果到csv文件

    此外,还可以考虑在分析过程中使用内存分析工具,如Redis的`MEMORY USAGE`命令,以更精确地获取单个键的内存占用。 总结来说,通过Key前缀分析Redis内存占用并导出结果到CSV文件,是一个涉及Redis内存管理、PHP编程...

    Algorithms in C++, Parts 1-4: Fundamentals, Data Structures, Sorting, Searching

    4. **Big-Oh Notation**: Big-Oh notation is a mathematical notation used to describe the upper bound of an algorithm’s running time or space usage. It provides a way to express the worst-case scenario...

    Writing High-Performance NET Code(2nd).pdf 2018新版

    More tool coverage, including expanded usage of Visual Studio More benchmarking New GC configuration options Code warmup techniques New .NET features such as ref-returns, value tuples, SIMD, and more ...

    Packt.Mastering.Csharp.and.NET.Programming

    - **A Quick Tip on the Execution and Memory Analysis of an Assembly in Visual Studio 2015**: Using tools like the Visual Studio Profiler to analyze the memory usage and performance of .NET ...

    lucene-3.3.0-src.zip

    - **内存使用优化(Memory Usage)**: 减少了内存占用,提升了系统在大规模索引时的稳定性。 4. **示例应用** `demo`目录下的代码展示了如何使用Lucene进行基本的索引创建、搜索和分析操作。开发者可以通过这些...

    Xcode.5.Start.to.Finish.iOS.and.OS.X.Development

    Analyzing processor and memory usage with Instruments Integrating with Mavericks Server’s sleek continuous integration system Register your book at www.informit.com/register for access to this title...

    SAP性能测试报告及分析---Part3

    - **Memory Usage Statistics**:跟踪每个事务执行时所使用的内存资源。 - **RFC Profiles**:分析远程函数调用(RFC)产生的系统负载,按事务、显示模式或目标端口分类。 - **User Profile**:显示每个用户执行...

    TMS320F28335

    ###### 3.2.4 Real-Time JTAG and Analysis - **Description**: Provides debugging capabilities through a JTAG interface. - **Functionality**: Enables developers to monitor and control the execution of ...

Global site tag (gtag.js) - Google Analytics