Linux Commands For Shared Library Management & Debugging Problem & strace

【声明】：本文内容来自网络，如有侵权请尽快通知本人（发表这篇文章的人肯定不懂中文，哈哈）。原文链接：http://www.cyberciti.biz/tips/linux-shared-library-management.html

Linux Commands For Shared Library Management & Debugging Problem

by VIVEK GITE onJANUARY 6, 2011 ·8 COMMENTS

If you are a developer, you will re-use code provided by others. Usually /lib, /lib64, /usr/local/lib, and other directories stores various shared libraries. You can write your own program using these shared libraries. As a sys admin you need to manage and install these shared libraries. Use the following commands for shared libraries management, security, and debugging problems.

What is a Library In Linux or UNIX?

In Linux or UNIX like operating system, a library is noting but a collection of resources such as subroutines / functions, classes, values or type specifications. There are two types of libraries:

1. Static libraries - All lib*.a fills are included into executables that use their functions. For example you can run asendmail binary in chrooted jail using statically liked libs.
2. Dynamic libraries or linking [ also known as DSO (dynamic shared object)] - All lib*.so* files are not copied into executables. The executable will automatically load the libraries using ld.so or ld-linux.so.

Linux Library Management Commands

1. ldconfig : Updates the necessary links for the run time link bindings.
2. ldd : Tells what libraries a given program needs to run.
3. ltrace : A library call tracer.
4. ld.so/ld-linux.so: Dynamic linker/loader.

Important Files

As a sys admin you should be aware of important files related to shared libraries:

1. /lib/ld-linux.so.* : Execution time linker/loader.
2. /etc/ld.so.conf : File containing a list of colon, space, tab, newline, or comma separated directories in which to search for libraries.
3. /etc/ld.so.cache : File containing an ordered list of libraries found in the directories specified in /etc/ld.so.conf. This file is not in human readable format, and is not intended to be edited. This file is created by ldconfig command.
4. lib*.so.version : Shared libraries stores in /lib, /usr/lib, /usr/lib64, /lib64, /usr/local/lib directories.

#1: ldconfig command

You need to use theldconfig command to create, update, and remove the necessary links and cache (for use by the run-time linker, ld.so) to the most recent shared libraries found in the directories specified on the command line, in the file /etc/ld.so.conf, and in the trusted directories (/usr/lib, /lib64 and /lib). The ldconfig command checks the header and file names of the libraries it encounters when determining which versions should have their links updated. This command also creates a file called/etc/ld.so.cache which used to speed linking.

Examples

In this example, you've installed a new set of shared libraries at /usr/local/lib/:
$ ls -l /usr/local/lib/
Sample outputs:
-rw-r--r-- 1 root root 878738 Jun 16 2010 libGeoIP.a
-rwxr-xr-x 1 root root 799 Jun 16 2010 libGeoIP.la
lrwxrwxrwx 1 root root 17 Jun 16 2010 libGeoIP.so -> libGeoIP.so.1.4.6
lrwxrwxrwx 1 root root 17 Jun 16 2010 libGeoIP.so.1 -> libGeoIP.so.1.4.6
-rwxr-xr-x 1 root root 322776 Jun 16 2010 libGeoIP.so.1.4.6
-rw-r--r-- 1 root root 72172 Jun 16 2010 libGeoIPUpdate.a
-rwxr-xr-x 1 root root 872 Jun 16 2010 libGeoIPUpdate.la
lrwxrwxrwx 1 root root 23 Jun 16 2010 libGeoIPUpdate.so -> libGeoIPUpdate.so.0.0.0
lrwxrwxrwx 1 root root 23 Jun 16 2010 libGeoIPUpdate.so.0 -> libGeoIPUpdate.so.0.0.0
-rwxr-xr-x 1 root root 55003 Jun 16 2010 libGeoIPUpdate.so.0.0.0

Now when you run an app related to libGeoIP.so, you will get an error about missing library. You need to run ldconfig command manually to link libraries by passing them as command line arguments with the -l switch:
# ldconfig -l /path/to/lib/our.new.lib.so
Another recommended options for sys admin is to create a file called/etc/ld.so.conf.d/geoip.conf as follows:
/usr/local/lib
Now just run ldconfig to update the cache:
# ldconfig
To verify new libs or to look for a linked library, enter:
# ldconfig -v
# ldconfig -v | grep -i geoip
Sample outputs:
libGeoIP.so.1 -> libGeoIP.so.1.4.6
libGeoIPUpdate.so.0 -> libGeoIPUpdate.so.0.0.0

Troubleshooting Chrooted Jails

You can print the current cache with the -p option:
# ldconfig -p
Putting web server such as Apache / Nginx / Lighttpd in a chroot jail minimizes the damage done by a potential break-in by isolating the web server to a small section of the filesystem. It is also necessary to copy all files required by Apache inside the filesystem rooted at /jail/ directory , including web server binaries, shared Libraries, modules, configuration files, and php/perl/html web pages. You need to also copy /etc/{ld.so.cache,ld.so.conf} files and /etc/ld.so.conf.d/ directory to /jail/etc/ directory. Use the ldconfig command to update, print and troubleshoot chrooted jail problems:
### chroot to jail bash
chroot /jail /bin/bash
### now update the cache in /jail ###
ldconfig
### print the cache in /jail ###
ldconfig -p
### copy missing libs ###
cp /path/to/some.lib /jail/path/to/some.lib
ldconfig
ldconfig -v | grep some.lib
### get out of jail ###
exit
### may be delete bash and ldconfig to increase security (NOTE path carefully) ###
cd /jail
rm sbin/ldconfig bin/bash
### now start nginx jail ###
chroot /jail /usr/local/nginx/sbin/nginx

Rootkits

A rootkit is a program (or combination of several programs) designed to take fundamental control of a computer system, without authorization by the system's owners and legitimate managers. Usually, rootkit use /lib, /lib64, /usr/local/lib directories to hide itself from real root users. You can use ldconfig command to view all the cache of all shared libraries and unwanted programs:
# /sbin/ldconfig -p | less
You can also usevarious tools to detect rootkits under Linux.

Common errors

You may see the errors as follows:

Dynamic linker error in foo

Can't map cache file cache-file

Cache file cache-file foo

All of the above errors means the linker cache file /etc/ld.so.cache is corrupt or does not exists. To fix these errors simply run the ldconfig command as follows:
# ldconfig

Can't find library xyz Error

The executable required a dynamically linked library that ld.so or ld-linux.so cannot find. It means a library called xyz needed by the program called foo not installed or path is not set. To fix this problem install xyz library and set path in /etc/ld.so.conf file or create a file in /etc/ld.so.conf.d/ directory.

#2: ldd command

ldd (List Dynamic Dependencies) is a Unix and Linux program to display the shared libraries required by each program. This tools is required to build and run various server programs in a chroot jail. A typical example is as follows to list the Apache server shared libraries, enter:
# ldd /usr/sbin/httpd
Sample outputs:
libm.so.6 => /lib64/libm.so.6 (0x00002aff52a0c000)
libpcre.so.0 => /lib64/libpcre.so.0 (0x00002aff52c8f000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00002aff52eab000)
libaprutil-1.so.0 => /usr/lib64/libaprutil-1.so.0 (0x00002aff530c4000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002aff532de000)
libldap-2.3.so.0 => /usr/lib64/libldap-2.3.so.0 (0x00002aff53516000)
liblber-2.3.so.0 => /usr/lib64/liblber-2.3.so.0 (0x00002aff53751000)
libdb-4.3.so => /lib64/libdb-4.3.so (0x00002aff5395f000)
libexpat.so.0 => /lib64/libexpat.so.0 (0x00002aff53c55000)
libapr-1.so.0 => /usr/lib64/libapr-1.so.0 (0x00002aff53e78000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aff5409f000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aff542ba000)
libc.so.6 => /lib64/libc.so.6 (0x00002aff544bf000)
libsepol.so.1 => /lib64/libsepol.so.1 (0x00002aff54816000)
/lib64/ld-linux-x86-64.so.2 (0x00002aff527ef000)
libuuid.so.1 => /lib64/libuuid.so.1 (0x00002aff54a5c000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00002aff54c61000)
libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x00002aff54e76000)
libssl.so.6 => /lib64/libssl.so.6 (0x00002aff5508f000)
libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00002aff552dc000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00002aff5562d000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00002aff5585c000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00002aff55af1000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00002aff55cf3000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00002aff55f19000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00002aff5612d000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00002aff56335000)

Now, you can copy all those libs one by one to /jail directory
# mkdir /jail/lib
# cp /lib64/libm.so.6 /jail/lib
# cp /lib64/libkeyutils.so.1 /jail/lib
You can write a bash script to automate the entire procedure:
cp_support_shared_libs(){
locald="$1" # JAIL ROOT
localpFILE="$2" # copy bin file libs
localfiles=""
## use ldd to get shared libs list ###
files="$(ldd $pFILE | awk '{ print $3 }' | sed '/^$/d')"

for iin$files
do
dcc="${i%/*}"# get dirname only
[ ! -d${d}${dcc}] &&mkdir -p${d}${dcc}
${_cp} -f $i${d}${dcc}
done

# Works with 32 and 64 bit ld-linux
sldl="$(ldd $pFILE | grep 'ld-linux' | awk '{ print $1}')"
sldlsubdir="${sldl%/*}"
[ ! -f${d}${sldl}] &&${_cp} -f ${sldl}${d}${sldlsubdir}
}
Call cp_support_shared_libs() it as follows:
cp_support_shared_libs"/jail""/usr/local/nginx/sbin/nginx"

Report Missing Functions

Type the following command:
$ ldd -d /path/to/executable

Report Missing Objects

Type the following command:
$ ldd -r /path/to/executable

Determine If Particulate Feature Supported Or Not

TCP Wrapper is a host-based Networking ACL system, used to filter network access to Internet. TCP wrappers was original written to monitor and stop cracking activities on the UNIX / Linux systems. To determine whether a given executable daemon supports TCP Wrapper or not, run the following command:
$ ldd /usr/sbin/sshd | grep libwrap
Sample outputs:
libwrap.so.0 => /lib64/libwrap.so.0 (0x00002abd70cbc000)
The output indicates that the OpenSSH (sshd) daemon supports TCP Wrapper.

Other usage of ldd command

You can use the ldd command when an executable is failing because of a missing dependency. Once you found a missing dependency, you can install it or update the cache with the ldconfig command as mentioned above.

#3: ltrace command

Theltrace command simply runs the specified command until it exits. It intercepts and records the dynamic library calls which are called by the executed process and the signals which are received by that process. It can also intercept and print the system calls executed by the program. Its use is very similar to strace command.
# ltrace /usr/sbin/httpd
# ltrace /sbin/chroot /usr/sbin/httpd
# ltrace /bin/ls
Sample outputs:
__libc_start_main(0x804fae0, 1, 0xbfbd6544, 0x805bce0, 0x805bcd0
strrchr("/bin/ls", '/') = "/ls"
setlocale(6, "") = "en_IN.utf8"
bindtextdomain("coreutils", "/usr/share/locale") = "/usr/share/locale"
textdomain("coreutils") = "coreutils"
__cxa_atexit(0x8052d10, 0, 0, 0xbfbd6544, 0xbfbd6498) = 0
isatty(1) = 1
getenv("QUOTING_STYLE") = NULL
getenv("LS_BLOCK_SIZE") = NULL
getenv("BLOCK_SIZE") = NULL
getenv("BLOCKSIZE") = NULL
getenv("POSIXLY_CORRECT") = NULL
getenv("BLOCK_SIZE") = NULL
getenv("COLUMNS") = NULL
ioctl(1, 21523, 0xbfbd6470) = 0
getenv("TABSIZE") = NULL
getopt_long(1, 0xbfbd6544, "abcdfghiklmnopqrstuvw:xABCDFGHI:"..., 0x0805ea40, -1) = -1
__errno_location() = 0xb76b8694
malloc(40) = 0x08c8e3e0
memcpy(0x08c8e3e0, "", 40) = 0x08c8e3e0
....
....
.....
..
output truncated
free(0x08c8e498) =
free(NULL) =
free(0x08c8e480) =
exit(0
__fpending(0xb78334e0, 0xbfbd6334, 0xb78876a3, 0xb78968f8, 0) = 0
fclose(0xb78334e0) = 0
__fpending(0xb7833580, 0xbfbd6334, 0xb78876a3, 0xb78968f8, 0) = 0
fclose(0xb7833580) = 0
+++ exited (status 0) +++

The ltrace command is a perfect debugging utility in Linux:

1. To monitor the library calls used by a program and all the signals it receives.
2. For tracking the execution of processes.
3. It can also show system calls, used by a program.

ltrace Command Examples

Consider the following c program:

#include <stdio.h>
int main(){
printf("Hello world\n");
return0;
}

Compile and run it as follows:
$ cc hello.c -o hello
$ ./hello
Now use the ltrace command to tracking the execution of processes:
$ ltrace -S -tt ./hello
Sample outputs:
15:20:38.561616 SYS_brk(NULL) = 0x08f42000
15:20:38.561845 SYS_access("/etc/ld.so.nohwcap", 00) = -2
15:20:38.562009 SYS_mmap2(0, 8192, 3, 34, -1) = 0xb7708000
15:20:38.562155 SYS_access("/etc/ld.so.preload", 04) = -2
15:20:38.562336 SYS_open("/etc/ld.so.cache", 0, 00) = 3
15:20:38.562502 SYS_fstat64(3, 0xbfaafe20, 0xb7726ff4, 0xb772787c, 3) = 0
15:20:38.562629 SYS_mmap2(0, 76469, 1, 2, 3) = 0xb76f5000
15:20:38.562755 SYS_close(3) = 0
15:20:38.564204 SYS_access("/etc/ld.so.nohwcap", 00) = -2
15:20:38.564372 SYS_open("/lib/tls/i686/cmov/libc.so.6", 0, 00) = 3
15:20:38.564561 SYS_read(3, "\177ELF\001\001\001", 512) = 512
15:20:38.564694 SYS_fstat64(3, 0xbfaafe6c, 0xb7726ff4, 0xb7705796, 0x8048234) = 0
15:20:38.564822 SYS_mmap2(0, 0x1599a8, 5, 2050, 3) = 0xb759b000
15:20:38.565076 SYS_mprotect(0xb76ee000, 4096, 0) = 0
15:20:38.565209 SYS_mmap2(0xb76ef000, 12288, 3, 2066, 3) = 0xb76ef000
15:20:38.565454 SYS_mmap2(0xb76f2000, 10664, 3, 50, -1) = 0xb76f2000
15:20:38.565604 SYS_close(3) = 0
15:20:38.565709 SYS_mmap2(0, 4096, 3, 34, -1) = 0xb759a000
15:20:38.565842 SYS_set_thread_area(0xbfab030c, 0xb7726ff4, 0xb759a6c0, 1, 0) = 0
15:20:38.566070 SYS_mprotect(0xb76ef000, 8192, 1) = 0
15:20:38.566185 SYS_mprotect(0x08049000, 4096, 1) = 0
15:20:38.566288 SYS_mprotect(0xb7726000, 4096, 1) = 0
15:20:38.566381 SYS_munmap(0xb76f5000, 76469) = 0
15:20:38.566522 __libc_start_main(0x80483e4, 1, 0xbfab04e4, 0x8048410, 0x8048400
15:20:38.566667 puts("Hello world"
15:20:38.566811 SYS_fstat64(1, 0xbfab0310, 0xb76f0ff4, 0xb76f14e0, 0x80484c0) = 0
15:20:38.566936 SYS_mmap2(0, 4096, 3, 34, -1) = 0xb7707000
15:20:38.567126 SYS_write(1, "Hello world\n", 12Hello world
) = 12
15:20:38.567282 <... puts resumed> ) = 12
15:20:38.567348 SYS_exit_group(0
15:20:38.567454 +++ exited (status 0) +++

You need to carefully monitor the order and arguments of selected functions such as open() [used to open and possibly create a file or device] or chown() [used to change ownership of a file] so that you can spot simple kinds of race conditions or security related problems. This is quite useful for evaluating the security of binary programs to find out what kind of changes made to the system.

ltrace: Debugging Memory & I/O Usage For HA Based Cluster Computers

The ltrace command can be used to trace memory usage of the malloc() and free() functions in C program. You can calculate the amount of memory allocated as follows:
[node303 ~]$ ltrace -e malloc,free ./simulator arg1 agr2 arg3
The ltrace will start ./simulator program and it will trace the malloc() and free() functions. You can find out I/O problems as follows:
[node303 ~]$ ltrace -e fopen,fread,fwrite,fclose ./simulator arg1 agr2 arg3
You may need to change function names as your programming languages or UNIX platform may use different memory allocation functions.

#4: ld.so/ld-linux.so Command

The ld.so or / ld-linux.so used as follows by Linux:

1. To load the shared libraries needed by a program.
2. To prepare the program to run, and then runs it.

List All Dependencies and How They Are Resolved

Type the following command:
# cd /lib
For 64 bit systems:
# cd /lib64
Pass the --list option, enter:
# ./ld-2.5.so --list /path/to/executable

Other options

From the man page:
--verify verify that given object really is a dynamically linked object we can handle
--library-path PATH use given PATH instead of content of the environment variable LD_LIBRARY_PATH
--inhibit-rpath LIST ignore RUNPATH and RPATH information in object names in LIST

Environment Variables

TheLD_LIBRARY_PATH can be used to set a library path for finding dynamic libraries using LD_LIBRARY_PATH, in the standard colon seperated format:
$ export LD_LIBRARY_PATH=/opt/simulator/lib:/usr/local/lib
TheLD_PRELOAD allow anextra library not specified in the executable to be loaded:
$ export LD_PRELOAD=/home/vivek/dirhard/libdiehard.so
Please note that these variables are ignored when executing setuid/setgid programs.

Recommend readings:

• HowTo: Debug Crashed Linux Application Core Files Like A Pro
• Debugging Tip: Trace the Process and See What It is Doing with strace
• man pages ldconfig, ld.so, ldd, ltrace
• Dynamic Linking and Loading and Shared libraries
• How to create shared library under Linux with lots of information about internal structures
• Anatomy of Linux dynamic libraries - This article investigates the process of creating and using dynamic libraries, provides details on the various tools for exploring them, and explores how these libraries work under the hood.

Featured Articles:

• 20 Linux System Monitoring Tools Every SysAdmin Should Know
• 20 Linux Server Hardening Security Tips
• My 10 UNIX Command Line Mistakes
• The Novice Guide To Buying A Linux Laptop
• 10 Greatest Open Source Software Of 2009
• Top 5 Email Client For Linux, Mac OS X, and Windows Users
• Top 20 OpenSSH Server Best Security Practices
• Top 10 Open Source Web-Based Project Management Software
• Top 5 Linux Video Editor Software

---

Debugging Tip: Trace the Process and See What It is Doing with strace

by VIVEK GITE onAPRIL 10, 2008 ·3 COMMENTS

strace is a useful diagnostic, instructional, and debugging tool. It can save lots of headache. System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. This is also useful to submit bug reports to open source developers.
Each line in the trace contains the system call name, followed by its arguments in parentheses and its return value.
Run strace against /bin/foo and capture its output to a text file in output.txt:
$ strace -o output.txt /bin/foo
You can strace the webserver process and see what it's doing. For example, strace php5 fastcgi process, enter:
$ strace -p 22254 -s 80 -o /tmp/debug.lighttpd.txt
To see only a trace of the open, read system calls, enter :
$ strace -e trace=open,read -p 22254 -s 80 -o debug.webserver.txt
Where,

• -o filename : Write the trace output to the file filename rather than to screen (stderr).
• -p PID : Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (hit CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to up to 32 processes in addition to command (which is optional if at least one -p option is given).
• -s SIZE : Specify the maximum string size to print (the default is 32).

Refer to strace man page for more information:
$ man strace

---

HowTo: Debug Crashed Linux Application Core Files Like A Pro

by VIVEK GITE onJUNE 3, 2010 ·18 COMMENTS
Core dumps are often used to diagnose or debug errors in Linux or UNIX programs. Core dumps can serve as useful debugging aids for sys admins to find out why Application like Lighttpd, Apache, PHP-CGI or any other program crashed. Many vendors and open source project author requests a core file to troubleshoot a program. A core file is generated when an application program abnormally terminates due to bug, operating system security protection schema, or program simply try to write beyond the area of memory it has allocated, and so on. This article explains how to turn on core file support and track down bugs in programs.

Turn On Core File Creation Support

By default most Linux distributions turn off core file creation (at least this is true for RHEL, CentOS, Fedora and Suse Linux). You need to use the ulimit command to configure core files.

See The Current Core File Limits

Type the following command:
# ulimit -c
Sample outputs:
0
The output 0 (zero) means core file is not created.

Change Core File Limits

In this example, set the size limit of core files to 75000 bytes:
# ulimit -c 75000

HowTo: Enable Core File Dumps For Application Crashes And Segmentation Faults

Edit /etc/profile file and find line that read as follows to make persistent configuration:
ulimit -S -c 0 > /dev/null 2>&1
Update it as follows:
ulimit -c unlimited >/dev/null 2>&1
Save and close the file. Edit /etc/sysctl.conf, enter:
# vi /etc/sysctl.conf
Append the following lines:
kernel.core_uses_pid = 1
kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t
fs.suid_dumpable = 2
Save and close the file. Where,

1. kernel.core_uses_pid = 1 - Appends the coring processes PID to the core file name.
2. fs.suid_dumpable = 2 - Make sure you get core dumps for setuid programs.
3. kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t - When the application terminates abnormally, a core file should appear in the /tmp. The kernel.core_pattern sysctl controls exact location of core file. You can define the core file name with the following template whih can contain % specifiers which are substituted by the following values when a core file is created:
   • %% - A single % character
   • %p - PID of dumped process
   • %u - real UID of dumped process
   • %g - real GID of dumped process
   • %s - number of signal causing dump
   • %t - time of dump (seconds since 0:00h, 1 Jan 1970)
   • %h - hostname (same as ’nodename’ returned by uname(2))
   • %e - executable filename
   
   Finally, enable debugging for all apps, enter (Redhat and friends specific):
   # echo "DAEMON_COREFILE_LIMIT='unlimited'" >> /etc/sysconfig/init
   Reload the settings in /etc/sysctl.conf by running the following command:
   # sysctl -p
   

   How Do I Enable Core Dumping For Specific Deamon?

   To enable core dumping for specific deamons, add the following line in the /etc/sysconfig/daemon-file file. In this example, edit /etc/init.d/lighttped and add line as follows:
   DAEMON_COREFILE_LIMIT='unlimited'
   Please note that DAEMON_COREFILE_LIMIT is Redhat specific, for all other distro add configuration as follows:
   ulimit -c unlimited >/dev/null 2>&1
   echo /tmp/core-%e-%s-%u-%g-%p-%t > /proc/sys/kernel/core_pattern
   Save and close the file. Restart / reload lighttpd:
   # /etc/init.d/lighttpd restart
   # su - lighttpd
   $ ulimit -c
   Sample outputs:
   unlimited
   Now, you can send core files to vendor or software writes.
   

   How Do I Read Core Files?

   You need use the gdb command as follows:
   $ gdb /path/to/application /path/to/corefile
   See the gdb command man page for more information.
   

   strace command

   System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. This is also useful to submit bug reports to open source developers. See how to use thestrace command under Linux to debug the problems.
   

   Recommended readings:

   • Debugging Tip: Trace the Process and See What It is Doing with strace
   • The Art of Debugging with GDB, DDD, and Eclipse
   • man pages core(5), strace, and bash
   
   Stay tunned for gdb tutorial which will explains how to use generated core file to track down problem.