Linux troubleshooting tools

An app is running “slow” on Linux, how do you go about figuring out what could be wrong? Typical bottle-necks for a slow application are cpu, disk i/o, network, memory, or database. For instance if Apache is running slow it could be memory starved. Some of the tools we can use to debug application are listed below. Keep in mind that when using these tools, you generally want to take multiple snapshots in time, and not necessarily base your information on a single moment in time.

1) Vmstat shows you a number of interesting statistics. Some things to look for in vmstat include the proc ‘r’ column, which shows the number of processes waiting for run time. If this number if “high” it indicates CPU starvation. CPU starved apps run slow. To fix, increase the number of CPUs in the system. Another field to look at in the output of vmstat is the ‘cs’ field under the system heading. ‘cs’ stands for context switching. A “high” number means that the CPU is having to switch between a lot of tasks. These tasks might be network, disk, or cpu related. One way to solve this problem is to see if the server is doing “too many” things, i.e., running too many applications. Reduce the number of applications running to solve this problem.

$ vmstat -s
     16301872  total memory
      5718716  used memory
      3179672  active memory
      2228164  inactive memory
     10583156  free memory
       166264  buffer memory
      2913548  swap cache
      8191992  total swap
          572  used swap
      8191420  free swap
     26013290 non-nice user cpu ticks
          881 nice user cpu ticks
     17260265 system cpu ticks
   7326487040 idle cpu ticks
     13013070 IO-wait cpu ticks
           72 IRQ cpu ticks
         8054 softirq cpu ticks
            0 stolen cpu ticks
     11212676 pages paged in
    180145305 pages paged out
           44 pages swapped in
          145 pages swapped out
   4176361125 interrupts
   1524882692 CPU context switches
   1367818576 boot time
      9070041 forks

2) mpstat: To view processor utilization based on per core.

$ mpstat -P ALL
Linux 2.6.32-358.6.1.el6.x86_64 (hostname) 	08/20/2013 	_x86_64_	(8 CPU)

10:48:05 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:48:05 PM  all    0.03    0.00    0.23    0.18    0.00    0.00    0.00    0.32   99.24
10:48:05 PM    0    0.05    0.00    0.42    0.72    0.00    0.00    0.00    0.69   98.13
10:48:05 PM    1    0.05    0.00    1.01    0.34    0.00    0.00    0.00    1.78   96.82
10:48:05 PM    2    0.04    0.00    0.11    0.04    0.00    0.00    0.00    0.03   99.78
10:48:05 PM    3    0.03    0.00    0.11    0.18    0.00    0.00    0.00    0.03   99.66
10:48:05 PM    4    0.02    0.00    0.07    0.01    0.00    0.00    0.00    0.02   99.87
10:48:05 PM    5    0.02    0.00    0.08    0.12    0.00    0.00    0.00    0.02   99.77
10:48:05 PM    6    0.01    0.00    0.04    0.00    0.00    0.00    0.00    0.01   99.93
10:48:05 PM    7    0.01    0.00    0.04    0.01    0.00    0.00    0.00    0.01   99.93

3) iostat : If you need help in figuring out how your disk subsystem is performing, use iostat. “The iostat command generates three types of reports, the CPU Utilization report, the Device Utilization report and the Network Filesystem report.” (From the man page of iostat) Items of concern here would be iowait%, which if high indicates that the CPU is waiting for disk I/O to complete. If %util hits 100% then device saturation occurs.

4) netstat: This is an extremely useful command to diagnose numerous network issues.
To view routing table use:

$ netstat -r
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
216.218.223.240 *               255.255.255.240 U         0 0          0 br0
216.218.223.240 *               255.255.255.240 U         0 0          0 em1
link-local      *               255.255.0.0     U         0 0          0 br0
default         uplink241.gigo. 0.0.0.0         UG        0 0          0 br0

If you are having network errors and you want to see interface statistics, use netstat -i:

$ netstat -i
Kernel Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
br0        1500   0 44804569      0      0      0  3708480      0      0      0 BMRU
em1        1500   0 49533635      0      0      0  7760855      0      0      0 BMRU
em1:1      1500   0      - no statistics available -                            BMRU
lo        16436   0    32131      0      0      0    32131      0      0      0 LRU
vnet0      1500   0        4      0      0      0      225      0      0      1 BMRU
vnet1      1500   0  3761563      0      0      0 45308606      0      0      0 BMRU

To view TCP/UDP statistics run:

$netstat -s
(snip)
Tcp:
    510 active connections openings
    170202 passive connection openings
    1233 failed connection attempts
    114 connection resets received
    1 connections established
    3038753 segments received
    3310750 segments send out
    11236 segments retransmited
    0 bad segments received.
    469 resets sent
(snip)
Udp:
    304134 packets received
    37 packets to unknown port received.
    0 packet receive errors
    304186 packets sent
(snip)

5) top : All time favorite for most engineers. Display a number of statistics, relating to CPU, memory and processes. Extremely useful to see which processes are taking up most resources on the system.

6) traceroute: Traceroute sends 3 UDP packets at a time to an invalid port address on the destination host. The TTL is set to 1 initially, as soon as the packets reach the first hop or router, the router will responds with an ICMP Time Exceeded message. Traceroute then sets the TTL to 2 and sends the packet again. Hop 2 will then respond with TTL exceeded, and traceroute will then sedn 3 packets with TTL of 3. Traceroute keeps doing this until it reaches the destination. Using traceroute we can figure out the network path to a destination. Also, if ICMP is blocked by some routers we can use TCP with the -T flag. In the output of traceroute if you see ‘*’ it means that router is not returning an ICMP message.

$ traceroute www.google.com
traceroute to www.google.com (74.125.239.147), 30 hops max, 60 byte packets
 1  66.220.4.225 (66.220.4.225)  8.269 ms  8.142 ms  8.581 ms
 2  10gigabitethernet1-1.core1.pao1.he.net (184.105.213.66)  8.382 ms  8.410 ms  2.025 ms
 3  184.105.224.254 (184.105.224.254)  12.300 ms  12.341 ms  12.355 ms
 4  209.85.240.114 (209.85.240.114)  1.242 ms  1.256 ms  1.257 ms
 5  66.249.95.31 (66.249.95.31)  12.384 ms  12.442 ms  12.402 ms
 6  nuq05s02-in-f19.1e100.net (74.125.239.147)  12.339 ms  11.187 ms  11.206 ms

7) ping: Ping uses ICMP ECHO_REQUEST to measure the time it takes to reach a destination. ICMP stands for internet control messaging protocol. Ping is very useful to see if a host is alive. You can ping the broadcast address of a network to see how many network devices are on the network. Ping is also used to measure network latency. Round-trip times and packet loss statistics are computed. If you see

		$ ping www.google.com
		PING www.google.com (74.125.239.144) 56(84) bytes of data.
		64 bytes from nuq05s02-in-f16.1e100.net (74.125.239.144): icmp_seq=1 ttl=59 time=1.18 ms
		64 bytes from nuq05s02-in-f16.1e100.net (74.125.239.144): icmp_seq=2 ttl=59 time=1.23 ms
		64 bytes from nuq05s02-in-f16.1e100.net (74.125.239.144): icmp_seq=3 ttl=59 time=1.20 ms
		64 bytes from nuq05s02-in-f16.1e100.net (74.125.239.144): icmp_seq=4 ttl=59 time=1.19 ms
		

8) Valgrind: This is normally not installed on CentOS box, you can do the installation using ‘sudo yum install valgrind -y’. Valgrind is good at figuring out memory leaks. I often use this program when I find a system is running out of memory on a reoccurring basis. To use valgrind, run ‘valgrind —leak-check=yes yourprogram’. Under leak summary you will see definitely lost with ‘X’ number of bytes if there is a memory leak. You will also see line numbers of code if you had compiled the code with -g option to gcc. Once you figure out if the program has memory leaks based on the output of valgrind, next step is to fix the memory leak with the help of a developer. If that is not possible, as a temporary stop gap measure you can restart the application on a period basis if the application is able to tolerate restarts without interrupting end users. For instance if you are using ‘Jetty’ and the Java app has a memory leak, restart Jetty will give you the memory back temporarily. Of course you have to ensure that there are other severs running Jetty which can handle the load while this particular server is restarted. (See http://valgrind.org for additional information)

9) oprofile: Oprofile is a system application profiler, very useful to figure out what an application is doing.

10) tcpdump: The most reliable way for snooping network traffic. In the below example I am using tcpdump to filter traffic with destination http://www.google.com. I then open a telnet session to http://www.google.com on port 80. As you can see from the tcpdump output below a 3 way TCP handshake is visible.

$ telnet www.google.com 80
Trying 74.125.239.116...
Connected to www.google.com.
Escape character is '^]'.
^]quit

telnet> quit
Connection closed.

# tcpdump host www.google.com
reading from file file, link-type EN10MB (Ethernet)
23:29:36.745130 IP myhost.com.54378 > nuq05s01-in-f16.1e100.net.http: Flags [S], seq 814284659, win 14600, options [mss 1460,sackOK,TS val 657765386 ecr 0,nop,wscale 7], length 0
23:29:36.746269 IP nuq05s01-in-f16.1e100.net.http > myhost.com.54378: Flags [S.], seq 217057752, ack 814284660, win 62392, options [mss 1430,sackOK,TS val 3024867195 ecr 657765386,nop,wscale 6], length 0
23:29:36.746293 IP myhost.com.54378 > nuq05s01-in-f16.1e100.net.http: Flags [.], ack 1, win 115, options [nop,nop,TS val 657765387 ecr 3024867195], length 0
23:29:39.736476 IP myhost.com.54378 > nuq05s01-in-f16.1e100.net.http: Flags [F.], seq 1, ack 1, win 115, options [nop,nop,TS val 657768378 ecr 3024867195], length 0
23:29:39.737655 IP nuq05s01-in-f16.1e100.net.http > myhost.com.54378: Flags [F.], seq 1, ack 2, win 975, options [nop,nop,TS val 3024870187 ecr 657768378], length 0
23:29:39.737673 IP myhost.com.54378 > nuq05s01-in-f16.1e100.net.http: Flags [.], ack 2, win 115, options [nop,nop,TS val 657768379 ecr 3024870187], length 0

11) perf: Perf collects performance counters for Linux.

12) tuned : RHEL has many profiles available that can be applied to a system. These profile include certain modifications to kernel/disk/network parameter which are suitable for the workload on a system. Tuned allows you to set preset profiles, such as virtual-host, virtual-guest, enterprise-storage, throughput-performance.

13) sar: sar collects and prints system activity report. This includes network, cpu, disk and memory.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s