How to create a Docker base image

Here is a compelete set of instructions to create your own Docker base image. Docker base images are the first layer in a Docker image. There are plenty of base images available, however you can create your own if you like as well.

First, install debootstrap package on any given Ubuntu host.

# apt-get install debootstrap -y

Next, install docker.

# apt-get install docker.io -y

Next, download Ubuntu.

# debootstrap ubuntu-suite-name directory-name-to-download-to > /dev/null

Example for 16.04: debootstrap xenial xenial > /dev/null
You can get a list of Ubunutu suite names from https://wiki.ubuntu.com/DevelopmentCodeNames
Now you can import the Xenial directory into Docker as an image.

# tar -C xenial -c . | docker import - xenial

That’s it to create an image and store it locally, you can verify the image using ‘docker images’.
If you want to run a container using the image try ‘docker run -it xenial /bin/bash’. This should run a BASH shell in the container and give you a BASH command prompt.

Next, if you want to push this to your Docker hub registry, try the below steps:

# docker login -u your-docker-hub-username
# docker tag image-id-that-you-got-from-docker-images-cmd your-docker-hub-username/ubuntu:16.04 
  Example for Xenial or Ubuntu 16.04: docker tag s3489dk349d0 syedaali/unbuntu:16.04
# docker push syedaali/unbuntu

To verify, visit in your browser hub.docker.com, login and check to make sure the image is present.
Alternatively you can run ‘docker search syedaali’ and it will show you the images that I own. Replace my username with yours of course.

Copying files in Linux

Copying files should be simple, yet there are a number of ways of transferring files.
Some of the ways that I could think of are listed here.

# -a=archive mode; equals -rlptgoD
# -r=recurse into directories
# -l=copy symlinks as symlinks
# -p=preserve permissions
# -t=preserve modification times
# -g=preserve group
# -o=preserve owner (super-user only)
# -D=preserve device files (super-user only) and special files
# -v=verbose
# -P=keep partially transferred files and show progress
# -H=preserve hardlinks
# -A=preserve ACLs
# -X=preserve selinux and other extended attributes
$ rsync -avPHAX /source /destination

# cross systems using ssh
# -z=compress
# -e=specify remote shell to use
$ rsync -azv -e ssh /source user@destinationhost:/destination-dir

# -xdev=Don’t  descend  directories on other filesystems
# -print=print the filenames found
# -p=Run in copy-pass mode
# -d=make directories
# -m=preserve-modification-time
# -v=verbose
$ find /source -xdev -path | cpio -pdmv /destination

# let's not forget good old cp
# -r=recursive
# -p=preserve mode,ownership,timestamps
# -v=verbose
$ cp -rpv --sparse=always /source /destination

# tar
# -c=create a new archive
# -v=verbose
# -f=use archive file
$ tar cvf - /source | (cd /destination && tar xvf -)

# scp
$ scp -r /source user@destinationhost:/destination-dir

# copy an entire partition
$ dd if=/dev/source-partition of=/dev/destination-partition bs=<block-size>

IP Header Explained

Mininum 20 bytes, maximum 60 bytes.
Maximum size is 65,535 bytes for IP header + data.

– Version field (4 bits) for IPv4 or IPv6
– IHL or Internet Header Length (4 bits), it is the number of 32-bit words in the header
– DS field, (6 bits) or Differentiated Service
– ECN (2 bits) explicit congestion notification
– Total length (16-bit) indicates total length of IP datagram
– Identification (16 bits) – Helps identify each datagram, counter based, very imp for fragmentation.
– Flags (3 bits) – Used to indiciate fragmentation.
– Fragment offset (13 bits) – Used to indicate offset of fragmentation.
– TTL (8 bits) upper limit of routers the datagram can pass through. Normally set at 64, although 128 or 255 is also common.
– Protocol (8 bits) – Specifies protocol encapsulated. 6 is TCP and 17 is UDP.
– Header Checksum (16 bits) – Checksum of header only and not payload.
– Source IP (32 bit)
– Destination IP (32 bit)
– Options (Up to 320 bits/40 bytes)
– IP Data if any, up to 65,515 bytes

Extending VM LVM disk

I have a CentOS VMs running in VMware, one of the VMs I was running was out of disk space. The disk was originally 85GB, I tried to increase it to 345GB. In an effort to increase the disk size, I tried:

  1. Shutdown the VM
  2. Increase the size of the disk
  3. Power on the VM
  4. Use parted to resize the partition

Unfortunately, step 4 did not work, parted complained that it could not detect the filesystem:

# parted
GNU Parted 2.1
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 344GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 1049kB 538MB 537MB primary ext4 boot
2 538MB 85.9GB 85.4GB primary lvm

(parted) resize
WARNING: you are attempting to use parted to operate on (resize) a file system.
parted's file system manipulation code is not as robust as what you'll find in
dedicated, file-system-specific packages like e2fsprogs.  We recommend
you use parted only to manipulate partition tables, whenever possible.
Support for performing most operations on most types of file systems
will be removed in an upcoming release.
Partition number? 2
Start?  [538MB]?
End?  [85.9GB]? 344GB
Error: Could not detect file system.

It looks like parted does not like LVM. The next thing that I did is to add a new partition, then use ‘pvcreate’ to create a new physical extent, and then add that to the volume group, as seen below:

# parted
GNU Parted 2.1
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 344GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  538MB   537MB   primary  ext4         boot
 2      538MB   85.9GB  85.4GB  primary               lvm

(parted) mkpart
Partition type?  primary/extended? primary
File system type?  [ext2]?
Start? 85.9
End? 344GB
Warning: You requested a partition from 85.9MB to 344GB.
The closest location we can manage is 85.9GB to 344GB.
Is this still acceptable to you?
Yes/No? yes
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy).  As a result, it may not reflect all of your changes until after reboot.
(parted) print
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 344GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  538MB   537MB   primary  ext4         boot
 2      538MB   85.9GB  85.4GB  primary               lvm
 3      85.9GB  344GB   258GB   primary

(parted) quit

#ls -l /dev/sda
sda   sda1  sda2  sda3

# pvcreate /dev/sda3
  dev_is_mpath: failed to get device for 8:3
  Physical volume "/dev/sda3" successfully created
  
# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda2  vg0  lvm2 a--   79.50g      0
  /dev/sda3       lvm2 a--  240.00g 240.00g
  
# vgextend  vg0 /dev/sda3
  Volume group "vg0" successfully extended

# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  vg0    2   4   0 wz--n- 319.49g 240.00g

High Availability using Keepalived

HAProxy (http://haproxy.1wt.eu/) is a popular solution for load balancing servers. In a typical load balanced configuration, there may be a number of web servers behind a pair of HAProxy load balancers. The question arises, how do you load balance the HAProxy servers themselves? One way is to use ‘keepalived’ (http://www.keepalived.org/). In order to use this solution, you need at least two HAProxy servers. On both of them install keepalived as explained below. Both servers will have a floating IP which you can create a DNS record for and give that name to your clients. For instance http://www.example.com may have IP 10.1.1.30 which is the floating IP between the two HAProxy servers. Clients will attempt to connect to 10.1.1.30. Depending on which HAProxy server is the master, the IP will be owned by that server. If that server fails, then the backup server will start to issue gratuitous ARP responses for the same IP of 10.1.1.30 and the requests to the web servers will then go through the backup HAProxy server which has now become the primary.

– Install keepalived

$sudo yum install keepalived -y

– Setup two hosts with the following IP address. The floating IP address will be assigned to the virtual router instance in the config later.

10.1.1.10 is h1.example.com (HAProxy server 1)
10.1.1.20 is h2.example.com (HAProxy server 2)
10.1.1.30 is floating IP (shared between the two server)
10.1.1.100 is SMTP server

– Sample basic config file for master. Note the use of ‘state MASTER’ and also ‘priority 101’

! Configuration File for keepalived

global_defs {
   notification_email {
     admin@example.com
   }
   notification_email_from keepalived@example.com
   smtp_server 10.1.1.100
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.1.1.30
    }
}

– Setup backup server. Sample basic config file for backup is below. Note the use of ‘state BACKUP’ and also ‘priority 100’. Priority should be lower on backup.

! Configuration File for keepalived

global_defs {
   notification_email {
     admin@example.com
   }
   notification_email_from keepalived@example.com
   smtp_server 10.1.1.100
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.1.1.30
    }
}

– Start keepalived on both master and backup.

$sudo service keepalived start

– Verify that keepalived is running. You can do this by checking the IP address on the MASTER.

On 10.1.1.10, MASTER, floating IP is assigned to eth0 when MASTER is up.

ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:bd:4c:c7 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.10/24 brd 10.1.1.255 scope global eth0
    inet 10.1.1.30/32 scope global eth0

As you can see the 10.1.1.30 IP is with the master. Check the IP of the BACKUP HAProxy host.
On BACKUP, we have only the BACKUP IP, and not the floating IP.

# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:bd:7a:5f brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.20/24 brd 10.1.1.255 scope global eth0

– You can also use tcpdump and verify that the master is sending VRRP advertisments.
VRRP uses Multicast to keep track of state, you can view multicast traffic using tcpdump as shown below.

#tcpdump net 224.0.0.0/4

Host h1.example.com is the master.

15:49:38.342468 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:49:39.342767 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:49:40.343062 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:49:41.343371 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

– To test the failover, turn off keepalived on the master using ‘service keepalived stop’.
Once I stop keepalived on the MASTER, I see the floating IP has now been assigned to the BACKUP.

# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:bd:7a:5f brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.20/24 brd 10.1.1.255 scope global eth0
    inet 10.1.1.30/32 scope global eth0

– If you keep tcpdump running you should see that the BACKUP host is now sending out the VRRP advertisments. Host h2.example.com is now the master.

15:49:45.953584 IP h2.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:49:46.953889 IP h2.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:49:47.954202 IP h2.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:49:48.954519 IP h2.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

– In /var/log/messages on the BACKUP HAPRoxy host, you should see it taking over the floating IP. Host h2.example.com is the master now.

Mar  8 15:49:44 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) Transition to MASTER STATE
Mar  8 15:49:45 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) Entering MASTER STATE
Mar  8 15:49:45 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) setting protocol VIPs.
Mar  8 15:49:45 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.10.30
Mar  8 15:49:45 h2.example.com Keepalived_healthcheckers[4651]: Netlink reflector reports IP 10.1.10.30 added
Mar  8 15:49:46 h2.example.com ntpd[2974]: Listen normally on 4 eth0 10.1.10.30 UDP 123
Mar  8 15:49:46 h2.example.com ntpd[2974]: peers refreshed
Mar  8 15:49:50 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.10.30
Mar  8 15:49:52 h2.example.com kernel: device eth0 left promiscuous mode

– Since the test has worked, Re-enable keepalived on the MASTER host and watch in tcpdump as host h1.example.com is back to being the master.
You can re-enable keepalived using ‘service keepalived start’.

16:16:41.841635 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
16:16:42.842722 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
16:16:43.843847 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
16:16:44.844982 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

– You can also check in /var/log/messages when host h1.example.com is the master.

Mar  8 15:50:18 h1.example.com Keepalived_vrrp[4324]: Kernel is reporting: interface eth0 UP
Mar  8 15:50:18 h1.example.com Keepalived_vrrp[4324]: VRRP_Instance(VI_1) Transition to MASTER STATE
Mar  8 15:50:19 h1.example.com Keepalived_vrrp[4324]: VRRP_Instance(VI_1) Entering MASTER STATE
Mar  8 15:50:19 h1.example.com Keepalived_vrrp[4324]: VRRP_Instance(VI_1) setting protocol VIPs.
Mar  8 15:50:19 h1.example.com Keepalived_healthcheckers[4323]: Netlink reflector reports IP 10.1.10.30 added
Mar  8 15:50:19 h1.example.com Keepalived_vrrp[4324]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.10.30
Mar  8 15:50:21 h1.example.com Keepalived_vrrp[4324]: Netlink reflector reports IP 10.1.10.10 added
    

– Once you return H1 host to being the MASTER by starting keepalived, in /var/log/messages on H2 host you will see it giving up the floating IP.
After I bring h1.example.com back online, h2.example.com becomes the backup.

Mar  8 15:50:18 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) Received higher prio advert
Mar  8 15:50:18 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) Entering BACKUP STATE
Mar  8 15:50:18 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) removing protocol VIPs.
Mar  8 15:50:18 h2.example.com Keepalived_healthcheckers[4651]: Netlink reflector reports IP 10.1.10.30 removed

– You can further verify your configuration by running a ping during the above exercise. Ping the floating IP from another host.
There is minimal packet loss as both MASTER, and the BACKUP take over each other’s services as appropriate.

$ ping 10.1.10.30
PING 10.1.10.30 (10.1.10.30): 56 data bytes
64 bytes from 10.1.10.30: icmp_seq=0 ttl=61 time=90.486 ms
64 bytes from 10.1.10.30: icmp_seq=1 ttl=61 time=89.514 ms
64 bytes from 10.1.10.30: icmp_seq=2 ttl=61 time=87.989 ms
64 bytes from 10.1.10.30: icmp_seq=3 ttl=61 time=98.162 ms
64 bytes from 10.1.10.30: icmp_seq=4 ttl=61 time=87.107 ms
64 bytes from 10.1.10.30: icmp_seq=5 ttl=61 time=89.163 ms
64 bytes from 10.1.10.30: icmp_seq=6 ttl=61 time=88.792 ms
64 bytes from 10.1.10.30: icmp_seq=7 ttl=61 time=89.156 ms
Request timeout for icmp_seq 8                                <------ At this point I stopped the keepalived on MASTER
Request timeout for icmp_seq 9
64 bytes from 10.1.10.30: icmp_seq=10 ttl=61 time=88.386 ms   <------ BACKUP has now started to respond to ping for floating IP
64 bytes from 10.1.10.30: icmp_seq=11 ttl=61 time=91.164 ms
64 bytes from 10.1.10.30: icmp_seq=12 ttl=61 time=88.215 ms
64 bytes from 10.1.10.30: icmp_seq=13 ttl=61 time=88.457 ms
64 bytes from 10.1.10.30: icmp_seq=14 ttl=61 time=87.170 ms
64 bytes from 10.1.10.30: icmp_seq=15 ttl=61 time=120.544 ms
64 bytes from 10.1.10.30: icmp_seq=16 ttl=61 time=91.861 ms
Request timeout for icmp_seq 17                               <------ I restarted keepalived on MASTER
64 bytes from 10.1.10.30: icmp_seq=18 ttl=61 time=89.658 ms
64 bytes from 10.1.10.30: icmp_seq=19 ttl=61 time=90.201 ms
64 bytes from 10.1.10.30: icmp_seq=20 ttl=61 time=88.008 ms
64 bytes from 10.1.10.30: icmp_seq=21 ttl=61 time=88.369 ms
^C
--- 10.1.10.30 ping statistics ---
22 packets transmitted, 19 packets received, 13.6% packet loss
round-trip min/avg/max/stddev = 87.107/91.179/120.544/7.315 ms

The above is a simple example of using IP based monitoring. You can also do application based monitoring. In order to do this, we will modify our config file on both master and slave to include a check which will check the status of the HAProxy service. If it is running, then it will continue to serve the floating IP through the MASTER. If the service stops, then the BACKUP will resume ownership. This level of monitoring is in addition to monitoring the network interface being up.

! Configuration File for keepalived

global_defs {
   notification_email {
     admin@example.com
   }
   notification_email_from keepalived@example.com
   smtp_server 10.1.1.100
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_script check_haproxy {
    script    "/sbin/service haproxy status"
    interval 2
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.1.1.30
    }
    track_script {
        check_haproxy
    }
}

Notice in the above we added two additinal sections. One is ‘vrrp_script check_haproxy’. This code will run haproxy status on the MASTER. If the return code is ‘0’ then the service is considered to be up. If the return code is other than ‘0’ then the service is considered to be down and the BACKUP host will then take-over the floating IP.

– Backup server config for application monitoring. Similar to the MASTER.

! Configuration File for keepalived

global_defs {
   notification_email {
     admin@example.com
   }
   notification_email_from keepalived@example.com
   smtp_server 10.1.1.100
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_script check_haproxy {
    script    "/sbin/service haproxy status"
    interval 2
    fall 2
    rise 2
}


vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.1.1.30
    }
    track_script {
        check_haproxy
    }
    
}

How do you use keepalived and HAPRoxy in your network? Share your comments.

Git on CentOS

Git is very popular source code management (SCM) system. It has replaced SVN as the preferred choice of coders. In this brief tutorial I will show you how to setup Git on your CentOS box, and start using it. First, install Git from source code, primarily because if you install from the yum repos you might get an older version. Git requires the following packages:

sudo yum install curl-devel expat-devel gettext-devel openssl-devel zlib-devel -y

I already had a previous version of Git installed, so I was able to download Git source using Git. If you do not have Git installed, then you can install it using ‘sudo yum install git -y’.

git clone https://github.com/git/git.git

Uninstall the old version of git, so that you can use the new version which you will build below.

sudo remove git -y

Now install some tools needed to make git:

sudo yum install make -y
sudo yum install gcc -y
sudo yum install cpan -y 
sudo yum install perl-DBI perl-DBD-MySQL -y #For MakeMaker

We are now ready to actually install Git, which is done as follows:

sudo make prefix=/usr/local all
sudo make prefix=/usr/local install

Now that you have Git installed, visit github.com and create your free account. After that create a repo on Github.com using the steps here https://help.github.com/articles/create-a-repo. Next in your shell we will clone the above repo and create a file in it.

mkdir -p git && cd git
git config --global user.name "syedaali"
git config --global user.email "youremailaddress@yahoo.com"
git config --global credential.helper cache
git config --global credential.helper 'cache --timeout=7200'
git clone https://github.com/syedaali/perlex.git
cd perlex
touch README
git add README
git commit -m "Adding README"
git push

Using git config I specified my username which I had created on GitHub.com. I also specified that git should cache my credentials for 7200 seconds, so that I do not have to enter my username and password repeatedly. On Github.com I had created a repo called perlex.git, I cloned that using the ‘git clone’ command. After that I created a file called README, committed it to my local repo and then pushed it to the remote Github.com repo. For further reading visit http://git-scm.com/documentation.

Linux troubleshooting tools

An app is running “slow” on Linux, how do you go about figuring out what could be wrong? Typical bottle-necks for a slow application are cpu, disk i/o, network, memory, or database. For instance if Apache is running slow it could be memory starved. Some of the tools we can use to debug application are listed below. Keep in mind that when using these tools, you generally want to take multiple snapshots in time, and not necessarily base your information on a single moment in time.

1) Vmstat shows you a number of interesting statistics. Some things to look for in vmstat include the proc ‘r’ column, which shows the number of processes waiting for run time. If this number if “high” it indicates CPU starvation. CPU starved apps run slow. To fix, increase the number of CPUs in the system. Another field to look at in the output of vmstat is the ‘cs’ field under the system heading. ‘cs’ stands for context switching. A “high” number means that the CPU is having to switch between a lot of tasks. These tasks might be network, disk, or cpu related. One way to solve this problem is to see if the server is doing “too many” things, i.e., running too many applications. Reduce the number of applications running to solve this problem.

$ vmstat -s
     16301872  total memory
      5718716  used memory
      3179672  active memory
      2228164  inactive memory
     10583156  free memory
       166264  buffer memory
      2913548  swap cache
      8191992  total swap
          572  used swap
      8191420  free swap
     26013290 non-nice user cpu ticks
          881 nice user cpu ticks
     17260265 system cpu ticks
   7326487040 idle cpu ticks
     13013070 IO-wait cpu ticks
           72 IRQ cpu ticks
         8054 softirq cpu ticks
            0 stolen cpu ticks
     11212676 pages paged in
    180145305 pages paged out
           44 pages swapped in
          145 pages swapped out
   4176361125 interrupts
   1524882692 CPU context switches
   1367818576 boot time
      9070041 forks

2) mpstat: To view processor utilization based on per core.

$ mpstat -P ALL
Linux 2.6.32-358.6.1.el6.x86_64 (hostname) 	08/20/2013 	_x86_64_	(8 CPU)

10:48:05 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:48:05 PM  all    0.03    0.00    0.23    0.18    0.00    0.00    0.00    0.32   99.24
10:48:05 PM    0    0.05    0.00    0.42    0.72    0.00    0.00    0.00    0.69   98.13
10:48:05 PM    1    0.05    0.00    1.01    0.34    0.00    0.00    0.00    1.78   96.82
10:48:05 PM    2    0.04    0.00    0.11    0.04    0.00    0.00    0.00    0.03   99.78
10:48:05 PM    3    0.03    0.00    0.11    0.18    0.00    0.00    0.00    0.03   99.66
10:48:05 PM    4    0.02    0.00    0.07    0.01    0.00    0.00    0.00    0.02   99.87
10:48:05 PM    5    0.02    0.00    0.08    0.12    0.00    0.00    0.00    0.02   99.77
10:48:05 PM    6    0.01    0.00    0.04    0.00    0.00    0.00    0.00    0.01   99.93
10:48:05 PM    7    0.01    0.00    0.04    0.01    0.00    0.00    0.00    0.01   99.93

3) iostat : If you need help in figuring out how your disk subsystem is performing, use iostat. “The iostat command generates three types of reports, the CPU Utilization report, the Device Utilization report and the Network Filesystem report.” (From the man page of iostat) Items of concern here would be iowait%, which if high indicates that the CPU is waiting for disk I/O to complete. If %util hits 100% then device saturation occurs.

4) netstat: This is an extremely useful command to diagnose numerous network issues.
To view routing table use:

$ netstat -r
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
216.218.223.240 *               255.255.255.240 U         0 0          0 br0
216.218.223.240 *               255.255.255.240 U         0 0          0 em1
link-local      *               255.255.0.0     U         0 0          0 br0
default         uplink241.gigo. 0.0.0.0         UG        0 0          0 br0

If you are having network errors and you want to see interface statistics, use netstat -i:

$ netstat -i
Kernel Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
br0        1500   0 44804569      0      0      0  3708480      0      0      0 BMRU
em1        1500   0 49533635      0      0      0  7760855      0      0      0 BMRU
em1:1      1500   0      - no statistics available -                            BMRU
lo        16436   0    32131      0      0      0    32131      0      0      0 LRU
vnet0      1500   0        4      0      0      0      225      0      0      1 BMRU
vnet1      1500   0  3761563      0      0      0 45308606      0      0      0 BMRU

To view TCP/UDP statistics run:

$netstat -s
(snip)
Tcp:
    510 active connections openings
    170202 passive connection openings
    1233 failed connection attempts
    114 connection resets received
    1 connections established
    3038753 segments received
    3310750 segments send out
    11236 segments retransmited
    0 bad segments received.
    469 resets sent
(snip)
Udp:
    304134 packets received
    37 packets to unknown port received.
    0 packet receive errors
    304186 packets sent
(snip)

5) top : All time favorite for most engineers. Display a number of statistics, relating to CPU, memory and processes. Extremely useful to see which processes are taking up most resources on the system.

6) traceroute: Traceroute sends 3 UDP packets at a time to an invalid port address on the destination host. The TTL is set to 1 initially, as soon as the packets reach the first hop or router, the router will responds with an ICMP Time Exceeded message. Traceroute then sets the TTL to 2 and sends the packet again. Hop 2 will then respond with TTL exceeded, and traceroute will then sedn 3 packets with TTL of 3. Traceroute keeps doing this until it reaches the destination. Using traceroute we can figure out the network path to a destination. Also, if ICMP is blocked by some routers we can use TCP with the -T flag. In the output of traceroute if you see ‘*’ it means that router is not returning an ICMP message.

$ traceroute www.google.com
traceroute to www.google.com (74.125.239.147), 30 hops max, 60 byte packets
 1  66.220.4.225 (66.220.4.225)  8.269 ms  8.142 ms  8.581 ms
 2  10gigabitethernet1-1.core1.pao1.he.net (184.105.213.66)  8.382 ms  8.410 ms  2.025 ms
 3  184.105.224.254 (184.105.224.254)  12.300 ms  12.341 ms  12.355 ms
 4  209.85.240.114 (209.85.240.114)  1.242 ms  1.256 ms  1.257 ms
 5  66.249.95.31 (66.249.95.31)  12.384 ms  12.442 ms  12.402 ms
 6  nuq05s02-in-f19.1e100.net (74.125.239.147)  12.339 ms  11.187 ms  11.206 ms

7) ping: Ping uses ICMP ECHO_REQUEST to measure the time it takes to reach a destination. ICMP stands for internet control messaging protocol. Ping is very useful to see if a host is alive. You can ping the broadcast address of a network to see how many network devices are on the network. Ping is also used to measure network latency. Round-trip times and packet loss statistics are computed. If you see

		$ ping www.google.com
		PING www.google.com (74.125.239.144) 56(84) bytes of data.
		64 bytes from nuq05s02-in-f16.1e100.net (74.125.239.144): icmp_seq=1 ttl=59 time=1.18 ms
		64 bytes from nuq05s02-in-f16.1e100.net (74.125.239.144): icmp_seq=2 ttl=59 time=1.23 ms
		64 bytes from nuq05s02-in-f16.1e100.net (74.125.239.144): icmp_seq=3 ttl=59 time=1.20 ms
		64 bytes from nuq05s02-in-f16.1e100.net (74.125.239.144): icmp_seq=4 ttl=59 time=1.19 ms
		

8) Valgrind: This is normally not installed on CentOS box, you can do the installation using ‘sudo yum install valgrind -y’. Valgrind is good at figuring out memory leaks. I often use this program when I find a system is running out of memory on a reoccurring basis. To use valgrind, run ‘valgrind —leak-check=yes yourprogram’. Under leak summary you will see definitely lost with ‘X’ number of bytes if there is a memory leak. You will also see line numbers of code if you had compiled the code with -g option to gcc. Once you figure out if the program has memory leaks based on the output of valgrind, next step is to fix the memory leak with the help of a developer. If that is not possible, as a temporary stop gap measure you can restart the application on a period basis if the application is able to tolerate restarts without interrupting end users. For instance if you are using ‘Jetty’ and the Java app has a memory leak, restart Jetty will give you the memory back temporarily. Of course you have to ensure that there are other severs running Jetty which can handle the load while this particular server is restarted. (See http://valgrind.org for additional information)

9) oprofile: Oprofile is a system application profiler, very useful to figure out what an application is doing.

10) tcpdump: The most reliable way for snooping network traffic. In the below example I am using tcpdump to filter traffic with destination http://www.google.com. I then open a telnet session to http://www.google.com on port 80. As you can see from the tcpdump output below a 3 way TCP handshake is visible.

$ telnet www.google.com 80
Trying 74.125.239.116...
Connected to www.google.com.
Escape character is '^]'.
^]quit

telnet> quit
Connection closed.

# tcpdump host www.google.com
reading from file file, link-type EN10MB (Ethernet)
23:29:36.745130 IP myhost.com.54378 > nuq05s01-in-f16.1e100.net.http: Flags [S], seq 814284659, win 14600, options [mss 1460,sackOK,TS val 657765386 ecr 0,nop,wscale 7], length 0
23:29:36.746269 IP nuq05s01-in-f16.1e100.net.http > myhost.com.54378: Flags [S.], seq 217057752, ack 814284660, win 62392, options [mss 1430,sackOK,TS val 3024867195 ecr 657765386,nop,wscale 6], length 0
23:29:36.746293 IP myhost.com.54378 > nuq05s01-in-f16.1e100.net.http: Flags [.], ack 1, win 115, options [nop,nop,TS val 657765387 ecr 3024867195], length 0
23:29:39.736476 IP myhost.com.54378 > nuq05s01-in-f16.1e100.net.http: Flags [F.], seq 1, ack 1, win 115, options [nop,nop,TS val 657768378 ecr 3024867195], length 0
23:29:39.737655 IP nuq05s01-in-f16.1e100.net.http > myhost.com.54378: Flags [F.], seq 1, ack 2, win 975, options [nop,nop,TS val 3024870187 ecr 657768378], length 0
23:29:39.737673 IP myhost.com.54378 > nuq05s01-in-f16.1e100.net.http: Flags [.], ack 2, win 115, options [nop,nop,TS val 657768379 ecr 3024870187], length 0

11) perf: Perf collects performance counters for Linux.

12) tuned : RHEL has many profiles available that can be applied to a system. These profile include certain modifications to kernel/disk/network parameter which are suitable for the workload on a system. Tuned allows you to set preset profiles, such as virtual-host, virtual-guest, enterprise-storage, throughput-performance.

13) sar: sar collects and prints system activity report. This includes network, cpu, disk and memory.