Link

TLS uses asymmetric and symmetric encryption. Asymmetric encryption is used for the initial communication, followed by faster symmetric key encryption.

Symmetric ciphers are stream based or block based. Stream based encrypt one message at a time. Block based take a number of bits, and encrypt them together as one. A few symmetric key encryption algorithms are:

– AES
– Blowfish
– RC4
– DES
– 3DES

A few asymmetric key encryption algorithms are:

– DH
– RSA
– Elliptic Curve
– DSS/DSA

A couple of message digest (MD) algorithms are:

– MD5
– SHA

If you want to see which algorithms an SSL server supports, use the tool ‘sslscan’ which can be installed using ‘yum install sslscan -y’. You might have to enable EPEL repository to install using yum. After installation, if you run:

 'sslscan www.google.com:443'

you will see a lot of very useful output, as show below. First you will see the algorithms that sslscan supports, followed by the ones that Google accepts. The most important item section is the one below:


Preferred Server Cipher(s):
SSLv2 0 bits (NONE)
SSLv3 128 bits ECDHE-RSA-RC4-SHA
TLSv1 128 bits ECDHE-RSA-RC4-SHA
TLS11 128 bits ECDHE-RSA-AES128-SHA
TLS12 128 bits ECDHE-RSA-AES128-GCM-SHA256

This is showing that Google prefers SSLv3, TLSv1,1.1 and 1.2. The cipher suites preferred are ECDE-RSA-RC4-SHA.

EDCE is Elliptic Curve Ephemeral Diffie Hellman which supports PFS or Perfect Forward Secrecy.

Normally with RSA, a symmetric key is picked once as part of the SSL HELLO protocol. After that the key does not change. This means that if the servers private key is compromised, then an attacker can get the symmetric key.

With EDCE and PFS, the symmetric key is changed every session, so even if one key is compromised, the other key will not be impacted.

You can configure Apache to prefer cipher suites, see:

  1. https://httpd.apache.org/docs/current/ssl/ssl_intro.html
  2. https://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslciphersuite

Extending VM LVM disk

Leave a reply

I have a CentOS VMs running in VMware, one of the VMs I was running was out of disk space. The disk was originally 85GB, I tried to increase it to 345GB. In an effort to increase the disk size, I tried:

  1. Shutdown the VM
  2. Increase the size of the disk
  3. Power on the VM
  4. Use parted to resize the partition

Unfortunately, step 4 did not work, parted complained that it could not detect the filesystem:

# parted
GNU Parted 2.1
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 344GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 1049kB 538MB 537MB primary ext4 boot
2 538MB 85.9GB 85.4GB primary lvm

(parted) resize
WARNING: you are attempting to use parted to operate on (resize) a file system.
parted's file system manipulation code is not as robust as what you'll find in
dedicated, file-system-specific packages like e2fsprogs.  We recommend
you use parted only to manipulate partition tables, whenever possible.
Support for performing most operations on most types of file systems
will be removed in an upcoming release.
Partition number? 2
Start?  [538MB]?
End?  [85.9GB]? 344GB
Error: Could not detect file system.

It looks like parted does not like LVM. The next thing that I did is to add a new partition, then use ‘pvcreate’ to create a new physical extent, and then add that to the volume group, as seen below:

# parted
GNU Parted 2.1
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 344GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  538MB   537MB   primary  ext4         boot
 2      538MB   85.9GB  85.4GB  primary               lvm

(parted) mkpart
Partition type?  primary/extended? primary
File system type?  [ext2]?
Start? 85.9
End? 344GB
Warning: You requested a partition from 85.9MB to 344GB.
The closest location we can manage is 85.9GB to 344GB.
Is this still acceptable to you?
Yes/No? yes
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy).  As a result, it may not reflect all of your changes until after reboot.
(parted) print
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 344GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  538MB   537MB   primary  ext4         boot
 2      538MB   85.9GB  85.4GB  primary               lvm
 3      85.9GB  344GB   258GB   primary

(parted) quit

#ls -l /dev/sda
sda   sda1  sda2  sda3

# pvcreate /dev/sda3
  dev_is_mpath: failed to get device for 8:3
  Physical volume "/dev/sda3" successfully created
  
# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda2  vg0  lvm2 a--   79.50g      0
  /dev/sda3       lvm2 a--  240.00g 240.00g
  
# vgextend  vg0 /dev/sda3
  Volume group "vg0" successfully extended

# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  vg0    2   4   0 wz--n- 319.49g 240.00g

How to hire talent

Leave a reply

Introduction

Have been on both sides of a site reliability engineer interview, I have come to realize that interviewing is as much art as it is science. Although there are companies which have tried to make interviewing a numbers game, and to abstract out the human element, interviewing continues to be an imperfect process. Almost anything can go wrong in an interview, from both sides. From a companies perspective, you should not consider a candidate confirmed until he/she shows up for work on the first day. In this article, I list a general format that if adhered to, may prove to be useful for interviewing of software engineers, or site reliability engineers.

Format of Interview

Once a candidate has been identified for a role, or has been “sourced” through different means such as LinkedIn, referrals, or otherwise, a recruiter should try to arrange to arrange a phone conversation with the candidate. Once a phone conversation has been arranged, start with the below steps.

– Over the phone, recruiter asks some basic technical questions, passing of which is mandatory. The recruiter also judges if the candidate is a good fit for the position based on experience and job description. In terms of what kind of technical questions a recruiter should ask, they should be limited to less than half a dozen, and should be straight forward, that do not require any technical expertise on the recruiter’s behalf. A few examples for a SRE (site reliability engineer) position can be “Which command do you use to view network traffic on a Linux host?”, “Describe the process of opening a TCP port on a Linux host”, “What is the difference between a hard link and a soft link on a filesystem?”.

– There may be a 45 minute or so technical phone interview with an engineer if a candidate passes the initial phone screen with the recruiter. In this technical phone interview, ask additional questions on various topics, such as networking, filesystems, Linux internals, processes, etc, to get a better idea of the candidates strenghts.
After a successfull technical phone interview, next step should be to schedule an onsite interview.

– Onsite interview is generally held by 4 or 5 engineers, and may be broken up in the following format:
* Unix/Linux system internals
* General troubleshooting questions
* Scripting/programming
* Project management
* Scalable architecture

You should try to avoid having different folks ask the same question. You can also split the 5 interviews into a couple of days if you like. Try to avoid non-technical questions, you can guage other aspects of the person

What to look for in a candidate

From a companies perspective, below are some qualities you should look for in a candidate:
– Smart
– Able to get things done
– Self-directed
– Diverse
– Takes Initiative
– Solves Problems on their own

The question you should be asking yourself is “Is this an awesome person who I want to work with?”. The idea is to hire across the board with consistent standards. Your job is not to break the candidate, it is to try to figure out what he/she does best and where they are lacking in strength.

During the Interview

Before the interview starts, try to make the candidate comfortable. This may sound obvious, but it is not. I remember being in 5 hour interviews without a single break, feeling tired and thirsty and unable to concentrate. Give the candidate frequent breaks for hydration if needed. You want the candidate to bring out the best in a candidate, to do this:
– Come with some hints
– Give candidates with opportunities to success
– Your job is not to hand a test
– Don’t get stuck on a problem
– Make an interview enjoyable
– Don’t make a candidate feel misreable
– If they do not feel good, they won’t be a good referral
– Try to limit general questions, such as definitions, which can be easy to memorize and do not give an idea of depth of a topic.

Good interview questions

– Start with easy questions, work you way to harder problems
– Don’t look for specific answers, as a question may have different valid answers
– Ask 2 or 3 questions
– One question is ok if it has multiple areas to dive into

Comparatively, not so good interview feedback is:
– Is generally incomplete
– Don’t include pictures of the candidate

What not to ask

Don’t ask about:
– Religion
– Age
– Disability
– Genetic info
– National origin
– Pregnancy
– Race and color
– Gender

Asking questions about the above can result in legal complications. If candidate does not get hired, he may turn around and sue your company if you asking him/her about the above questions. He/she may claim that they did not get hired because they belong to a particular race or religion.

Limit questions on:
– Textbook knowledge questions
– Culture questions
– Communications questions

You can pick up on culture fit and communication questions based on the technical questions you ask.

Providing interview feedback

Good Interview Feedback is factual. It should in the least:
– Lists your questions
– Lists the answers as given by the candidate
– Additionally, feedback should include analysis of the interviewer.
– Transcibe code
– Include a high level summary
– Evaluate culture fit
– Role level knowledge
– General cognitive ability

About recordings:
– Avoid audio, video recording

How do you conduct your interviews?

How to setup a BIND DNS Server

2 Replies

The Internet runs mostly on ISC’s BIND DNS server. In this artcile I will explain how to setup a simple BIND server on a Linux box.
The basic steps involved in setting up BIND are:

– Download BIND from ISC’s website
When downloading BIND, you might be better off if you pick an ESV, or an extended support version. As of the writing of this article, 9.9.5 is the latest ESV.
ESV versions are supported for at least 3 years by ISC. You can check the latest ESV version here https://www.isc.org/downloads/. If you are wondering as to why not to use the vendor provided BIND version, it’s because the vendor may not be using the ESV version, and may also lag behind in patches. Using the latest ESV from BIND will get you the most stable version for production use.

– Compile BIND
You will need gcc, make to compile bind. You can run ‘sudo yum groupinstall “Development Tools” -y’ on your CentOS or RedHat box to install GCC. In order to compile BIND, after downloading it, untar it, and then try the following from the directory you have untarred the distriution in.

$sudo yum groupinstall "Development Tools" -y
$./configure --prefix=/opt/bind-9.9.5 --enable-newstats --with-libxml2
$make
#make install

The above will install BIND in /opt/bind-9.9.5, after that you can link /opt/bind to /opt/bind-9.9.5 with “sudo ln -s /opt/bind-9.9.5 /opt/bind”.
I have enabled newstats and with libxml because this allows me to view BIND query stats in a format which I find easy to use.

– Configure a master server
Now that you have downloaded BIND, and compiled it, as well as installed it, we will move onto configuring it. The basic steps in configuration are as follows:

I have placed extensive comments in the conf file and in the zone files, please review them in order to understand what information to place in the files

A. Create a named.conf zone file in etc dire of chroot. See https://github.com/syedaali/configs/blob/master/bind-master-named.conf for a sample with comments.
B. Install a root hints file, example is here https://github.com/syedaali/configs/blob/master/bind-root.hints.
C. Create a forward lookup zone file, sample is here https://github.com/syedaali/configs/blob/master/bind-example.com.zone.
D. Create a reverse lookup zone file, sample is here https://github.com/syedaali/configs/blob/master/bind-10.1.1.rev.
E. Create a loopback forward and reverse zone. Forward sample is here https://github.com/syedaali/configs/blob/master/bind-master.localhost.
Sample of reverse lookup for localhost is here https://github.com/syedaali/configs/blob/master/bind-127.0.0.rev.

– Install one slave server
A. Compile BIND as seen in above steps for master
B. Install BIND in /opt/bind-9.9.5
C. Link to /opt/bin
D. Repeat steps B through E from master, with the exception that in the zone file, specify slave instead of master.
Sample slave zone file is here https://github.com/syedaali/configs/blob/master/bind-slave-named.conf.

– Configure your clients to point to the slave server
It is generally a good idea to avoid pointing your BIND clients to the BIND master. Instead create two slaves, and point the clients to the slaves. For instance if your master has IP address 10.1.1.10, and you have two BIND slaves, with IP 10.1.1.11 and 10.1.1.12, then in your clients /etc/resolv.conf do the following:

$cat /etc/resolv.conf
domain example.com
nameserver 10.1.1.11
nameserver 10.1.1.2

In future blogs I will specify how to monitor BIND. Do share your experience with BIND setup here.

High Availability using Keepalived

6 Replies

HAProxy (http://haproxy.1wt.eu/) is a popular solution for load balancing servers. In a typical load balanced configuration, there may be a number of web servers behind a pair of HAProxy load balancers. The question arises, how do you load balance the HAProxy servers themselves? One way is to use ‘keepalived’ (http://www.keepalived.org/). In order to use this solution, you need at least two HAProxy servers. On both of them install keepalived as explained below. Both servers will have a floating IP which you can create a DNS record for and give that name to your clients. For instance http://www.example.com may have IP 10.1.1.30 which is the floating IP between the two HAProxy servers. Clients will attempt to connect to 10.1.1.30. Depending on which HAProxy server is the master, the IP will be owned by that server. If that server fails, then the backup server will start to issue gratuitous ARP responses for the same IP of 10.1.1.30 and the requests to the web servers will then go through the backup HAProxy server which has now become the primary.

– Install keepalived

$sudo yum install keepalived -y

– Setup two hosts with the following IP address. The floating IP address will be assigned to the virtual router instance in the config later.

10.1.1.10 is h1.example.com (HAProxy server 1)
10.1.1.20 is h2.example.com (HAProxy server 2)
10.1.1.30 is floating IP (shared between the two server)
10.1.1.100 is SMTP server

– Sample basic config file for master. Note the use of ‘state MASTER’ and also ‘priority 101’

! Configuration File for keepalived

global_defs {
   notification_email {
     admin@example.com
   }
   notification_email_from keepalived@example.com
   smtp_server 10.1.1.100
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.1.1.30
    }
}

– Setup backup server. Sample basic config file for backup is below. Note the use of ‘state BACKUP’ and also ‘priority 100’. Priority should be lower on backup.

! Configuration File for keepalived

global_defs {
   notification_email {
     admin@example.com
   }
   notification_email_from keepalived@example.com
   smtp_server 10.1.1.100
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.1.1.30
    }
}

– Start keepalived on both master and backup.

$sudo service keepalived start

– Verify that keepalived is running. You can do this by checking the IP address on the MASTER.

On 10.1.1.10, MASTER, floating IP is assigned to eth0 when MASTER is up.

ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:bd:4c:c7 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.10/24 brd 10.1.1.255 scope global eth0
    inet 10.1.1.30/32 scope global eth0

As you can see the 10.1.1.30 IP is with the master. Check the IP of the BACKUP HAProxy host.
On BACKUP, we have only the BACKUP IP, and not the floating IP.

# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:bd:7a:5f brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.20/24 brd 10.1.1.255 scope global eth0

– You can also use tcpdump and verify that the master is sending VRRP advertisments.
VRRP uses Multicast to keep track of state, you can view multicast traffic using tcpdump as shown below.

#tcpdump net 224.0.0.0/4

Host h1.example.com is the master.

15:49:38.342468 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:49:39.342767 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:49:40.343062 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:49:41.343371 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

– To test the failover, turn off keepalived on the master using ‘service keepalived stop’.
Once I stop keepalived on the MASTER, I see the floating IP has now been assigned to the BACKUP.

# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:bd:7a:5f brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.20/24 brd 10.1.1.255 scope global eth0
    inet 10.1.1.30/32 scope global eth0

– If you keep tcpdump running you should see that the BACKUP host is now sending out the VRRP advertisments. Host h2.example.com is now the master.

15:49:45.953584 IP h2.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:49:46.953889 IP h2.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:49:47.954202 IP h2.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:49:48.954519 IP h2.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

– In /var/log/messages on the BACKUP HAPRoxy host, you should see it taking over the floating IP. Host h2.example.com is the master now.

Mar  8 15:49:44 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) Transition to MASTER STATE
Mar  8 15:49:45 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) Entering MASTER STATE
Mar  8 15:49:45 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) setting protocol VIPs.
Mar  8 15:49:45 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.10.30
Mar  8 15:49:45 h2.example.com Keepalived_healthcheckers[4651]: Netlink reflector reports IP 10.1.10.30 added
Mar  8 15:49:46 h2.example.com ntpd[2974]: Listen normally on 4 eth0 10.1.10.30 UDP 123
Mar  8 15:49:46 h2.example.com ntpd[2974]: peers refreshed
Mar  8 15:49:50 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.10.30
Mar  8 15:49:52 h2.example.com kernel: device eth0 left promiscuous mode

– Since the test has worked, Re-enable keepalived on the MASTER host and watch in tcpdump as host h1.example.com is back to being the master.
You can re-enable keepalived using ‘service keepalived start’.

16:16:41.841635 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
16:16:42.842722 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
16:16:43.843847 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
16:16:44.844982 IP h1.example.com > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

– You can also check in /var/log/messages when host h1.example.com is the master.

Mar  8 15:50:18 h1.example.com Keepalived_vrrp[4324]: Kernel is reporting: interface eth0 UP
Mar  8 15:50:18 h1.example.com Keepalived_vrrp[4324]: VRRP_Instance(VI_1) Transition to MASTER STATE
Mar  8 15:50:19 h1.example.com Keepalived_vrrp[4324]: VRRP_Instance(VI_1) Entering MASTER STATE
Mar  8 15:50:19 h1.example.com Keepalived_vrrp[4324]: VRRP_Instance(VI_1) setting protocol VIPs.
Mar  8 15:50:19 h1.example.com Keepalived_healthcheckers[4323]: Netlink reflector reports IP 10.1.10.30 added
Mar  8 15:50:19 h1.example.com Keepalived_vrrp[4324]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.10.30
Mar  8 15:50:21 h1.example.com Keepalived_vrrp[4324]: Netlink reflector reports IP 10.1.10.10 added

– Once you return H1 host to being the MASTER by starting keepalived, in /var/log/messages on H2 host you will see it giving up the floating IP.
After I bring h1.example.com back online, h2.example.com becomes the backup.

Mar  8 15:50:18 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) Received higher prio advert
Mar  8 15:50:18 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) Entering BACKUP STATE
Mar  8 15:50:18 h2.example.com Keepalived_vrrp[4652]: VRRP_Instance(VI_1) removing protocol VIPs.
Mar  8 15:50:18 h2.example.com Keepalived_healthcheckers[4651]: Netlink reflector reports IP 10.1.10.30 removed

– You can further verify your configuration by running a ping during the above exercise. Ping the floating IP from another host.
There is minimal packet loss as both MASTER, and the BACKUP take over each other’s services as appropriate.

$ ping 10.1.10.30
PING 10.1.10.30 (10.1.10.30): 56 data bytes
64 bytes from 10.1.10.30: icmp_seq=0 ttl=61 time=90.486 ms
64 bytes from 10.1.10.30: icmp_seq=1 ttl=61 time=89.514 ms
64 bytes from 10.1.10.30: icmp_seq=2 ttl=61 time=87.989 ms
64 bytes from 10.1.10.30: icmp_seq=3 ttl=61 time=98.162 ms
64 bytes from 10.1.10.30: icmp_seq=4 ttl=61 time=87.107 ms
64 bytes from 10.1.10.30: icmp_seq=5 ttl=61 time=89.163 ms
64 bytes from 10.1.10.30: icmp_seq=6 ttl=61 time=88.792 ms
64 bytes from 10.1.10.30: icmp_seq=7 ttl=61 time=89.156 ms
Request timeout for icmp_seq 8                                <------ At this point I stopped the keepalived on MASTER
Request timeout for icmp_seq 9
64 bytes from 10.1.10.30: icmp_seq=10 ttl=61 time=88.386 ms   <------ BACKUP has now started to respond to ping for floating IP
64 bytes from 10.1.10.30: icmp_seq=11 ttl=61 time=91.164 ms
64 bytes from 10.1.10.30: icmp_seq=12 ttl=61 time=88.215 ms
64 bytes from 10.1.10.30: icmp_seq=13 ttl=61 time=88.457 ms
64 bytes from 10.1.10.30: icmp_seq=14 ttl=61 time=87.170 ms
64 bytes from 10.1.10.30: icmp_seq=15 ttl=61 time=120.544 ms
64 bytes from 10.1.10.30: icmp_seq=16 ttl=61 time=91.861 ms
Request timeout for icmp_seq 17                               <------ I restarted keepalived on MASTER
64 bytes from 10.1.10.30: icmp_seq=18 ttl=61 time=89.658 ms
64 bytes from 10.1.10.30: icmp_seq=19 ttl=61 time=90.201 ms
64 bytes from 10.1.10.30: icmp_seq=20 ttl=61 time=88.008 ms
64 bytes from 10.1.10.30: icmp_seq=21 ttl=61 time=88.369 ms
^C
--- 10.1.10.30 ping statistics ---
22 packets transmitted, 19 packets received, 13.6% packet loss
round-trip min/avg/max/stddev = 87.107/91.179/120.544/7.315 ms

The above is a simple example of using IP based monitoring. You can also do application based monitoring. In order to do this, we will modify our config file on both master and slave to include a check which will check the status of the HAProxy service. If it is running, then it will continue to serve the floating IP through the MASTER. If the service stops, then the BACKUP will resume ownership. This level of monitoring is in addition to monitoring the network interface being up. 

! Configuration File for keepalived

global_defs {
   notification_email {
     admin@example.com
   }
   notification_email_from keepalived@example.com
   smtp_server 10.1.1.100
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_script check_haproxy {
    script    "/sbin/service haproxy status"
    interval 2
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.1.1.30
    }
    track_script {
        check_haproxy
    }
}

Notice in the above we added two additinal sections. One is ‘vrrp_script check_haproxy’. This code will run haproxy status on the MASTER. If the return code is ‘0’ then the service is considered to be up. If the return code is other than ‘0’ then the service is considered to be down and the BACKUP host will then take-over the floating IP.

– Backup server config for application monitoring. Similar to the MASTER.

! Configuration File for keepalived

global_defs {
   notification_email {
     admin@example.com
   }
   notification_email_from keepalived@example.com
   smtp_server 10.1.1.100
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_script check_haproxy {
    script    "/sbin/service haproxy status"
    interval 2
    fall 2
    rise 2
}


vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.1.1.30
    }
    track_script {
        check_haproxy
    }
    
}

How do you use keepalived and HAPRoxy in your network? Share your comments.

Adding a new LVM partition with GNU parted

1 Reply

In this brief article I will explain how to add a new physical parition to an existing disk, and then to use that new parition to create a mountable logical volume.
Let’s assume that we have a 1TB disk, and we are running CentOS/RedHat, in this case I am using version 6.5

First attempt to view partition table, turns out that disk has GUID Partition table:

# fdisk -l /dev/sda

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sda: 999.7 GB, 999653638144 bytes
255 heads, 63 sectors/track, 121534 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x6929f946

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1      121535   976224255+  ee  GPT

Let’s try to view partition table using GNU parted, and then add a 189GB partition.
The command ‘print free’ shows the free space at the end.
I used the command ‘mkpart’ to add the partition. Notice that partition 3 stops at 211GB.
So I can create a new partition from 211GB onwards. The end of free space is 1000GB, so I picked 400GB as my end, leaving another 600GB free for later use.
I called my new partition ‘bigdisk’ and it’s numbered 4.

# parted /dev/sda
GNU Parted 2.1
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print free
Model: Dell Virtual Disk (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name  Flags
        17.4kB  1049kB  1031kB  Free Space
 1      1049kB  211MB   210MB   fat16              boot
 2      211MB   840MB   629MB   ext4
 3      840MB   211GB   210GB                      lvm
        211GB   1000GB  789GB   Free Space

(parted) mkpart
Partition name?  []? bigdisk
File system type?  [ext2]? ext4
Start? 211GB
End? 400GB
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy).  As a result, it may not reflect all of your changes until after reboot.
(parted) print
Model: Dell Virtual Disk (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name     Flags
 1      1049kB  211MB  210MB  fat16                 boot
 2      211MB   840MB  629MB  ext4
 3      840MB   211GB  210GB                        lvm
 4      211GB   400GB  189GB               bigdisk

I have decided to remove the 189GB partition and increase it’s size. So instead of starting at 211GB and ending at 400GB, I am ending at 600GB.
This gives me a 389GB partition.

# parted
GNU Parted 2.1
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: Dell Virtual Disk (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name     Flags
 1      1049kB  211MB  210MB  fat16                 boot
 2      211MB   840MB  629MB  ext4
 3      840MB   211GB  210GB                        lvm
 4      211GB   400GB  189GB               bigdisk

(parted) rm 4
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy).  As a result, it may not reflect all of your changes until after reboot.
(parted) mkpart
Partition name?  []? bigdisk
File system type?  [ext2]? ext4
Start? 211GB
End? 600GB
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy).  As a result, it may not reflect all of your changes until after reboot.
(parted) print
Model: Dell Virtual Disk (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name     Flags
 1      1049kB  211MB  210MB  fat16                 boot
 2      211MB   840MB  629MB  ext4
 3      840MB   211GB  210GB                        lvm
 4      211GB   600GB  389GB               bigdisk

(parted) quit
Information: You may need to update /etc/fstab.

I will now view my LVS physical extent:

# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sda3  vg_hv1 lvm2 a--  195.31g 11.71g

I need to add the 389GB partition as a physical extent. I know it’s /dev/sda4 because when I ran parted it showed partition #4.

# pvcreate /dev/sda4
  dev_is_mpath: failed to get device for 8:4
  Physical volume "/dev/sda4" successfully created
# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sda3  vg_hv1 lvm2 a--  195.31g  11.71g
  /dev/sda4         lvm2 a--  362.70g 362.70g

I have one volume group on my system:

# vgs
  VG     #PV #LV #SN Attr   VSize   VFree
  vg_hv1   1   7   0 wz--n- 195.31g 11.71g

I am going to add another volume group, I don’t have to do this. I can simply extend my existing volume group, but I want to make another volume group in order to logically separate my applications.

# vgcreate vg_ic  /dev/sda4
  Volume group "vg_ic" successfully created
# vgs
  VG     #PV #LV #SN Attr   VSize   VFree
  vg_hv1   1   7   0 wz--n- 195.31g  11.71g
  vg_ic    1   0   0 wz--n- 362.70g 362.70g

Now I can create my logical volumes as needed. My existing logical volumes are:

# lvs
  LV      VG     Attr       LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_home vg_hv1 -wi-ao---- 19.53g
  lv_root vg_hv1 -wi-ao---- 39.06g
  lv_swap vg_hv1 -wi-ao----  7.81g
  lv_tmp  vg_hv1 -wi-ao----  9.77g
  lv_var  vg_hv1 -wi-ao----  9.77g
  lv_vm1  vg_hv1 -wi-ao---- 48.83g
  lv_vm2  vg_hv1 -wi-ao---- 48.83g

Adding additional logical volumes:

# lvcreate -L 80G vg_ic -n lv_cdrive
  Logical volume "lv_cdrive" created
# lvs
  LV        VG     Attr       LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_home   vg_hv1 -wi-ao---- 19.53g
  lv_root   vg_hv1 -wi-ao---- 39.06g
  lv_swap   vg_hv1 -wi-ao----  7.81g
  lv_tmp    vg_hv1 -wi-ao----  9.77g
  lv_var    vg_hv1 -wi-ao----  9.77g
  lv_vm1    vg_hv1 -wi-ao---- 48.83g
  lv_vm2    vg_hv1 -wi-ao---- 48.83g
  lv_cdrive vg_ic  -wi-a----- 80.00g

That’s about it! Next step can be to use ‘mkfs.ext4 /dev/mapper/vg_ic-lv_cdrive’ if you want to install ext4 on the LVM, followed by mounting it in /etc/fstab.
Or you can use it with KVM to install a VM on. If you like to add disk in another way, do share your experience.

Using Chromebook to connect to an OpenVPN server

21 Replies

There are two ways that I know of which would allow Chromebook to connect to an OpenVPN server. One method is using developer mode of Chromebook, and the other is using the UI. I am explaining the method which uses the UI below. This is based on certificate authentication, and not a username/password combination.

Using the Chromebook web based user interface method:

– Collect your OpenVPN CA certificate (ca.crt), your private keys (client.keys) and your client certificate (client.crt).
If you are not sure on how to get these files, you should ask your OpenVPN server administrator.
I will explain in a soon to follow blog on how to setup an OpenVPN sevrer.
You will also need an ONC file, instructions are below for this file.

– Export your client.crt into pkcs12 format
You need to do this since Chromebook OS understand pkcs12 format, which stores your private keys with the client certificate.

openssl pkcs12 -export -in client.crt -inkey client.key -out client.p12 -name MyClient

– Upload your ca.crt to Chromebook authority repository
* In your Chromebook, for the URL type in chrome://settings/certificates
* Click on ‘Authorities’
* Click on Import
* Click on Google Drive and then the ca.crt file
– Upload your client.crt to Chromebook certificates repo and bind to an interface

– Upload your client.crt to Chromebook certificates repo and bind to an interface
* In your Chromebook, for the URL type in chrome://settings/certificates
* Click on ‘Your Certificates’
* Click on ‘Import and Bind to Device’
* Click on Google Drive and then the client.p12 file

– Create an OpenVPN ONC file and upload to Chromebook
* In your Chromebook, for the URL type in chrome://net-internals/#chromeos
* Click on Choose File
* Click on Google Drive and then the openvpn.onc file (see below for ONC)

– Try to connect as follows
* In your Chromebook, for the URL type in chrome://settings
* Under Private network, click on MyVPNServer
* If you get asked for a username/password, enter whatever random characters, they don’t matter since we are using certificate based authentication

ONC FIle

A sample of this file can be found here https://github.com/syedaali/configs/blob/master/openvpn-sample.onc. Replace the UUID with a random UUID. You can generate one from http://uuidgenerator.net/ if you like. Replace the name of MyOpenVPN with whatever name you want. For the host, type in your OpenVPN server hostname or IP address. The cacert and the clientcert sections are the important ones. For the CA cert copy your CA cert and paste it in the lines as shown. For the client certificate, copy just the PEM format, or the one that starts with —-BEGIN CERTIFICATE—-.

That’s about it. It’s not very easy, but it’s not too hard either. Perhaps in the future the developers of Chromebook OS will make it a bit easier to connect via VPN.

Apache SSL Offloading

2 Replies

Application servers such as Jetty abd Tomcat are widely used in today’s world. If SSL termination is done on the application server, such as Jetty, it may impact the performance of Jetty. You can also terminate SSL on the load balancer, but load balancers often have a limit on the amount of SSL traffic they can handle. For instance a load balancer might be able to handle 10Gbps, so if your traffic is more than that, it will cause a problem. Another way of reducing load on Jetty is to offload SSL termination to Apache. Apache will then pass the traffic to Jetty in an unencrypted connection. Let’s say you have a load balancer that is in front of your Jetty servers, you can install Apache on each of the servers running Jetty. Each Jetty server will also run Apache. So the connection would look something like this:

Clients--(SSL)-->Load Balancer--(SSL)-->Apache--(Unencrypted)-->Jetty

Apache will be listening on port 443, and Jetty on port 8080. Apache will then forward the traffic to port 8080 on the same host. In the logs of Apache you will see the source IP address of the clients. Configure the load balancer to use DSR, or direct server return. Apache acts as a proxy server. You can find a configuration for Apache here https://github.com/syedaali/configs/blob/master/apache-proxy.conf. I will explain some of the configuration settings below.

#Load the SSL module that is needed to terminate SSL on Apache
LoadModule ssl_module modules/mod_ssl.so

#We don't want to pass Apache server status to the Jetty server
ProxyPass /server-status !

#Apache is listening on port 443
Listen 443

SSLSessionCache         shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout  300
SSLMutex default
SSLRandomSeed startup file:/dev/urandom  256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin

#Location for SSL error logs
ErrorLog logs/ssl_error_log

#Location for SSL traffic logs
TransferLog logs/ssl_access_log

#Log level, this can be emrg, alert, crti, error, warn, notice, info, or debug
#See https://httpd.apache.org/docs/2.2/mod/core.html#loglevel for details
LogLevel warn

#Dont' use SSLv2, instead use SSLv3 and TLSv1
SSLProtocol all -SSLv2

#When choosing a cipher during an SSLv3 ot TLSv1 handshake, normally the client's preference is used. We want Apache to use the server's preference.
SSLHonorCipherOrder On

#SSL Tuning. We want to optimize our SSL chipher by removing some and adding other
SSLCipherSuite ALL:!ADH:!EXP:!LOW:!RC2:!3DES:!SEED:!RC4:+HIGH:+MEDIUM

#SSL certificate file location
#you can generate a self signed certificate file using this command
#$sudo openssl req -x509 -newkey rsa:2048 -keyout apache.key -out apache.crt -days 999 -nodes
#the ca.crt file is a certificate chain file
SSLCertificateFile /etc/httpd/certs/apache.crt

#SSL private key file location
SSLCertificateKeyFile /etc/httpd/certs/apache.key

#This directive sets the all-in-one file where you can assemble the Certificates of Certification Authorities (CA) whose clients you deal with.
SSLCACertificateFile /etc/httpd/certs/ca.crt

#The * is for listening on all IP interfaces.
<VirtualHost *:443>
ServerName <fill-in-server-name>
SSLEngine on

#Allow %2F in URLs. See https://httpd.apache.org/docs/2.2/mod/core.html#allowencodedslashes
AllowEncodedSlashes On
</VirtualHost>

#Disable forward proxy. See https://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxyrequests
ProxyRequests Off

#Any incoming URL is forwarded to localhost port 8080 where Jetty is listening. The noncanon allows raw passing of URLs without any canonicalise of URLS.
# See https://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypassmatch.
ProxyPassMatch  (.*) http://localhost:8080 nocanon

#"When enabled, this option will pass the Host: line from the incoming request to the proxied host, instead of the hostname specified in the ProxyPass line." See #https://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypreservehost
ProxyPreserveHost On

#Display proxy status in server status Apache page See https://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxystatus.
ProxyStatus On

#Keep alive proxy connections
SetEnv proxy-nokeepalive 0

#Set the X-Forwarded-Proto to be https for Jetty to understand. See https://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader.
RequestHeader set X-Forwarded-Proto "https" env=HTTPS

Git on CentOS

Leave a reply

Git is very popular source code management (SCM) system. It has replaced SVN as the preferred choice of coders. In this brief tutorial I will show you how to setup Git on your CentOS box, and start using it. First, install Git from source code, primarily because if you install from the yum repos you might get an older version. Git requires the following packages:

sudo yum install curl-devel expat-devel gettext-devel openssl-devel zlib-devel -y

I already had a previous version of Git installed, so I was able to download Git source using Git. If you do not have Git installed, then you can install it using ‘sudo yum install git -y’.

git clone https://github.com/git/git.git

Uninstall the old version of git, so that you can use the new version which you will build below.

sudo remove git -y

Now install some tools needed to make git:

sudo yum install make -y
sudo yum install gcc -y
sudo yum install cpan -y 
sudo yum install perl-DBI perl-DBD-MySQL -y #For MakeMaker

We are now ready to actually install Git, which is done as follows:

sudo make prefix=/usr/local all
sudo make prefix=/usr/local install

Now that you have Git installed, visit github.com and create your free account. After that create a repo on Github.com using the steps here https://help.github.com/articles/create-a-repo. Next in your shell we will clone the above repo and create a file in it.

mkdir -p git && cd git
git config --global user.name "syedaali"
git config --global user.email "youremailaddress@yahoo.com"
git config --global credential.helper cache
git config --global credential.helper 'cache --timeout=7200'
git clone https://github.com/syedaali/perlex.git
cd perlex
touch README
git add README
git commit -m "Adding README"
git push

Using git config I specified my username which I had created on GitHub.com. I also specified that git should cache my credentials for 7200 seconds, so that I do not have to enter my username and password repeatedly. On Github.com I had created a repo called perlex.git, I cloned that using the ‘git clone’ command. After that I created a file called README, committed it to my local repo and then pushed it to the remote Github.com repo. For further reading visit http://git-scm.com/documentation.

Load Balancing Algorithms

2 Replies

Load balancers use a number of algorithms to direct traffic. Some of the most common algorithms are listed below.

1)Random: Using a random number generator, the load balancer directs connections randomly to the web servers behind it. This type of algorithm may be used if the web servers are of similar or same hardware specifications. If connection monitoring is not enabled, then the load balancer will continue sending traffic to failed web servers.

2)Round Robin : Using a circular queue, the load balancer walks through it, sending one request per server. Same as the random method, this works best when the web servers are of similar or same hardware specifications.

3)Weighted Round Robin: Web servers or a group of web servers are assigned a static weight. For instance, new web servers that can handle more load are assigned a higher weight and older web servers are assigned a lower weight. The load balancer will send more traffic to the servers with a higher weight than the ones to the lower weight. For instance if web server ‘A’ has a weight of 3 and web server ‘B’ has the weight of one, then web server ‘A’ will get 3 times as much traffic as web server ‘B’.

4)Dynamic Round Robin: Servers are not assigned a static weight, instead the weight is built dynamically based on metrics. These metrics may be generated on the servers and the load balancer will check them. The reason this is round robin is because if two servers have the same dynamic weight, the load balancer will use round robin between the different weights.

5)Fastest: The load balancer keeps track of response time from the web server, and will prefer to send connections to those servers that respond the quickest.

6)Least Connection: Keeps a track of the connections to the web servers, and prefers to send connections to the servers with the least number of connections.

7)Observed: Uses a combination of least connection and fastest algorithms. “With this method, servers are ranked based on a combination of the number of current connections and the response time. Servers that have a better balance of fewest connections and fastest response time receive a greater proportion of the connections. ”

8)Predictive: Works well in any environment, uses the observed method, however tries to predict which server will perform well based on rank of observed and will send more traffic to servers with a higher rank.

For additional details read http://bit.ly/1dyGelW.