Monday, 22 August 2016

Compile and install new kernel on CentOS

Objective: Compile and install new kernel 

OS: CentOS 6.7

Current kernel version  : 2.6.32
Upgraded kernel version : 3.14.77

Download the latest kernel version from https://www.kernel.org/ to /tmp.

1. Make sure you install "Development Tools" using yum which includes gcc packages which are required for compiling kernel. 

[root@centos ~]# yum groupinstall "Development Tools"
Loaded plugins: fastestmirror, security
Setting up Group Process
Loading mirror speeds from cached hostfile
Package 1:make-3.81-20.el6.x86_64 already installed and latest version
.
<snip>
.

2. Extract the newer kernel to /usr/src directory and create an soft link for 'linux' directory.  That would be the directory which is required for you to compile your newer kernel.

[root@centos ~]# tar -Jxvf linux-4.7.2.tar.xz -C /usr/src/
[root@centos src]# ln -s linux-4.7.2/ linux
[root@centos src]# ls
debug  kernels  linux  linux-4.7.2
[root@centos src]# cd linux
[root@centos linux]# ls
arch   certs    CREDITS  Documentation  firmware  include  ipc     Kconfig  lib          Makefile  net     REPORTING-BUGS  scripts   sound  usr
block  COPYING  crypto   drivers        fs        init     Kbuild  kernel   MAINTAINERS  mm        README  samples         security  tools  virt
[root@centos linux]#


3. Since I have only newer kernel and not much to be cleaned for older unnecessary modules, I would choose 'mid-range' i...e mrproper. You can choose depending on your compilations. for more info type 'make help' 

Cleaning targets:
  clean           - Remove most generated files but keep the config and
                    enough build support to build external modules
  mrproper        - Remove all generated files + config + various backup files
  distclean       - mrproper + remove editor backup and patch files

[root@centos linux]# make mrproper
[root@centos linux]#

4. Despite installing "Development Tools", you additional are required to install 'ncurses-devel' for menuconfig (custom configuration of the new kernel i..e which modules needs to be compiled or loadable by kernel) and after saving you it would create a .config file. 

[root@centos linux]#make menuconfig

The one chosen with * modules will be compiled and others which are unchecked they wouldn't be compiled. Hence un-necessary modules which are not required for your system need not be checked here. so you can only choose what your system wishes to do.



5. Compile your actual kernel, which will take around 15-20 mins depending on your number of CPU.

[root@centos linux]# make bzImage
  HOSTCC  scripts/kconfig/conf.o
  HOSTLD  scripts/kconfig/conf
scripts/kconfig/conf  --silentoldconfig Kconfig
.
.
<snip>
.
.
  BUILD   arch/x86/boot/bzImage
Setup is 15164 bytes (padded to 15360 bytes).
System is 4374 kB
CRC 96b7ee84
Kernel: arch/x86/boot/bzImage is ready  (#1)
[root@centos linux]#


6. compile your modules would considerably takes more time than kernel ( approx 60-90 mins ) depending on your nummber of CPU

root@centos linux]# make modules
make[1]: Nothing to be done for `all'.
make[1]: Nothing to be done for `relocs'.
  CHK     include/config/kernel.release
  CHK     include/generated/uapi/linux/version.h
  CHK     include/generated/utsrelease.h
  CALL    scripts/checksyscalls.sh
 .
 .
 <snip>
 .
 .
  H16TOFW firmware/edgeport/down2.fw
  IHEX    firmware/edgeport/down3.bin
  IHEX2FW firmware/whiteheat_loader.fw
  IHEX2FW firmware/whiteheat.fw
  IHEX2FW firmware/keyspan_pda/keyspan_pda.fw
  IHEX2FW firmware/keyspan_pda/xircom_pgs.fw
[root@centos linux]#

7. copies all the modules to a new directory .. /lib/modules/3.14.77/

[root@centos linux]# make modules_install
  INSTALL arch/x86/crypto/aes-x86_64.ko
  INSTALL arch/x86/crypto/aesni-intel.ko
  INSTALL arch/x86/crypto/crc32c-intel.ko
  INSTALL arch/x86/crypto/crct10dif-pclmul.ko
  .
  .
  <snip>
  .
  .
  INSTALL /lib/firmware/keyspan_pda/keyspan_pda.fw
  INSTALL /lib/firmware/keyspan_pda/xircom_pgs.fw
  DEPMOD  3.14.77
[root@centos linux]#

8. Move kernel to the right location and name it correctly , update initramfs, grub so boot up with new kernel.

[root@centos linux]# make install
sh /usr/src/linux-3.14.77/arch/x86/boot/install.sh 3.14.77 arch/x86/boot/bzImage \
                System.map "/boot"
ERROR: modinfo: could not find module parport
ERROR: modinfo: could not find module snd_page_alloc
[root@centos linux]#

On summary, below are the commands used for building newer kernel.
make mrproper
make menuconfig
make bzImage
make modules
make modules_install
make install

9 Reboot system and boot into new kernel from the grub menu.

[root@centos linux]# reboot


[root@centos ~]# uptime
 10:39:41 up 0 min,  2 users,  load average: 0.00, 0.00, 0.00
[root@centos ~]#

[root@centos ~]# uname -r
3.14.77
[root@centos ~]#

Thanks for re-sharing. 

Saturday, 30 July 2016

Troubleshoot network problem using tshark

How tshark works ?

When a packet arrives at the network card, the MAC destination address is checked to see if it matches yours, in which case an interrupt service routine will be generated and handled by the network driver. 

Subsequently, the received data is copied to a memory block defined in the kernel and from there it will be processed by the corresponding protocol stack to be delivered to the appropriate application in user space. Parallel to this process, when Tshark is capturing traffic, the network driver sends a copy of the packets to a kernel subsystem called Packet Filter, which will filter and store in a buffer the desired packets. These packets will be received by Dumpcap (in user space) whose main goal will be to write them into a libpcap file format to be subsequently read by Tshark. As new packets arrive, Dumpcap will add them to the same capture file and it will notify Tshark about their arrival so that they can be processed.




My objective would be to give you brief tutorial on how to find problems related to performance of network, could be due to bandwidth etc.. so we could use tshark to try and find out which hosts are generating more traffic and what type of data are they sending..

List all the network interfaces - tshark -D

Capture traffic from network interface and write to file -
#tshark -i <interface> -w traffic.pacap

How to capture and analyze traffic using tshark ? 

1. Determine which IPs in your VLAN(IPADDRES/NETMASK) could be misusing the network would be able to get IP list. list by dfault would be sorted according to total number of frames, so it could give an idea of heavy talkers.

#tshark -r traffic.pcap -q -z "conv,ip,ip.addr==74.125.130.0/24"

================================================================================
IPv4 Conversations
Filter:ip.addr==74.125.130.0/24
                                               |       <-      | |       ->      | |     Total     |   Rel. Start   |   Duration   |
                                               | Frames  Bytes | | Frames  Bytes | | Frames  Bytes |                |              |
74.125.130.102       <-> 10.0.2.15                105     27191     129     21393     234     48584   112.306444555       260.0255
74.125.130.95        <-> 10.0.2.15                 34      3395      36     11639      70     15034   263.618378290       108.6899
74.125.130.93        <-> 10.0.2.15                 32      3601      37     11601      69     15202   109.882120656       177.0934
================================================================================

2. With above inforamtion we know that IP 74.125.130.102 represents one of the host which is generating more traffic to communicate with other machines on the network 74.125.130.0/24

You could create another pcap file just with the traffic generated by that machine(74.125.130.102)

#tshark -r traffic.pcap -R "ip.addr==74.125.130.102" -w ip.pcap
# capinfos ip.pcap | grep "Number\|time:"
Number of packets:   234
Start time:          Fri Jul 29 20:37:12 2016
End time:            Fri Jul 29 20:41:32 2016

3. Check that your host is not breaking any of your policies of your network, only HTTP & HTTPS is allowed. Below commands will tells us outbound connections to ports other than any (HTTP or HTTPS)

#tshark -o column.format:'" Source ","%s","Destination","%d", "dstport", "%uD","Protocol", "%p"' -r ip.pcap -R "ip.src == 74.125.130.102 && ! dns && tcp.dstport != 80 && tcp.dstport != 443"  | sort -u

74.125.130.102 -> 10.0.2.15    43536 TCP
74.125.130.102 -> 10.0.2.15    43536 TLSv1.2
74.125.130.102 -> 10.0.2.15    43540 TCP
74.125.130.102 -> 10.0.2.15    43540 TLSv1.2

4. I don't have any traffic violating my policies, anyway lets suppose we say if that do exists, then we would have those machines IP address and the port on which they are connected. so to make sure that the traffic is not from other service using the FTP port, lauch tcp stream of that session.

#tshark -o column.format:'"Source","%s","srcport", "%uS","Destination","%d", "dstport", "%uD","Protocol", "%p"' -r ip.pcap -R "tcp.dstport == 43536" | head -1
74.125.130.102 443 10.0.2.15    43536 TCP

#tshark -r ip.pcap -q -z  "follow,tcp,ascii,74.125.130.102:443,10.0.2.15:43536,1"
===================================================================
Follow: tcp,ascii
Filter: ((ip.src eq 74.125.130.102 and tcp.srcport eq 443) and (ip.dst eq 10.0.2.15 and tcp.dstport eq 43536)) or ((ip.src eq 10.0.2.15 and tcp.srcport eq 43536) and (ip.dst eq 74.125.130.102 and tcp.dstport eq 443))
195
............sZ..@G"......!s.....?W...$..5......+./.....
.......3.9./.5.
s.youtube.com..........
...............................h2.spdy/3.1.http/1.1..........
===================================================================

5. Now you could observe that it was "youtube.com" was actullay consuming more bandwidth responsible for slowdown in network.

If you do come across any FTP sessions, troubleshoot the above way, also additionally you will check all the files downloaded by the client.

#tshark -r ip.pcap -q -z  "follow,tcp,ascii,74.125.130.102:443,<Destination machine>:21,1" | grep RETR

6. tshark also allows us to break down each of the protocols captured. Thus we can see hierarchically the number of frames and bytes associated with each protocol. Using capture file, let's see for example the distribution of HTTP and HTTPS traffic used by the IP 74.125.130.102:

#tshark -r traffic.pcap -q -z io,phs,"ip.addr==74.125.130.102 && ssl || http"
===================================================================
Protocol Hierarchy Statistics
Filter: ip.addr==74.125.130.102 && ssl || http

eth                                      frames:122 bytes:40644
  ip                                     frames:122 bytes:40644
    tcp                                  frames:122 bytes:40644
      ssl                                frames:122 bytes:40644
        tcp.segments                     frames:2 bytes:2589
          ssl                            frames:2 bytes:2589
===================================================================

7. It would practically tells us that SSL represents all traffic, let's see the IP's associated with that communication.

#tshark -o column.format:'"destination","%d"' -r  traffic.pcap -R "ip.src ==74.125.130.102 && ssl"| sort -u
10.0.2.15

#whois 10.0.2.15 | grep -i "netname\|netrange"
NetRange:       10.0.0.0 - 10.255.255.255
NetName:        PRIVATE-ADDRESS-ABLK-RFC1918-IANA-RESERVED

With whatever application or information your would get for the IP address/ports, you can create ACLs or IPtables rules to deny certain types of traffic, do a shutdown of a specific port, limit the bandwidth of some protocols so on ...

More references : 

Thanks for re-sharing !

Sunday, 19 June 2016

Docker Basics & Container Customization - Linux

Objective:
Learn how to customize a Docker container image and use it to instantiate application instances across different Linux servers

Introduction: 
Docker captures full application environment into a virtual container that can be deployed across different Linux servers. System administrators and software developers are learning that Docker can help them deploy application images on Linux quickly, reliably, and consistently without dependency and portability problems. Docker containers can define application and its dependencies using small text file(Dockerfile) that can be moved to different Linux releases and quickly rebuilt.  Also Dockerized application are very easy to migrate to another different linux servers either executed as a bare metal in a virtual machine or Linux instances in the cloud.

I would demonstrate how to create Docker container on RHEL 7, modify and use to deploy multiple application instance.  Docker containers are a lightweight virtualization technology for Linux. They provide isolation from other applications and processes running on the same system but make system calls to the same shared Linux kernel, similar to Linux LXC application containers. Docker containers have their own namespace, so they are fully isolated from one another—processes running in one container can't see or impact processes running in another. By default, each container gets its own networking stack, private network interfaces, and IP address, and Docker creates a virtual bridge so containers can communicate.



Getting Started 
I am usig Docker installation on Redhat 7.2 and installtion document can be found at https://docs.docker.com/v1.8/installation/rhel/

You could have your own Docker hub repository to store images that can be used to build running containers. I would pull few of the images from the Docker hub repository for test environment.

[sunlnx@sandbox ~]$docker pull ubuntu:latest
[sunlnx@sandbox ~]$docker pull oraclelinux:6
[sunlnx@sandbox ~]$docker pull oraclelinux:7
[sunlnx@sandbox ~]$docker pull rhel:latest
[sunlnx@sandbox ~]$docker pull mysql/mysql-server
[sunlnx@sandbox ~]$docker pull nginx:latest

To list all the docker images that were pulled above 
[sunlnx@sandbox ~]$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
nginx                latest              0d409d33b27e        2 weeks ago         182.7 MB
ubuntu               latest              2fa927b5cdd3        3 weeks ago         122 MB
oraclelinux          6                   768a3d7b605a        4 weeks ago         222.8 MB
oraclelinux          7                   df602a268e64        5 weeks ago         276.1 MB
rhel                 latest              bf2034427837        6 weeks ago         203.4 MB
mysql/mysql-server   latest              18a962a188ee        11 days ago         366.9 MB
[sunlnx@sandbox ~]$

Container Customization 
I would like to provide multiple, identical web servers across multiple Linux servers, Docker makes it easy to create a preconfigured in a container image. I would then use this pre built image and deploy it across one or many other Linux hosts. I would install "myweb" container and would configure that to deliver web content to the clients. In order to customize I would get an interactive bash shell to run an rhel "myweb" container. 

[sunlnx@sandbox ~]$ docker run -it --name myweb oraclelinux:6 /bin/bash
[root@5b62adeb3abb /]#

In a shell on my Linux host, the docker ps command shows information about the running guest container, 

[sunlnx@sandbox ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
5b62adeb3abb        oraclelinux:6       "/bin/bash"         6 minutes ago       Up 20 seconds                           myweb
[sunlnx@sandbox ~]$

on myweb, I will install httpd using yum and would configure the web server to display. I will create an index.html in /var/www/html on it. 

[root@5b62adeb3abb /]# yum install -y httpd
[root@5b62adeb3abb /]# echo "Web servers main page" > /var/www/html/index.html
[root@5b62adeb3abb /]# exit

Now, I want to create a new Docker image that reflects the contents of the guest container that I just configured. The following docker commit command captures the modified container into a new image named mywebser/httpd:r1

[sunlnx@sandbox ~]$ docker commit -m "ol6-httpd" `docker ps -l -q` mywebser/httpd:r1
sha256:79cf91b1a67f4ac6419b038e76c4e2de492f0eda978d2b07203e217290454108
[sunlnx@sandbox ~]$

The commit command takes as input the image ID number of the myweb container and assigns and returns an ID number for the new image. Running the docker images command now lists the new image mywebser/httpd 

[sunlnx@sandbox ~]$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED              SIZE
mywebser/httpd       r1                  79cf91b1a67f        About a minute ago   766 MB

Incase if I don't require this container I can remove with docker rm command

[sunlnx@sandbox ~]$docker rm myweb 

Because Docker containers persist even though they're no longer running, removing unneeded containers is simply a housekeeping step to reduce clutter on my host, and it allows me to reuse the name myweb1 for a new container.

Deploy Docker Image:
I can deploy any number of web servers now using the new Docker image as a template. The following docker run commands run the container image mywebser/httpd:r1, creating the containers myweb1, myweb2, myweb3, myweb4 and myweb5 executing httpd in each one:

[sunlnx@sandbox ~]$ docker run -d --name myweb1 -p 8080:80 mywebser/httpd:r1 /usr/sbin/httpd -D FOREGROUND
924018f9f7374b3a0ac24d71b6e7b41407dc1492344ef522a4796162fc0e6822
[sunlnx@sandbox ~]$ docker run -d --name myweb2 -p 8081:80 mywebser/httpd:r1 /usr/sbin/httpd -D FOREGROUND
2fc28962e5ab690edfc4e08c529a4206c3285c823ce924514da07ba0c196593a
[sunlnx@sandbox ~]$ docker run -d --name myweb3 -p 8082:80 mywebser/httpd:r1 /usr/sbin/httpd -D FOREGROUND
48964b1e06b29029781630b9734d734bb163603e13a00c2dd0a59f1e4d94ee23
[sunlnx@sandbox ~]$ docker run -d --name myweb4 -p 8083:80 mywebser/httpd:r1 /usr/sbin/httpd -D FOREGROUND
a5e970efd3f3586f8aa6d5e79b03484625ffcd4f22bac869878949eb6b5aaa48
[sunlnx@sandbox ~]$ docker run -d --name myweb5 -p 8084:80 mywebser/httpd:r1 /usr/sbin/httpd -D FOREGROUND
92bbf522aa41c07838626b03630bae63770c0678d06b7d698f05f203e8ed8b69
[sunlnx@sandbox ~]$

[sunlnx@sandbox ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                  NAMES
92bbf522aa41        mywebser/httpd:r1   "/usr/sbin/httpd -D F"   About a minute ago   Up About a minute   0.0.0.0:8084->80/tcp   myweb5
a5e970efd3f3        mywebser/httpd:r1   "/usr/sbin/httpd -D F"   About a minute ago   Up About a minute   0.0.0.0:8083->80/tcp   myweb4
48964b1e06b2        mywebser/httpd:r1   "/usr/sbin/httpd -D F"   About a minute ago   Up About a minute   0.0.0.0:8082->80/tcp   myweb3
2fc28962e5ab        mywebser/httpd:r1   "/usr/sbin/httpd -D F"   About a minute ago   Up About a minute   0.0.0.0:8081->80/tcp   myweb2
924018f9f737        mywebser/httpd:r1   "/usr/sbin/httpd -D F"   About a minute ago   Up About a minute   0.0.0.0:8080->80/tcp   myweb1
[sunlnx@sandbox ~]$

Using a web browser or curl, I can test the web server running in each guest:

[sunlnx@sandbox ~]$ curl http://sandbox:8080
Web servers main page
[sunlnx@sandbox ~]$ curl http://sandbox:8081
Web servers main page
[sunlnx@sandbox ~]$ curl http://sandbox:8082
Web servers main page
[sunlnx@sandbox ~]$ curl http://sandbox:8083
Web servers main page
[sunlnx@sandbox ~]$ curl http://sandbox:8084
Web servers main page

The Docker Engine also assigns each running container a virtual network interface, which you can see with the docker inspect command:
[sunlnx@sandbox ~]$ docker inspect myweb1
[sunlnx@sandbox ~]$ docker inspect -f '{{ .NetworkSettings.IPAddress }}' myweb1
172.17.0.2
[sunlnx@sandbox ~]$

Saving Docker image:
You could backup the image to a tar using docker command

[sunlnx@sandbox ~]$ docker save -o webserver1.tar mywebser/httpd:r1
[sunlnx@sandbox ~]$

Dockerfile:
Now that you've seen how to create and manipulate Docker containers using the command line, the preferred way to build and customize containers is actually using Dockerfiles. A Dockerfile is a small text file that contains the instructions required to construct a container. When a Dockerflle is built, each instruction adds a layer to the container in a step-by-step process. The build creates a container, runs the next instruction in that container, and then commits the container. Docker then runs the committed image as the basis for adding the next layer. The benefit of this layered approach is that Dockerfiles with the same initial instructions reuse layers.
Dockerfiles also create an easily readable and modifiable record of the steps used to create a Docker image. You can find the reference from https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/

[sunlnx@sandbox dockercfg]$ cat /home/sunlnx/dockercfg/Dockerfile
FROM centos
MAINTAINER sunlnx <sunlnx@doc.com>
RUN  yum install -y httpd
RUN echo "Web servers main page" > /var/www/html/index.html
EXPOSE 80
CMD /usr/sbin/httpd -D FOREGROUND
[sunlnx@sandbox dockercfg]$

The docker build command constructs a new Docker image from this Dockerfile, creating and removing temporary containers as needed during its step-by-step build process:

[sunlnx@sandbox dockercfg]$ docker build -t centos/httpd:r1 .
Sending build context to Docker daemon 3.584 kB
Step 1 : FROM centos
latest: Pulling from library/centos
a3ed95caeb02: Pull complete
da71393503ec: Pull complete
Digest: sha256:1a62cd7c773dd5c6cf08e2e28596f6fcc99bd97e38c9b324163e0da90ed27562
Status: Downloaded newer image for centos:latest
 ---> 904d6c400333
Step 2 : MAINTAINER sunlnx <sunlnx@doc.com>
 ---> Running in f9303082b870
 ---> fd756b44b2d3
Removing intermediate container f9303082b870
Step 3 : RUN yum install -y httpd
 ---> Running in f0affc8dc005
Loaded plugins: fastestmirror, ovl
.
.
<snip>

Complete!
 ---> d8f46afa67e1
Removing intermediate container f0affc8dc005
Step 4 : RUN echo "Web servers main page" > /var/www/html/index.html
 ---> Running in a732be9c4d06
 ---> f1825360762f
Removing intermediate container a732be9c4d06
Step 5 : EXPOSE 80
 ---> Running in 318e22854e4e
 ---> eeb133e3722a
Removing intermediate container 318e22854e4e
Step 6 : CMD /usr/sbin/httpd -D FOREGROUND
 ---> Running in 1da7959c9c03
 ---> 47416f98d5ad
Removing intermediate container 1da7959c9c03
Successfully built 47416f98d5ad
[sunlnx@sandbox dockercfg]$

[sunlnx@sandbox dockercfg]$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
centos/httpd         r1                  47416f98d5ad        28 minutes ago      311 MB

[sunlnx@sandbox ~]$ docker run -d --name centosweb -p 8085:80 centos/httpd:r1 /usr/sbin/httpd -D FOREGROUND

[sunlnx@sandbox ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                  NAMES
7779813db3df        centos/httpd:r1     "/usr/sbin/httpd -D F"   About a minute ago   Up About a minute   0.0.0.0:8085->80/tcp   centosweb

[sunlnx@sandbox ~]$ curl http://sandbox:8085
Web servers main page
[sunlnx@sandbox ~]$

More information can be found in https://docs.docker.com/ please do visit and enjoy Dockering !!

Thanks for re-sharing !

Monday, 23 May 2016

NFS common errors and troubleshooting - Linux/Unix

I have seen some of the most common NFS Error/Issues which occurs in very common now and then to most of Linux/Unix based system admins. So I decided to put at one palace. Hope this helps most of them.

Environment: Linux/Unix

Error: "Server Not Responding"

Check your NFS server and the client using RPC message and they must be functional/online. 

use ping, traceroute to check are they reaching each other, if not check your NIC using ethtool to verify IP address.

sometimes due to heavy server or network loads causes the RPC message response to time out causing error message. try to increase timeout option.

Error: "rpc mount export: RPC: Timed out " 

NFS server or client was unable to resolve DNS. check forward/reverse DNS name resolution works. 
Check your DNS servers or /etc/hosts

 Error: "Access Denied" or "Permission Denied"

check export permission for the NFS file systems.
#showmount -e nfsserver  ==> client 
#exportfs -a ==> server

check you dont have any syntax issues in file /etc/exports(e.g  space, permissions, typos..etc) 

Error: "RPC: Port mapper failure - RPC: Unable to receive"

NFS requires both NFS service and portmapper service running on both client and the server

#rpcinfo -p
       or
#/etc/init.d/portmap status

if not, start the portmap service

Error: "NFS Stale File Handle"

system call 'open' calls to access NFS file in the same way application uses local file they by returns a file descriptor or handle which programs useses I/O commands to identify the file manipulations

When an NFS file share is either unshared or NFS server changes the file handler, and any NFS client which attempts to do further I/O on the share will receive the 'NFS Stale File Handler'.

on the client :

umount -f /nfsmount or if it is unable to inmount and remount 
kill the processes which uses that /nfsmount

or 

incase if above options didn't work, you can reboot the client to clear the stale NFS.

Error: "No route to host"

this could be reported when client attempts to mount the NFS file system, even when the client can ping them successfully.

This can be due to RPC messages being filtered by either host firewall, client firewall or network switch. verify firewall rules. 
stop suing iptables and try to check the port 2049 

Hope this helps all who might use NFS most of the times. I have figured out these commonly in my experience.

Thanks for sharing !

Sunday, 15 May 2016

CentOS/RHEL 7 kernel dump & debug

Applies : CentOS / RHEL / OEL 7 

Arch : x86_64

When kdump enabled, the system is booted from the context of another kernel. This second kernel reserves a small amount of memory, and its only purpose is to capture the core dump image in case the system crashes. Since being able to analyze the core dump helps significantly to determine the exact cause of the system failure.

Configuring kdump :

kdump service comes with kexec-tools package which needs to be installed

#yum install kexec-tools

Modify the amount of memory needs to be configured for kdump and set crashkernel=<size> parameter


# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=centos/root crashkernel=128M  vconsole.keymap=us rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
#

Re-generate grub and reboot to make kernel parameter effect

# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-123.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-123.el7.x86_64.img
Warning: Please don't use old title `CentOS Linux, with Linux 3.10.0-123.el7.x86_64' for GRUB_DEFAULT, use `Advanced options for CentOS Linux>CentOS Linux, with Linux 3.10.0-123.el7.x86_64' (for versions before 2.00) or `gnulinux-advanced-1a06e03f-ad9b-44bf-a972-3a821fca1254>gnulinux-3.10.0-123.el7.x86_64-advanced-1a06e03f-ad9b-44bf-a972-3a821fca1254' (for 2.00 or later)
Found linux image: /boot/vmlinuz-0-rescue-ae1ddf63f5e04857b5e89cd8fcf1f9e1
Found initrd image: /boot/initramfs-0-rescue-ae1ddf63f5e04857b5e89cd8fcf1f9e1.img
done
#

Modify Kump in /etc/kdump.conf

By default vmcore will be stored in /var/crash directory and if you like it needs to be dumped in which ever partition or disk or you want or NFS it must be defined here.

ext3 /dev/sdd1
or
net nfs.yourdomain.com:/export/dump

compress the vmcore file to reduce the size 
core_collector makedumpfile -c

when crash is captured, root fs will be mounted and /sbin/init is run. change the behaviour as below
default reboot

Start your kdump: 

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-123.el7.x86_64 root=UUID=1a06e03f-ad9b-44bf-a972-3a821fca1254 ro rd.lvm.lv=centos/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=centos/root crashkernel=128M vconsole.keymap=us rhgb quiet

# grep -v  '#' /etc/sysconfig/kdump | sed '/^$/d'
KDUMP_KERNELVER=""
KDUMP_COMMANDLINE=""
KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug"
KEXEC_ARGS=""
KDUMP_BOOTDIR="/boot"
KDUMP_IMG="vmlinuz"
KDUMP_IMG_EXT=""
#

# systemctl enable kdump.service
# systemctl start kdump.service
# systemctl is-active kdump
active
#

Test your configuration 

# echo 1 > /proc/sys/kernel/sysrq
# echo c > /proc/sysrq-trigger



You could see that the crash was generated and we could install debug kernel packages to analyse crash. 

#yum install crash

I was able to download from https://oss.oracle.com/ol7/debuginfo/ and check your kernel version to download the version of debug kernel.

#rpm -ivh kernel-debuginfo-common-x86_64-3.10.0-123.el7.x86_64.rpm \
               kernel-debuginfo-3.10.0-123.el7.x86_64.rpm \
               kernel-debug-debuginfo-3.10.0-123.el7.x86_64.rpm

# ls -lh /var/crash/127.0.0.1-2016.05.15-04\:50\:40/vmcore
-rw-------. 1 root root 168M May 15 04:51 /var/crash/127.0.0.1-2016.05.15-04:50:40/vmcore
#

# crash /var/crash/127.0.0.1-2016.05.15-04\:50\:40/vmcore /usr/lib/debug/lib/modules/`uname -r`/vmlinux

WARNING: kernel version inconsistency between vmlinux and dumpfile

      KERNEL: /usr/lib/debug/lib/modules/3.10.0-123.el7.x86_64/vmlinux
    DUMPFILE: /var/crash/127.0.0.1-2016.05.15-04:50:40/vmcore
        CPUS: 1
        DATE: Sun May 15 04:50:38 2016
      UPTIME: 00:10:24
LOAD AVERAGE: 0.02, 0.07, 0.05
       TASKS: 104
    NODENAME: slnxcen01
     RELEASE: 3.10.0-123.el7.x86_64
     VERSION: #1 SMP Mon Jun 30 12:09:22 UTC 2014
     MACHINE: x86_64  (2294 Mhz)
      MEMORY: 1.4 GB
       PANIC: "Oops: 0002 [#1] SMP " (check log for details)
         PID: 2266
     COMMAND: "bash"
        TASK: ffff880055650b60  [THREAD_INFO: ffff880053fb2000]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)

crash>


crash> bt
PID: 2266   TASK: ffff880055650b60  CPU: 0   COMMAND: "bash"
 #0 [ffff880053fb3a98] machine_kexec at ffffffff81041181
 #1 [ffff880053fb3af0] crash_kexec at ffffffff810cf0e2
 #2 [ffff880053fb3bc0] oops_end at ffffffff815ea548
.
.
.
crash> files
PID: 2266   TASK: ffff880055650b60  CPU: 0   COMMAND: "bash"
ROOT: /    CWD: /root
 FD       FILE            DENTRY           INODE       TYPE PATH
  0 ffff880053c47a00 ffff8800563383c0 ffff880055bad2f0 CHR  /dev/tty1
  1 ffff8800542a9100 ffff88004dd4ff00 ffff88004dc0b750 REG  /proc/sysrq-trigger
.
.
.
That will conclude the article. 

References :