Sunday, 25 September 2016

Linux Containers Beginners Guide

LXC(Linux Containers) is a user-space interface for the Linux kernel container features. Through a powerful API and simple tools, it lets Linux users easily create and manage system or application containers. I tried to consolidate most of the daily usages who ever is working on the lxc in simple form.


Centos: yum install lxc lxc-templates
Ubuntu: sudo apt-get install lxc lxc-templates

Additional information: 

LXD is a container "hypervisor" founded and currently led by canonical and ubuntu with contributions from a range of other companies and individual contributors.
Some of the biggest features of LXD are :

1. Secure by design(resource restrictions ..etc )
2. Scalable ( from containers on laptop to many compute nodes..)
3. Live migration
4. Image based ... etc 

LXD uses LXC through liblxc and its go binding to create and manage containers. we could consider that to be an alternative to LXC's tools and distribution to template systems with added features that coming from being controllable over the network.

LXC Command Manual:

lxc-checkconfigcheck the current kernel for lxc support
lxc-createcreates a container, -t template shortname of an existing lxc-template script called by lxc-create. e.g ls   /usr/share/lxc/templateslxc-create -t centos -n firstcreate a "first" container using "centos" template
lxc-startrun an application inside a container  lxc-start -n firststart container "first" in background
lxc-consoleLaunch a console for the specified container (ctrl-a + q to detach)lxc-console -n firstenter container in one of the way i.e using console
lxc-attachrun the specified command inside container(bypassing console login) Kernel version >=3.8lxc-attach -n first
lxc-attach -n first -- /etc/init.d/sshd restart
spawn bash directly in the container and restart openssh
Running inside the container
lxc-configquery LXC system configurationlxc-config -l firstlets you know all the valid keys on the container
lxc-infoqueries and shows information about containerslxc-info -n firstDisplay information of first container, once you have the IP
you can SSH to container
lxc-stopBy default, it will request clean shutdown(SIGPWR) of the container to container's init process waiting for 60 sec for exit and then returning. If it fails(SIGKILL) to force shutdownlxc-stop -n first
lxc-stop -n first -k
stop container cleanly from outside incase it fails provide "-k"
lxc-autostartcontainers with = 1 to start at boot time. To edit each container has /var/lib/lxc/<container>/config edit over there. You can use Lxc-autostart to start/stop/reboot/kill when = 1 = centos belongs to centos grouplxc-autostart -r -g centosrestart containers belongs to group of centos
lxc-lslist the containers existing on the systemlxc-ls -flist all containers per line with its name,state,ipv4,ipv6 address
lxc-freezefreeze all process inside containers, useful for batch managers to
Schdule a group of processes
lxc-freeze -n firstprocess inside first container frozen, to view state lxc-ls -f
lxc-unfreezethaw all process previously frozen containerslxc-unfreeze -n firstprocess resumed in first container, view staus lxc-ls -f
lxc-devicemanage devices of running containerslxc-device [add/del] -n first /dev/ttyS0creates/deletes /dev/ttyS0 in container first based on
matching device on the host
lxc-copylxc-clone and lxc-start-ephemeral has been depreciated and lxc-copy has Been replaced. Creates and optionally starts copies of existing containers Whole root fs copied to new containerlxc-copy -n first -N clone-firstCreate a clone of first container
lxc-snapshotsnapshot existing container, i.e small copy-on-write copies of the
original container.
lxc-snapshot -n first -c snap-first
lxc-snapshot -n first -L -C
Lxc-snapshot -n first -r snap-first
lxc-snapshot -n first -r snap0 firstsnap0
capture the snapshot
confirm your snapshot created
revert container at a later point of time if you wish
restore snapshot as its own container
lxc-topdisplays container statistics.for performance reasons kernel does not account kernel memory unless kernel memory set
lxc-destroydestroys the system object previously created by the lxc-create command.
Stop container before destroying
lxc-destroy -n firstDestroy first container

LXC Reference : 

Thanks for re-sharing !

Monday, 22 August 2016

Compile and install new kernel on CentOS

Objective: Compile and install new kernel 

OS: CentOS 6.7

Current kernel version  : 2.6.32
Upgraded kernel version : 3.14.77

Download the latest kernel version from to /tmp.

1. Make sure you install "Development Tools" using yum which includes gcc packages which are required for compiling kernel. 

[root@centos ~]# yum groupinstall "Development Tools"
Loaded plugins: fastestmirror, security
Setting up Group Process
Loading mirror speeds from cached hostfile
Package 1:make-3.81-20.el6.x86_64 already installed and latest version

2. Extract the newer kernel to /usr/src directory and create an soft link for 'linux' directory.  That would be the directory which is required for you to compile your newer kernel.

[root@centos ~]# tar -Jxvf linux-4.7.2.tar.xz -C /usr/src/
[root@centos src]# ln -s linux-4.7.2/ linux
[root@centos src]# ls
debug  kernels  linux  linux-4.7.2
[root@centos src]# cd linux
[root@centos linux]# ls
arch   certs    CREDITS  Documentation  firmware  include  ipc     Kconfig  lib          Makefile  net     REPORTING-BUGS  scripts   sound  usr
block  COPYING  crypto   drivers        fs        init     Kbuild  kernel   MAINTAINERS  mm        README  samples         security  tools  virt
[root@centos linux]#

3. Since I have only newer kernel and not much to be cleaned for older unnecessary modules, I would choose 'mid-range' i...e mrproper. You can choose depending on your compilations. for more info type 'make help' 

Cleaning targets:
  clean           - Remove most generated files but keep the config and
                    enough build support to build external modules
  mrproper        - Remove all generated files + config + various backup files
  distclean       - mrproper + remove editor backup and patch files

[root@centos linux]# make mrproper
[root@centos linux]#

4. Despite installing "Development Tools", you additional are required to install 'ncurses-devel' for menuconfig (custom configuration of the new kernel i..e which modules needs to be compiled or loadable by kernel) and after saving you it would create a .config file. 

[root@centos linux]#make menuconfig

The one chosen with * modules will be compiled and others which are unchecked they wouldn't be compiled. Hence un-necessary modules which are not required for your system need not be checked here. so you can only choose what your system wishes to do.

5. Compile your actual kernel, which will take around 15-20 mins depending on your number of CPU.

[root@centos linux]# make bzImage
  HOSTCC  scripts/kconfig/conf.o
  HOSTLD  scripts/kconfig/conf
scripts/kconfig/conf  --silentoldconfig Kconfig
  BUILD   arch/x86/boot/bzImage
Setup is 15164 bytes (padded to 15360 bytes).
System is 4374 kB
CRC 96b7ee84
Kernel: arch/x86/boot/bzImage is ready  (#1)
[root@centos linux]#

6. compile your modules would considerably takes more time than kernel ( approx 60-90 mins ) depending on your nummber of CPU

root@centos linux]# make modules
make[1]: Nothing to be done for `all'.
make[1]: Nothing to be done for `relocs'.
  CHK     include/config/kernel.release
  CHK     include/generated/uapi/linux/version.h
  CHK     include/generated/utsrelease.h
  CALL    scripts/
  H16TOFW firmware/edgeport/down2.fw
  IHEX    firmware/edgeport/down3.bin
  IHEX2FW firmware/whiteheat_loader.fw
  IHEX2FW firmware/whiteheat.fw
  IHEX2FW firmware/keyspan_pda/keyspan_pda.fw
  IHEX2FW firmware/keyspan_pda/xircom_pgs.fw
[root@centos linux]#

7. copies all the modules to a new directory .. /lib/modules/3.14.77/

[root@centos linux]# make modules_install
  INSTALL arch/x86/crypto/aes-x86_64.ko
  INSTALL arch/x86/crypto/aesni-intel.ko
  INSTALL arch/x86/crypto/crc32c-intel.ko
  INSTALL arch/x86/crypto/crct10dif-pclmul.ko
  INSTALL /lib/firmware/keyspan_pda/keyspan_pda.fw
  INSTALL /lib/firmware/keyspan_pda/xircom_pgs.fw
  DEPMOD  3.14.77
[root@centos linux]#

8. Move kernel to the right location and name it correctly , update initramfs, grub so boot up with new kernel.

[root@centos linux]# make install
sh /usr/src/linux-3.14.77/arch/x86/boot/ 3.14.77 arch/x86/boot/bzImage \
ERROR: modinfo: could not find module parport
ERROR: modinfo: could not find module snd_page_alloc
[root@centos linux]#

On summary, below are the commands used for building newer kernel.
make mrproper
make menuconfig
make bzImage
make modules
make modules_install
make install

9 Reboot system and boot into new kernel from the grub menu.

[root@centos linux]# reboot

[root@centos ~]# uptime
 10:39:41 up 0 min,  2 users,  load average: 0.00, 0.00, 0.00
[root@centos ~]#

[root@centos ~]# uname -r
[root@centos ~]#

Thanks for re-sharing. 

Saturday, 30 July 2016

Troubleshoot network problem using tshark

How tshark works ?

When a packet arrives at the network card, the MAC destination address is checked to see if it matches yours, in which case an interrupt service routine will be generated and handled by the network driver. 

Subsequently, the received data is copied to a memory block defined in the kernel and from there it will be processed by the corresponding protocol stack to be delivered to the appropriate application in user space. Parallel to this process, when Tshark is capturing traffic, the network driver sends a copy of the packets to a kernel subsystem called Packet Filter, which will filter and store in a buffer the desired packets. These packets will be received by Dumpcap (in user space) whose main goal will be to write them into a libpcap file format to be subsequently read by Tshark. As new packets arrive, Dumpcap will add them to the same capture file and it will notify Tshark about their arrival so that they can be processed.

My objective would be to give you brief tutorial on how to find problems related to performance of network, could be due to bandwidth etc.. so we could use tshark to try and find out which hosts are generating more traffic and what type of data are they sending..

List all the network interfaces - tshark -D

Capture traffic from network interface and write to file -
#tshark -i <interface> -w traffic.pacap

How to capture and analyze traffic using tshark ? 

1. Determine which IPs in your VLAN(IPADDRES/NETMASK) could be misusing the network would be able to get IP list. list by dfault would be sorted according to total number of frames, so it could give an idea of heavy talkers.

#tshark -r traffic.pcap -q -z "conv,ip,ip.addr=="

IPv4 Conversations
                                               |       <-      | |       ->      | |     Total     |   Rel. Start   |   Duration   |
                                               | Frames  Bytes | | Frames  Bytes | | Frames  Bytes |                |              |       <->                105     27191     129     21393     234     48584   112.306444555       260.0255        <->                 34      3395      36     11639      70     15034   263.618378290       108.6899        <->                 32      3601      37     11601      69     15202   109.882120656       177.0934

2. With above inforamtion we know that IP represents one of the host which is generating more traffic to communicate with other machines on the network

You could create another pcap file just with the traffic generated by that machine(

#tshark -r traffic.pcap -R "ip.addr==" -w ip.pcap
# capinfos ip.pcap | grep "Number\|time:"
Number of packets:   234
Start time:          Fri Jul 29 20:37:12 2016
End time:            Fri Jul 29 20:41:32 2016

3. Check that your host is not breaking any of your policies of your network, only HTTP & HTTPS is allowed. Below commands will tells us outbound connections to ports other than any (HTTP or HTTPS)

#tshark -o column.format:'" Source ","%s","Destination","%d", "dstport", "%uD","Protocol", "%p"' -r ip.pcap -R "ip.src == && ! dns && tcp.dstport != 80 && tcp.dstport != 443"  | sort -u ->    43536 TCP ->    43536 TLSv1.2 ->    43540 TCP ->    43540 TLSv1.2

4. I don't have any traffic violating my policies, anyway lets suppose we say if that do exists, then we would have those machines IP address and the port on which they are connected. so to make sure that the traffic is not from other service using the FTP port, lauch tcp stream of that session.

#tshark -o column.format:'"Source","%s","srcport", "%uS","Destination","%d", "dstport", "%uD","Protocol", "%p"' -r ip.pcap -R "tcp.dstport == 43536" | head -1 443    43536 TCP

#tshark -r ip.pcap -q -z  "follow,tcp,ascii,,,1"
Follow: tcp,ascii
Filter: ((ip.src eq and tcp.srcport eq 443) and (ip.dst eq and tcp.dstport eq 43536)) or ((ip.src eq and tcp.srcport eq 43536) and (ip.dst eq and tcp.dstport eq 443))

5. Now you could observe that it was "" was actullay consuming more bandwidth responsible for slowdown in network.

If you do come across any FTP sessions, troubleshoot the above way, also additionally you will check all the files downloaded by the client.

#tshark -r ip.pcap -q -z  "follow,tcp,ascii,,<Destination machine>:21,1" | grep RETR

6. tshark also allows us to break down each of the protocols captured. Thus we can see hierarchically the number of frames and bytes associated with each protocol. Using capture file, let's see for example the distribution of HTTP and HTTPS traffic used by the IP

#tshark -r traffic.pcap -q -z io,phs,"ip.addr== && ssl || http"
Protocol Hierarchy Statistics
Filter: ip.addr== && ssl || http

eth                                      frames:122 bytes:40644
  ip                                     frames:122 bytes:40644
    tcp                                  frames:122 bytes:40644
      ssl                                frames:122 bytes:40644
        tcp.segments                     frames:2 bytes:2589
          ssl                            frames:2 bytes:2589

7. It would practically tells us that SSL represents all traffic, let's see the IP's associated with that communication.

#tshark -o column.format:'"destination","%d"' -r  traffic.pcap -R "ip.src == && ssl"| sort -u

#whois | grep -i "netname\|netrange"
NetRange: -

With whatever application or information your would get for the IP address/ports, you can create ACLs or IPtables rules to deny certain types of traffic, do a shutdown of a specific port, limit the bandwidth of some protocols so on ...

More references : 

Thanks for re-sharing !

Sunday, 19 June 2016

Docker Basics & Container Customization - Linux

Learn how to customize a Docker container image and use it to instantiate application instances across different Linux servers

Docker captures full application environment into a virtual container that can be deployed across different Linux servers. System administrators and software developers are learning that Docker can help them deploy application images on Linux quickly, reliably, and consistently without dependency and portability problems. Docker containers can define application and its dependencies using small text file(Dockerfile) that can be moved to different Linux releases and quickly rebuilt.  Also Dockerized application are very easy to migrate to another different linux servers either executed as a bare metal in a virtual machine or Linux instances in the cloud.

I would demonstrate how to create Docker container on RHEL 7, modify and use to deploy multiple application instance.  Docker containers are a lightweight virtualization technology for Linux. They provide isolation from other applications and processes running on the same system but make system calls to the same shared Linux kernel, similar to Linux LXC application containers. Docker containers have their own namespace, so they are fully isolated from one another—processes running in one container can't see or impact processes running in another. By default, each container gets its own networking stack, private network interfaces, and IP address, and Docker creates a virtual bridge so containers can communicate.

Getting Started 
I am usig Docker installation on Redhat 7.2 and installtion document can be found at

You could have your own Docker hub repository to store images that can be used to build running containers. I would pull few of the images from the Docker hub repository for test environment.

[sunlnx@sandbox ~]$docker pull ubuntu:latest
[sunlnx@sandbox ~]$docker pull oraclelinux:6
[sunlnx@sandbox ~]$docker pull oraclelinux:7
[sunlnx@sandbox ~]$docker pull rhel:latest
[sunlnx@sandbox ~]$docker pull mysql/mysql-server
[sunlnx@sandbox ~]$docker pull nginx:latest

To list all the docker images that were pulled above 
[sunlnx@sandbox ~]$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
nginx                latest              0d409d33b27e        2 weeks ago         182.7 MB
ubuntu               latest              2fa927b5cdd3        3 weeks ago         122 MB
oraclelinux          6                   768a3d7b605a        4 weeks ago         222.8 MB
oraclelinux          7                   df602a268e64        5 weeks ago         276.1 MB
rhel                 latest              bf2034427837        6 weeks ago         203.4 MB
mysql/mysql-server   latest              18a962a188ee        11 days ago         366.9 MB
[sunlnx@sandbox ~]$

Container Customization 
I would like to provide multiple, identical web servers across multiple Linux servers, Docker makes it easy to create a preconfigured in a container image. I would then use this pre built image and deploy it across one or many other Linux hosts. I would install "myweb" container and would configure that to deliver web content to the clients. In order to customize I would get an interactive bash shell to run an rhel "myweb" container. 

[sunlnx@sandbox ~]$ docker run -it --name myweb oraclelinux:6 /bin/bash
[root@5b62adeb3abb /]#

In a shell on my Linux host, the docker ps command shows information about the running guest container, 

[sunlnx@sandbox ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
5b62adeb3abb        oraclelinux:6       "/bin/bash"         6 minutes ago       Up 20 seconds                           myweb
[sunlnx@sandbox ~]$

on myweb, I will install httpd using yum and would configure the web server to display. I will create an index.html in /var/www/html on it. 

[root@5b62adeb3abb /]# yum install -y httpd
[root@5b62adeb3abb /]# echo "Web servers main page" > /var/www/html/index.html
[root@5b62adeb3abb /]# exit

Now, I want to create a new Docker image that reflects the contents of the guest container that I just configured. The following docker commit command captures the modified container into a new image named mywebser/httpd:r1

[sunlnx@sandbox ~]$ docker commit -m "ol6-httpd" `docker ps -l -q` mywebser/httpd:r1
[sunlnx@sandbox ~]$

The commit command takes as input the image ID number of the myweb container and assigns and returns an ID number for the new image. Running the docker images command now lists the new image mywebser/httpd 

[sunlnx@sandbox ~]$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED              SIZE
mywebser/httpd       r1                  79cf91b1a67f        About a minute ago   766 MB

Incase if I don't require this container I can remove with docker rm command

[sunlnx@sandbox ~]$docker rm myweb 

Because Docker containers persist even though they're no longer running, removing unneeded containers is simply a housekeeping step to reduce clutter on my host, and it allows me to reuse the name myweb1 for a new container.

Deploy Docker Image:
I can deploy any number of web servers now using the new Docker image as a template. The following docker run commands run the container image mywebser/httpd:r1, creating the containers myweb1, myweb2, myweb3, myweb4 and myweb5 executing httpd in each one:

[sunlnx@sandbox ~]$ docker run -d --name myweb1 -p 8080:80 mywebser/httpd:r1 /usr/sbin/httpd -D FOREGROUND
[sunlnx@sandbox ~]$ docker run -d --name myweb2 -p 8081:80 mywebser/httpd:r1 /usr/sbin/httpd -D FOREGROUND
[sunlnx@sandbox ~]$ docker run -d --name myweb3 -p 8082:80 mywebser/httpd:r1 /usr/sbin/httpd -D FOREGROUND
[sunlnx@sandbox ~]$ docker run -d --name myweb4 -p 8083:80 mywebser/httpd:r1 /usr/sbin/httpd -D FOREGROUND
[sunlnx@sandbox ~]$ docker run -d --name myweb5 -p 8084:80 mywebser/httpd:r1 /usr/sbin/httpd -D FOREGROUND
[sunlnx@sandbox ~]$

[sunlnx@sandbox ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                  NAMES
92bbf522aa41        mywebser/httpd:r1   "/usr/sbin/httpd -D F"   About a minute ago   Up About a minute>80/tcp   myweb5
a5e970efd3f3        mywebser/httpd:r1   "/usr/sbin/httpd -D F"   About a minute ago   Up About a minute>80/tcp   myweb4
48964b1e06b2        mywebser/httpd:r1   "/usr/sbin/httpd -D F"   About a minute ago   Up About a minute>80/tcp   myweb3
2fc28962e5ab        mywebser/httpd:r1   "/usr/sbin/httpd -D F"   About a minute ago   Up About a minute>80/tcp   myweb2
924018f9f737        mywebser/httpd:r1   "/usr/sbin/httpd -D F"   About a minute ago   Up About a minute>80/tcp   myweb1
[sunlnx@sandbox ~]$

Using a web browser or curl, I can test the web server running in each guest:

[sunlnx@sandbox ~]$ curl http://sandbox:8080
Web servers main page
[sunlnx@sandbox ~]$ curl http://sandbox:8081
Web servers main page
[sunlnx@sandbox ~]$ curl http://sandbox:8082
Web servers main page
[sunlnx@sandbox ~]$ curl http://sandbox:8083
Web servers main page
[sunlnx@sandbox ~]$ curl http://sandbox:8084
Web servers main page

The Docker Engine also assigns each running container a virtual network interface, which you can see with the docker inspect command:
[sunlnx@sandbox ~]$ docker inspect myweb1
[sunlnx@sandbox ~]$ docker inspect -f '{{ .NetworkSettings.IPAddress }}' myweb1
[sunlnx@sandbox ~]$

Saving Docker image:
You could backup the image to a tar using docker command

[sunlnx@sandbox ~]$ docker save -o webserver1.tar mywebser/httpd:r1
[sunlnx@sandbox ~]$

Now that you've seen how to create and manipulate Docker containers using the command line, the preferred way to build and customize containers is actually using Dockerfiles. A Dockerfile is a small text file that contains the instructions required to construct a container. When a Dockerflle is built, each instruction adds a layer to the container in a step-by-step process. The build creates a container, runs the next instruction in that container, and then commits the container. Docker then runs the committed image as the basis for adding the next layer. The benefit of this layered approach is that Dockerfiles with the same initial instructions reuse layers.
Dockerfiles also create an easily readable and modifiable record of the steps used to create a Docker image. You can find the reference from

[sunlnx@sandbox dockercfg]$ cat /home/sunlnx/dockercfg/Dockerfile
FROM centos
MAINTAINER sunlnx <>
RUN  yum install -y httpd
RUN echo "Web servers main page" > /var/www/html/index.html
CMD /usr/sbin/httpd -D FOREGROUND
[sunlnx@sandbox dockercfg]$

The docker build command constructs a new Docker image from this Dockerfile, creating and removing temporary containers as needed during its step-by-step build process:

[sunlnx@sandbox dockercfg]$ docker build -t centos/httpd:r1 .
Sending build context to Docker daemon 3.584 kB
Step 1 : FROM centos
latest: Pulling from library/centos
a3ed95caeb02: Pull complete
da71393503ec: Pull complete
Digest: sha256:1a62cd7c773dd5c6cf08e2e28596f6fcc99bd97e38c9b324163e0da90ed27562
Status: Downloaded newer image for centos:latest
 ---> 904d6c400333
Step 2 : MAINTAINER sunlnx <>
 ---> Running in f9303082b870
 ---> fd756b44b2d3
Removing intermediate container f9303082b870
Step 3 : RUN yum install -y httpd
 ---> Running in f0affc8dc005
Loaded plugins: fastestmirror, ovl

 ---> d8f46afa67e1
Removing intermediate container f0affc8dc005
Step 4 : RUN echo "Web servers main page" > /var/www/html/index.html
 ---> Running in a732be9c4d06
 ---> f1825360762f
Removing intermediate container a732be9c4d06
Step 5 : EXPOSE 80
 ---> Running in 318e22854e4e
 ---> eeb133e3722a
Removing intermediate container 318e22854e4e
Step 6 : CMD /usr/sbin/httpd -D FOREGROUND
 ---> Running in 1da7959c9c03
 ---> 47416f98d5ad
Removing intermediate container 1da7959c9c03
Successfully built 47416f98d5ad
[sunlnx@sandbox dockercfg]$

[sunlnx@sandbox dockercfg]$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
centos/httpd         r1                  47416f98d5ad        28 minutes ago      311 MB

[sunlnx@sandbox ~]$ docker run -d --name centosweb -p 8085:80 centos/httpd:r1 /usr/sbin/httpd -D FOREGROUND

[sunlnx@sandbox ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                  NAMES
7779813db3df        centos/httpd:r1     "/usr/sbin/httpd -D F"   About a minute ago   Up About a minute>80/tcp   centosweb

[sunlnx@sandbox ~]$ curl http://sandbox:8085
Web servers main page
[sunlnx@sandbox ~]$

More information can be found in please do visit and enjoy Dockering !!

Thanks for re-sharing !