Wednesday 31 December 2014

strace & system call tracing

I had written this as few of my colleagues had requested to know how strace works, hope below articles helps out.

Today being the last day of 2014, I thought to share publicly and wishing all readers HAPPY  NEW  YEAR - 2015 " 

The strace tool is one of the most powerful problem determination tools available for Linux. It traces the thin layer (the system calls) between a process and the Linux kernel. System call tracing is particularly useful as a first investigation tool or for problems that involve a call to the operating system.

A system call is a special type of function that is run inside the kernel. It provides fair and secure access to system resources such as disk, network, and memory. System calls also provide access to kernel services such as inter-process communication and system information.

When to use ?

The strace tool should be used as a first investigation tool or for problems that are related or involved  at the operating system level. the system call trace will clearly show the cause of problem. Experienced users might use strace either way until they narrow down the scope of a problem.

The following example uses a simple program to show how to use strace, I would try to open an file which doesn't exist.


root@localhost]# cat main.c 

#include<sys/types.h>
#include<sys/stat.h>
#include<fcntl.h>

int main()
{

int fd;
int i=0;

fd = open( "/tmp/foo", O_RDONLY);

if (fd < 0)
i=5;
else
i=2;

return i;
}

# gcc main.c -o ./main
# strace -o main.strace ./main
# cat -n ./main.strace 
     1 execve("./main", ["./main"], [/* 21 vars */]) = 0
     2 brk(0)                                  = 0x8697000
     3 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7774000
     4 access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
     5 open("/etc/ld.so.cache", O_RDONLY)      = 3
     6 fstat64(3, {st_mode=S_IFREG|0644, st_size=28116, ...}) = 0
     7 mmap2(NULL, 28116, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb776d000
     8 close(3)                                = 0
     9 open("/lib/libc.so.6", O_RDONLY)        = 3
    10 read(3, "\177ELF\1\1\1\3\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220\356\300\0004\0\0\0"..., 512) = 512
    11 fstat64(3, {st_mode=S_IFREG|0755, st_size=1906308, ...}) = 0
    12 mmap2(0xbf8000, 1661356, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xbf8000
    13 mmap2(0xd88000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x190) = 0xd88000
    14 mmap2(0xd8b000, 10668, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xd8b000
    15 close(3)                                = 0
    16 mprotect(0xd88000, 8192, PROT_READ)     = 0
    17 mprotect(0xbe6000, 4096, PROT_READ)     = 0
    18 munmap(0xb776d000, 28116)               = 0
    19 open("/tmp/foo", O_RDONLY)              = -1 ENOENT (No such file or directory)
    20 exit_group(5)                           = ?

In this strace output, the vast majority of the system calls are actually for process initialization. In fact, the only system call (on line 19) from the actual program code is open("/tmp/foo", O _ RDONLY ). Also notice that there are no system calls from the if statement or any other code in the program because the if statement does not invoke a system call.

Below would be detail how above code works line by line: 

Line #1:  The execve system call (or one of the exec system calls) is always the first system call in the strace output if strace is used to trace a program off the command line. The strace tool forks, executes the program, and the exec system call actually returns as the first system call in the new process.


Line #2: The brk system call is called with an argument of zero to find the current "break point." This is the beginning of memory management for the process.

Line #3: The mmap call is used to create an anonymous 4KB page. The address of this page is at 0xb7774000

Line #4: This line attempts to open the ld.so.preload file. This file contains a list of ELF shared libraries that are to be pre-loaded before a program is able to run.

Line #5-#9: These lines involve finding and loading the libc library.

Line #10: Loads in the ELF header for the libc library.

Line #11: Gets more information (including size) for the libc library file.

Line #12:  This line actually loads ( mmaps ) the contents of libc into memory at address at 0xbf8000

Line #13: This line loads the data section at address 0xd88000 for 12288 bytes, from the beginning of memory segment (0x00d88000). According to the ELF layout of libc.so.6, the data section starts at 0x00d881c8 , but that section must be aligned on 0x1000 boundaries (hence the offset of 0x00d88000 )

# readelf -l /lib/libc.so.6

Elf file type is DYN (Shared object file) Entry point 0xc0ee90 There are 10 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  PHDR           0x000034 0x00bf8034 0x00bf8034 0x00140 0x00140 R E 0x4
  INTERP         0x15c2c8 0x00d542c8 0x00d542c8 0x00013 0x00013 R   0x1
      [Requesting program interpreter: /lib/ld-linux.so.2]
  LOAD           0x000000 0x00bf8000 0x00bf8000 0x18ff98 0x18ff98 R E 0x1000
  LOAD           0x1901c8 0x00d881c8 0x00d881c8 0x027d4 0x057e4 RW  0x1000
  DYNAMIC        0x191d7c 0x00d89d7c 0x00d89d7c 0x000f8 0x000f8 RW  0x4

Line #14: Creates an anonymous memory segment for the bss section This is a special section of a loaded executable or shared library for uninitialized data. Because the data is not initialized, the storage for it is not included in an ELF object like a shared library (there are no real data values to store). Instead, memory is allocated(0xd8b000)for the bss section when the library is loaded.

Line #15: Closes the file descriptor for libc.

Line #16-#17: This removes any protection for a region of memory at 0xd88000

Line #18: unmap files or devices into memory

Line #19: The only system call from the actual program code.

Line #20: Exits the process with a return code of 5.

It can also be useful to time both the difference between system call entries and the time spent in the system calls. With this information, it is possible to get the time spent in the user code between the system calls.


# strace -Tr ./main
     0.000000 execve("./main", ["./main"], [/* 21 vars */]) = 0 <0.000169>
     0.000406 brk(0)                    = 0x8b26000 <0.000014>
     0.000194 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77e3000 <0.000028>
     0.000111 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000022>
     0.000117 open("/etc/ld.so.cache", O_RDONLY) = 3 <0.000022>
     0.000068 fstat64(3, {st_mode=S_IFREG|0644, st_size=28116, ...}) = 0 <0.000000>
.
.

Another useful way to time system calls is with the -c switch. This switch summarizes the output in tabular form:

# strace -c ./main
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  -nan    0.000000           0         1           read
  -nan    0.000000           0         3         1 open
  -nan    0.000000           0         2           close
  -nan    0.000000           0         1           execve
  -nan    0.000000           0         1         1 access
.
.

Sometimes it is necessary to trace an existing process that is running, such as a Web daemon (such as apache) or xinetd. The strace tool provides a simple way to attach to running processes with the -p switch:

# strace -p <PID>

I would conclude this article, by letting everyone know that strace when used effectively can narrow the scope of the problem analysis.  

Tuesday 23 December 2014

automate kickstart installations in spacewalk #Redhat 6 / CentOS 6



In my previous article ( http://sunlnx.blogspot.in/2014/12/installation-of-spacewalk-centos-66_2.html ), I had written on how to configure and install spacewalk. Here, I would let you know on how to automate the installations using spacewalk.

My Environment:
Hostname: spacewalk
Environment: CentOS 6.6 x86_64
Spacewalk version: 2.2​

For automating the installation of a Linux system a method called kickstart can be used. First, we have to setup a directory structure on the spacewalk server, copy the directory of the below from your CentOS/Redhat DVD to /var/distro-trees/CentOS6.6-x86_64.

- images
- isolinux
- repodata

Next, open the spacewalk console and navigate to the following location:

systems -> kickstart -> distributions -> new distribution.




Next step is to create a kickstart profile for the channel and distribution. Open the spacewalk console and navigate to the following location:

systems -> kickstart -> create new kickstart profile

Enter the following parameters for the new kickstart profile:

Label: CentOS66-minimal
Base channel: CentOS 6.6 Base - x86_64
Kickstartable tree: CentOS6.6-x86_64
Virtualization type: none

Also, have a look at the other tabs to have an idea of the configuration options which are available, possible interesting areas are provided as below snap where it is self-explanatory reader could work themselves.



We are now creating a new virtual machine, so make sure that your spacewalk server is able to resolve the name resolution. I would leave this to reader to configure DNS.

Next, build an ISO image as described below, on which your ISO image(generated.iso) from where you executed the command.


[root@spacewalk ~]# cobbler buildiso
task started: 2014-12-23_112833_buildiso
task started (id=Build Iso, time=Tue Dec 23 11:28:33 2014)
using/creating buildisodir: /var/cache/cobbler/buildiso
building tree for isolinux
copying miscellaneous files
copying kernels and initrds for profiles
generating a isolinux.cfg
generating profile list
done writing config
running: mkisofs -o /root/generated.iso -r -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -V Cobbler\ Install -R -J -T /var/cache/cobbler/buildiso
received on stdout: 
received on stderr: I: -input-charset not specified, using utf-8 (detected in locale settings)
Size of boot image is 4 sectors -> No emulation
 26.00% done, estimate finish Tue Dec 23 11:28:34 2014
 51.90% done, estimate finish Tue Dec 23 11:28:34 2014
 77.89% done, estimate finish Tue Dec 23 11:28:34 2014
Total translation table size: 4029
Total rockridge attributes bytes: 1320
Total directory bytes: 4700
Path table size(bytes): 40
Max brk space used 1b000
19272 extents written (37 MB)

ISO build complete
You may wish to delete: /var/cache/cobbler/buildiso
The output file is: /root/generated.iso
*** TASK COMPLETE ***
[root@spacewalk ~]#


[root@spacewalk ~]# ls -l generated.iso
-rw-r--r-- 1 root root 39469056 Dec 23 11:28 generated.iso
[root@spacewalk ~]# 

On your host, create a new virtual machine and provide it with the generated.iso file to boot from. Upon boot you will see a menu allowing you to specify the Centos66-minimal setup to be installed.



Select this entry and the setup will install a base 64 bit CentOS 6.6 Linux system. If all goes well, this will happen completely automated, without any user intervention whatsoever.

I, would configure new server as a client to the spacewalk server in coming articles.

Tuesday 2 December 2014

Install/Configure spacewalk - CentOS 6.6


Spacewalk is an open source Linux systems management solution.Spacewalk is the upstream community project from which the Red Hat Satellite product is derived​.

Differences between Spacewalk and Red Hat Satellite​:



What Can Spacewalk Do?​

- Inventory your systems (hardware and software information)
- Install and update software on your systems
- Collect and distribute your custom software packages into manageable groups
- Provision (kickstart) your systems
- Manage and deploy configuration files to your systems
- Monitor your systems
- Provision and start/stop/configure virtual guests
- Distribute content across multiple geographical sites in an efficient manner.


Prerequisites:

- Outbound open ports 80, 443, 4545 (only if you want to enable monitoring)
- Inbound open ports 80, 443, 5222 (only if you want to push actions to client machines) and 5269 (only for push actions to a Spacewalk Proxy), 69 udp if you want to use tftp
- Storage for database: 250 KiB per client system + 500 KiB per channel + 230 KiB per package in -channel (i.e. 1.1GiB for channel with 5000 packages)
- Storage for packages (default /var/satellite): Depends on what you're storing; Red Hat recommend 6GB per channel for their channels
- 2GB RAM minimum, 4GB recommended
- Make sure your underlying OS is fully up-to-date.​


​My Environment :
Hostname: spacewalk
Environment: CentOS 6.6 x86_64
Spacewalk version : 2.2​

Setting up Spacewalk repo:
​​
[root@spacewalk ~]# rpm -Uvh http://yum.spacewalkproject.org/2.2/RHEL/6/x86_64/spacewalk-repo-2.2-1.el6.noarch.rpm
Retrieving http://yum.spacewalkproject.org/2.2/RHEL/6/x86_64/spacewalk-repo-2.2-1.el6.noarch.rpm
warning: /var/tmp/rpm-tmp.rexVMa: Header V4 DSA/SHA1 Signature, key ID 863a853d: NOKEY
Preparing...                ########################################### [100%]
   1:spacewalk-repo         ########################################### [100%]

[root@spacewalk ~]# 

Jpackage repository:

For Spacewalk on EL 6, a couple of additional dependencies are needed from jpackage configure the following yum repository before beginning your Spacewalk installation
​.
​​
root@spacewalk:~]# cat > /etc/yum.repos.d/jpackage-generic.repo << EOF
[jpackage-generic]
name=JPackage generic
#baseurl=http://mirrors.dotsrc.org/pub/jpackage/5.0/generic/free/
mirrorlist=http://www.jpackage.org/mirrorlist.php?dist=generic&type=free&release=5.0
enabled=1
gpgcheck=1
gpgkey=http://www.jpackage.org/jpackage.asc
EOF

root@spacewalk:~]# 

Spacewalk requires a Java Virtual Machine with version 1.6.0 or greater. ​EPEL - Extra Packages for Enterprise Linux contains a version of the openjdk that works with Spacewalk. Other dependencies can get installed from EPEL as well. To get packages from EPEL just install this RPM:



Retrieving http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
warning: /var/tmp/rpm-tmp.1MCWEN: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing...                ########################################### [100%]
   1:epel-release           ########################################### [100%]

​​[root@spacewalk ~]# 

Database server:
You can let Spacewalk setup the PostgreSQL server on your machine without any manual intervention.​
​​
[root@spacewalk ~]# yum install spacewalk-setup-postgresql
Loaded plugins: fastestmirror, security
Setting up Install Process
.
.
Dependency Installed:
  postgresql-pltcl.x86_64 0:8.4.20-1.el6_5                            postgresql-server.x86_64 0:8.4.20-1.el6_5                            tcl.x86_64 1:8.5.7-6.el6                           
Complete!

[root@spacewalk ~]# 

Installing Spacewalk:

yum to install the necessary packages. This will pull down and install the set of RPMs required to get Spacewalk to run
​​
[root@spacewalk ~]# yum install spacewalk-postgresql
Loaded plugins: fastestmirror, security
Setting up Install Process
Loading mirror speeds from cached hostfile
.
.
Transaction Summary
==============================================================================================================================================================================================
Install     323 Package(s)
Upgrade       2 Package(s)

Total download size: 241 M
Is this ok [y/N]:y
.
.
.
warning: rpmts_HdrFromFdno: Header V4 DSA/SHA1 Signature, key ID 863a853d: NOKEY
Retrieving key from http://yum.spacewalkproject.org/RPM-GPG-KEY-spacewalk-2012
Importing GPG key 0x863A853D:
 Userid: "Spacewalk <spacewalk-devel@redhat.com>"
 From  : http://yum.spacewalkproject.org/RPM-GPG-KEY-spacewalk-2012
Is this ok [y/N]: y
warning: rpmts_HdrFromFdno: Header V3 DSA/SHA1 Signature, key ID c431416d: NOKEY
Retrieving key from http://www.jpackage.org/jpackage.asc
Importing GPG key 0xC431416D:
 Userid: "JPackage Project (JPP Official Keys) <jpackage@zarb.org>"
 From  : http://www.jpackage.org/jpackage.asc
Is this ok [y/N]: y
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID c105b9de: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
Importing GPG key 0xC105B9DE:
 Userid : CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org>
 Package: centos-release-6-6.el6.centos.12.2.x86_64 (@anaconda-CentOS-201410241409.x86_64/6.6)
 From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
Is this ok [y/N]: y
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
Importing GPG key 0x0608B895:
 Userid : EPEL (6) <epel@fedoraproject.org>
 Package: epel-release-6-8.noarch (installed)
 From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
  Installing : libgcj-4.4.7-11.el6.x86_64                                                                                                                                              1/327 
  Updating   : 1:java-1.6.0-openjdk-1.6.0.33-1.13.5.1.el6_6.x86_64        
.
.
.
Dependency Updated:
  java-1.6.0-openjdk.x86_64 1:1.6.0.33-1.13.5.1.el6_6                                              policycoreutils.x86_64 0:2.0.83-19.47.el6_6.1                                             
Complete!

[root@spacewalk ~]#

Configuring Spacewalk:
​​
[root@spacewalk ~]# spacewalk-setup --disconnected
** Database: Setting up database connection for PostgreSQL backend.
** Database: Installing the database:
** Database: This is a long process that is logged in:
** Database:   /var/log/rhn/install_db.log
*** Progress: #
** Database: Installation complete.
** Database: Populating database.
*** Progress: #####################################
* Setting up users and groups.
** GPG: Initializing GPG and importing key.
** GPG: Creating /root/.gnupg directory
You must enter an email address.
Admin Email Address? root@localhost
* Performing initial configuration.
* Activating Spacewalk.
** Loading Spacewalk Certificate.
** Verifying certificate locally.
** Activating Spacewalk.
* Enabling Monitoring.
* Configuring apache SSL virtual host.
Should setup configure apache's default ssl server for you (saves original ssl.conf) [Y]? y
** /etc/httpd/conf.d/ssl.conf has been backed up to ssl.conf-swsave
* Configuring tomcat.
* Configuring jabberd.
* Creating SSL certificates.
CA certificate password? 
Re-enter CA certificate password? 
Organization? oratestlabs.com
Organization Unit [spacewalk]?                   
Email Address [root@localhost]? 
City? bancross
State? KA
Country code (Examples: "US", "JP", "IN", or type "?" to see a list)? IN
** SSL: Generating CA certificate.
** SSL: Deploying CA certificate.
** SSL: Generating server certificate.
** SSL: Storing SSL certificates.
* Deploying configuration files.
* Update configuration in database.
* Setting up Cobbler..
Processing /etc/cobbler/modules.conf
`/etc/cobbler/modules.conf' -> `/etc/cobbler/modules.conf-swsave'
Processing /etc/cobbler/settings
`/etc/cobbler/settings' -> `/etc/cobbler/settings-swsave'
Cobbler requires tftp and xinetd services be turned on for PXE provisioning functionality. Enable these services [Y]? y
* Restarting services.
Installation complete.
Visit https://spacewalk to create the Spacewalk administrator account.
[root@spacewalk ~]# 

​Post Installation:​

[root@spacewalk ~]#  spacewalk-service status
postmaster (pid 7077) is running...
router (pid 7104) is running...
sm (pid 7114) is running...
c2s (pid 7124) is running...
s2s (pid 7134) is running...
tomcat6 (pid 7207) is running... [ OK ]
httpd (pid 7335) is running...
osa-dispatcher (pid 7352) is running...
rhn-search is running (7406).
cobblerd (pid 7437) is running...
RHN Taskomatic is running (7464).
[root@spacewalk ~]#  


Login to the web ​interface ​and create first user login.





spacewalk was installed successfully.