Sunday, 6 April 2014

Configure log rotation - Solaris 10

In order to have an easy administration of systems which generates large number of log files, you can configure your log files according by an utility called logroate, which allows automatic rotation, compression, removal and also mailing of log files which can be handled daily, weekly or when it grows too large.

Objective: compress and rotate logs after certain threshold on the file size.

Environment: Solaris 10 32-bit

The system log rotation is defined in the /etc/logadm.conf file. This file includes log rotation entries for processes such as syslogd. For example, one entry in the /etc/logadm.conf file specifies that the /var/log/ciscofirewall.log file is rotated weekly unless the file is empty. The most recent ciscofirewall.log file becomes ciscofirewall.log.0, the next most recent becomes ciscofirewall.log.1, and so on. Eight previous ciscofirewall log files are kept.
The /etc/logadm.conf file also contains time stamps of when the last log rotation occurred.
# vi /etc/logadm.conf 
/var/log/ciscofirewall.log -C 9 -s 10240k -z 4 -N -a 'kill -HUP `cat /var/run/`'

     -C = expire old logs until count remain.( 9 log files created and rotated )
     -N = not an error if log file nonexistent.
     -s = only rotate if given size or greater.
     -a = execute command after taking actions.
     -z = gzip old logs except most recent count ( last 5 log files would be compressed )

- Restart the syslogd to take changes effectively.

The command is often run on a cron job, which has the effect of fully automatic log rotation.

# crontab -l 
10 3 * * * /usr/sbin/logadm

# ls -l /var/log/ciscofirewall*.log.* | wc -l

# ls -l /var/log/ciscofirewall*.log.*.gz | wc -l

# ls -ltr /var/log/ciscofirewall.log*
-rw-r--r--   1 root     root       41048 Apr  6 14:38 /var/log/ciscofirewall.log.8.gz
-rw-r--r--   1 root     root       42076 Apr  6 14:39 /var/log/ciscofirewall.log.7.gz
-rw-r--r--   1 root     root       41621 Apr  6 14:40 /var/log/ciscofirewall.log.6.gz
-rw-r--r--   1 root     root       41524 Apr  6 14:41 /var/log/ciscofirewall.log.5.gz
-rw-r--r--   1 root     root       41410 Apr  6 14:42 /var/log/ciscofirewall.log.4.gz
-rw-r--r--   1 root     root     21510944 Apr  6 14:43 /var/log/ciscofirewall.log.3
-rw-r--r--   1 root     root     21139079 Apr  6 14:44 /var/log/ciscofirewall.log.2
-rw-r--r--   1 root     root     21536814 Apr  6 14:45 /var/log/ciscofirewall.log.1
-rw-r--r--   1 root     root     21399755 Apr  6 14:46 /var/log/ciscofirewall.log.0
-rw-r--r--   1 root     root     16434041 Apr  6 14:46 /var/log/ciscofirewall.log

all your logs has been rotated in a discipline manner, which would be easy to troubleshoot in-case of any errors.

Monday, 10 March 2014

OpenSSH security/hardening tips #Linux/Unix

Objective: Strengthen security on SSH

Environment: CentOS-6.3 (32-bit)

OpenSSH version : OpenSSH_5.3p1

SSH Protocol version : 2

I had observed more lines which were commented in sshd_config, hence I had enabled few of the options which could possibly strengthen ssh.

Any changed made to the /etc/ssh/sshd_config would require its corresponding daemon(sshd) to be restarted. Hence I would assume the reader would probably be aware and continuing explaining below options.

1. Change the port number on which sshd listens or specify the local address along with the port number on which sshd listens.

2. The server disconnects if the user has not successfully logged within 120 seconds.
LoginGraceTime 2m

You may receive below snap if you would continue to be without logging in.

3. Once after logging into the server, and if there is no data communication for around 5 Minutes then we could force auto-logout option.

# grep -i tmout /etc/bashrc 
export TMOUT=300

You may receive the below error incase if the session is idle for 5 Minutes.

# timed out waiting for input: auto-logout
Connection to XXX.XXX.XXX.XXX closed.

4. Use the authentication methods(RSA, DSA) to grant access to the users, i..e password less logis to the users. How to create RSA/DSA to authenticate

PermitRootLogin without-password
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile      .ssh/authorized_keys
AuthorizedKeysCommand yes

# To disable tunneled clear text passwords, change to no here!
PermitEmptyPasswords no
PasswordAuthentication no

5. sshd should check file modes and ownership of the user's file and the home directories before accepting login.
StrictModes yes

6. Password attempts to be prompted would be ideally two, incase of any failures would be disconnected.
MaxAuthTries 3

7. Restricted the maximum concurrent un-authenticated connections to the SSH daemon.
MaxStartups 2

8. You can use allow/deny directives for users/groups to allow/disallow based on the primary and the supplementary groups lists matches one of the pattern.
Allowgroups sshgroup

9. Finally, you can configure chrootDirectory which specifies the pathname of a directory to jail the user after authentication. All the components of the pathname must be root owned.
If this interests you on how to jailusers Click here 

I, hereby conclude that most of the options related in securing communication between server & client by OpenSSH had been discussed.

Saturday, 4 January 2014

Automatically Chroot Jail for SSH Access - #Linux_CentOS

Objective: User to be provided with a limited system environment.

Environment: CentOS-6.3 (32-bit)

I wanted to setup a way to allow SSH access to my machine but limit their abilities heavily. This can be achieved by jailing the user and providing limited access to the system.

Change plan:

1. List the user to be jailed, and gather details on his limitation for the system environment.
2. Creating basic "chroot" environment.
3. Create chroot user group.
4. Configure SSHd to chroot your users.

1. Create a test user and the group to be jailed for users.

#groupadd jailgroup
#useradd -g jailgroup jailuser

# tail  -1 /etc/passwd
# tail  -1 /etc/group


2. Setting up the jail environment.

Create the directories, which needs to emulate the / directory to a bare minimum. That is we need a dev, etc, lib, usr, and bin directory as well as usr/bin/. The bash directory has to be owned by root.
#mkdir -p /chroot/{dev,etc,lib,usr,bin,home}
#mkdir -p  /chroot/usr/bin

We need a special file to disposing of unwanted output streams of a process, hence I would add "/dev/null" to my jail environment.
#mknod -m 666 /chroot/dev/null c 1 3

3. Copy few files to /chroot/etc directory.

# cp /etc/ /chroot/etc

# cp /etc/ /chroot/etc
# cp /etc/nsswitch.conf /chroot/etc

# cp /etc/hosts /chroot/etc
# cp /bin/ls /chroot/usr/bin
# cp /bin/bash /chroot/usr/bin/

Once done, you need to figure out what command you want accessible by your limited users. I want them to get into bash and list and write using 'vim' editor.
Location can be found as below 

#which cat vim ls

Now that you've got all the binaries in place, you need to add the proper shared libraries. To find out what libraries are as below 
# ldd /bin/ls =>  (0x008b3000) => /lib/ (0x00101000) => /lib/ (0x0012a000) => /lib/ (0x0018e000) => /lib/ (0x00183000) => /lib/ (0x00af5000) => /lib/ (0x00c89000)
        /lib/ (0x00acf000) => /lib/ (0x00a11000) => /lib/ (0x00122000)

Since you need to manually copy each file, I had an piece of code from "". copy paste the below code and execute.

# cat
for i in $( ldd $*  | grep -v dynamic |cut  -d" "  -f3  | sed  's/://' | sort | uniq )

#--parents - using full source file name under DIRECTORY.
cp --parents  $i  $CHROOT

# ARCH  i386
if  [ -f  /lib/ ];  then
cp  --parents /lib/ /$CHROOT
echo   "Chroot jail is ready."

#chmod +x
#./ /bin/{ls,cat,bash} /usr/bin/vim 

4. Configure SSHd to chroot your users.

Add the following line to /etc/ssh/sshd_config file
# tail -5 /etc/ssh/sshd_config
Match group jailgroup
          ChrootDirectory /chroot/./
          X11Forwarding no
          AllowTcpForwarding no

- Try to login to the server as the user.
# ssh jailuser@0 jailuser@0's password: Last login: Sat Jan 4 11:40:57 2014 from [-bash-4.1:~]$ pwd /home/jailuser [-bash-4.1:~]$ ls [-bash-4.1:~]$ rm -bash: rm: command not found [-bash-4.1:~]$ touch -bash: touch: command not found [-bash-4.1:~]$    
If a user does not have its home user directory available in a chroot jail after login s/he will end up in /. You can create and further configure your chroot by creating a user home directory, defining bash environment i..e bashrc file which sets PS1 needs to be customized as your needs to look better. 
I hence conclude this article by providing a limited system environment to the user via SSH access.

Monday, 9 December 2013

Server Hardware’s Fault ! - diagnosis & troubleshooting

I will cover some of the common hardware failure you might run into, along with the steps to troubleshoot and confirm them. 

Environment: CentOS_6.3 (32-bit)

Few of the more common pieces of hardware that fail are 
1. Hard drive(HDD).
2. RAM. 
3. Network card failures.
4. Server's temperature.

1. HDD

Hard drive manufacturers all have their own hard drive testing tools, modern hard drives should also support SMART(Self-Monitoring, Analysis and Reporting Technology; often written as SMART), which monitors the overall health of the hard drive and can alert you when the drive will fail soon. 

Install package smarttools, pass the -H option to smartctl to check the health of a drive:

# smartctl -H /dev/sda
smartctl 5.40 2010-10-16 r3189 [x86_64-redhat-linux-gnu] (local build)

Copyright (C) 2002-10 by Bruce Allen,

SMART Health Status: OK 

The test has been passed without any errors. If it would be there then it would have displayed the WARNING/ERROR 

You can also pull much more information about a hard drive using smartctl with the -a option. That option will pull out all of the SMART information 
about the drive

#smartctl -a /dev/sda


Tool for detecting memory would be Memtest86  which needs to be installed additionally which is added automatically to your GRUB configurations.

No matter how you invoke it at boot time, once you start Memtest86, it will immediately launch and start scanning your RAM.

If there are found to be few errors, try to change a new RAM and again re-run the test.

3. Network card failures

When a network card or some other network component to which your host is connected starts to fail, you can see it in packet errors on your system. The 
ifconfig command you may have used for network troubleshooting before can also tell you about TX (transmit) or RX (receive) errors for a card.

# ifconfig eth0 | egrep "RX|TX"
          RX packets:2144865318 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2339638100 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000

If you start to see a lot of errors here, then it's worth troubleshooting your physical net-work components. It's possible a network card, cable, or switch port is going bad.

4. Server Temperature

A poorly cooled server can cause premature hard drive failure and premature failure in the rest of the server components as well.  

Linux provides tools that allow you to probe CPU and motherboard temperatures, and, in some cases, the temperatures of PCI devices and even fan speeds. All of this support is provided by the lm-sensors package.

Once the lm-sensors package is installed, run the sensors-detect as root :


This interactive script will probe the hardware on your system so it knows
how to query for temperature. 
Once the sensors-detect script is completed, you can pull data about your server by running the command sensors.

# sensors
Adapter: ISA adapter
Core 0:       +46.0°C  (high = +84.0°C, crit = +100.0°C)

Adapter: ISA adapter
Core 1:       +42.0°C  (high = +84.0°C, crit = +100.0°C)

Adapter: ISA adapter
Core 2:       +42.0°C  (high = +84.0°C, crit = +100.0°C)

Adapter: ISA adapter
Core 3:       +42.0°C  (high = +84.0°C, crit = +100.0°C)

Adapter: ISA adapter
Ch. 0 DIMM 0:  +58.0°C  (low  = +118.0°C, high = +124.0°C)
Ch. 1 DIMM 0:  +64.0°C  (low  = +118.0°C, high = +124.0°C)
Ch. 2 DIMM 0:  +72.0°C  (low  = +118.0°C, high = +124.0°C)
Ch. 3 DIMM 0:  +64.0°C  (low  = +118.0°C, high = +124.0°C)

You could sue sensors -f for the temperatures in Fahrenheit.

Examine the airflow around the server and make sure the vents in and out of the server aren't clogged with dust.If you have the bad habit of not rack mounting your servers but instead installing a shelf and stacking servers one on top of the other, that will also contribute to poor airflow and overheating. 

I would like to conclude the article by saying that these are the few of the common failures for the hardware faults.

Wednesday, 16 October 2013

Recovering volume group metadata in LVM with UUID disk #CentOS_6_RHEL

Objective: Recover volume group metadata in LVM.

Environment: CentOS_6.3 (32-bit)

LVM version: 2.02.95

Brief introduction on LVM:

To use the device for an LVM logical volume the device must be initialized as a physical volume (PV), which on initializing places a label near the start of the device. By default, the LVM label is placed in the second 512-byte sector. The LVM label identifies the device as an LVM physical volume which contains the UUID for the physical name. It also stores the size of block devices in bytes, and LVM metadata stored on the disk.

The LVM metadata contains the configuration details of the LVM volume groups on your system. By default, an identical copy of the metadata is maintained in every metadata area in every physical volume within the volume group.

Error which I had received when accessing an disk, where metadata was corrupted.

# pvs -o pv_name,uuid
  Couldn't find device with uuid U3qzRV-jakK-0JFC-zVmq-F1aa-nxrg-9JYsXy.
- We would be able to find the UUID for the PV in /etc/lvm/archive directory, deactivating the volume and setting the partial(-P) argument will
enable you to find the UUID of the missing corrupted physical volume.

# vgchange -an --partial
  Partial mode. Incomplete logical volumes will be processed.
  Couldn't find device with uuid U3qzRV-jakK-0JFC-zVmq-F1aa-nxrg-9JYsXy.
  0 logical volume(s) in volume group "testvg" now active

# vgck testvg
  Couldn't find device with uuid U3qzRV-jakK-0JFC-zVmq-F1aa-nxrg-9JYsXy.
  The volume group is missing 1 physical volumes.

# pvck /dev/sdd
  Could not find LVM label on /dev/sdd

- pvcreate comes in handy restores the physical volume label with the metadata information contained in VG group(testvg). The restore file argument instructs the   pvcreate command to make the new physical volume compatible with the old one on the volume group, ensuring that the new metadata will not be placed where the old physical volume contained data.

NOTE: The pvcreate command overwrites only the LVM metadata areas and does not affect the existing data areas.

# pvcreate --uuid "U3qzRV-jakK-0JFC-zVmq-F1aa-nxrg-9JYsXy" --restore /etc/lvm/backup/testvg /dev/sdd
  Couldn't find device with uuid U3qzRV-jakK-0JFC-zVmq-F1aa-nxrg-9JYsXy.
  Writing physical volume data to disk "/dev/sdd"
  Physical volume "/dev/sdd" successfully created

- vgcfgrestore command to restore the volume group's metadata

# vgcfgrestore testvg
  Restored volume group testvg

# pvs -o pv_name,uuid | grep "U3qzRV-jakK-0JFC-zVmq-F1aa-nxrg-9JYsXy"
  /dev/sdd   U3qzRV-jakK-0JFC-zVmq-F1aa-nxrg-9JYsXy

Recovered volume metadata from LVM.

Thursday, 10 October 2013

Disk Encryption - eCryptfs & dm-crypt + LUKS - #CentOS_6/RHEL_6

Objective :- Disk encryption techniques.

Environment :- CentOS release 6.3(32-bit)

I would like to discuss few techniques available in linux for cryptographically protecting a logical part of a storage disk(folder, partition, whole disk, ...), so that all data that is written to it is automatically encrypted, and decrypted on-the-fly when read again.

Below two methods are discussed as below :-

1.  eCryptfs 
2.  dm-crypt with LUKS


I would describe the basic use of eCryptfs, which will guide through the process of creating a secure and a private directory which can store your sensitive and private data.

This doesn't require special on-disk storage allocation effort, such as seperate partition, you can mount eCryptfs on top of any single directory to protect it.All cryptographic metadata is stored in the headers of files, so encrypted data can be easily moved, stored for backup and recovered. 

There are few of the drawbacks, for instance eCryptfs is not suitable for encrypting complete partitions, however instead you can combine it with dm-crypt. But in this article I would be combining dm-crypt with LUKS mechanism and demonstrate.

On summarizing, eCryptfs, a "pseudo-file system" which provides data and filename encryption on a per-file basis, it is a file system layer that resides on top of an actual file system,  providing encryption capabilities.

- Install the package.
#yum install ecryptfs-utils -y

Assume while creating a new file system(/db), I had created another file system layer that which resides on the top of the actual file system.
# df -h /db

Filesystem            Size  Used Avail Use% Mounted on
                      485M   11M  449M   3% /db

# mount -t ecryptfs /db /db  
Select key type to use for newly created files:
1) tspi
2) openssl
3) passphrase  
Selection: 3  
Select cipher:  
1) aes: blocksize = 16;min keysize = 16; max keysize = 32 (not loaded)  
2) blowfish: blocksize = 16; min keysize = 16; max keysize = 56 (not loaded)  
3) des3_ede: blocksize = 8; min keysize = 24; max keysize = 24 (not loaded)  
4) twofish: blocksize = 16; min keysize = 16; max keysize = 32 (not loaded)  
5) cast6: blocksize = 16; min keysize = 16; max keysize = 32 (not loaded)  
6) cast5: blocksize = 8; min keysize = 5; max keysize = 16 (not loaded)  
Selection [aes]: 
Select key bytes:  
1) 16  
2) 32  
3) 24  
Selection [16]:  
Enable plaintext passthrough (y/n) [n]:  
Enable filename encryption (y/n) [n]:  
Attempting to mount with the following options:  
Mounted eCryptfs  
- Now there are two /db file system, one which is an actual file system, another which resides on the top of the file system.
# df -h | grep db
                      485M   11M  449M   3% /db
/db                   485M   11M  449M   3% /db

- Any thing written on the /db file system will now be encrypted.
:/db]# cat >encrypt_file
All files and directories will be encrypted in this file system.

- I would now unmount /db, which is encrypted file system. viewing the actual file system will result in garbled random looking characters.
Reading the encrypted sectors without permission will return garbled random-looking data instead of the actual files.
#umount /db 

:/db]# tail encrypt_file

fÓxpu;¬0éb¶EyºlH$ÖR/'úÚÿ±' £¢?ß¹[]v­*ã*Ʀ¡²Ò
Æéþ½ rll9æ½vÿså¢GÙMøåÇûÀÅë·ÙúKoí:/db]#

Hence we have successfully encrypted the files in the file system.

Snap is provided which is self-explanatory.

dm-crypt with LUKS

dm-crypt is the standard device-mapper encryption functionality provided by the Linux kernel. It can be used directly by those who like to have full control over all aspects of partition and key management.

LUKS(Linux Unified Keystartup) is an additional convenience layer which stores all of the needed setup information for dm-crypt on the disk itself and abstracts partition and key management in an attempt to improve ease of use.

Summarizing, LUKS is a standard format for device encryption. LUKS encrypts the parttion or volume, the volume must be decrypted before the file system in it can be mounted.

- Create a new partiton on the disk.

- Encrypt the new partition and set the decryption password.
# cryptsetup luksFormat /dev/sdd1

This will overwrite data on /dev/sdd1 irrevocably.
Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase:
Verify passphrase:

- You need to unlock the encrypted volume.
# cryptsetup luksOpen /dev/sdd1 cryptvg-cryptlv
Enter passphrase for /dev/sdd1:

- Create an ext4 file system on the decrypted volume.
[root@kickstart ~]# mkfs -t ext4 /dev/mapper/cryptvg-cryptlv

- Create a directory mount point and mount the file system.
# mount /dev/mapper/cryptvg-cryptlv /secret #

- When finished, unmount and lock the encrypted volume.
# umount /secret/

# cryptsetup luksClose /dev/mapper/cryptvg-cryptlv #

Persistant Mount Encrypted partition.

- Add your list of devices to be unlocked during the system startup.
# cat /etc/crypttab
Name         Device         /path/to/password
cryptvg-cryptlv /dev/sdd1       /root/encdisk

- Create an entry in fstab.
# tail -1 /etc/fstab
/dev/mapper/cryptvg-cryptlv     /secret         ext4    defaults        1 2

- Create a keyfile that includes the password. Make sure it is owned by the root and the mode is 600. Add the key for LKS.
#echo -n "passphrase" > /root/encdisk
#chown root /root/encdisk
#chmod 600 /root/encdisk
#cryptsetup luksAddKey /dev/sdd1 /root/encdisk

Reboot your system, it should not ask for any passphrase during the boot. Check your file system should be mounted automatically without providing the passphrase.

Providing the snap, which is self-explanatory.

NOTE: The device in the /etc/fstab and the /etc/crypttab should be the same. Since I made a mistake in these two files which made system to PANIC. This would be kernel was unable to boot with the name as in fstab.


Monday, 30 September 2013

PXE boot[Installation/Rescue-Environment]with kickstart configurations - #linux_CentOS/Redhat

Objective: How to automate installation/Resuce(PXE using kickstart) CentOS.

I have already discussed about the kickstart installation in my previous post  CentOS kickstart

In my earlier post, I had generated kickstart file then boot from CD/DVD where I pointed the location of kickstart file either from HTTP/FTP/NFS method. But in this post I have used the same kickstart file to NIC of the destination server which acts like a boot device, eliminating CD/DVD.

Environment: RHEL/CentOS 6.3 (i386)

What is PXE ?

Preboot eXecution Environment is an environment to boot computers using a network interface independently of data storage devices (like hard disks) or installed operating systems.

PXE works with NIC of the system making it function like a boot device. PXE enabled NIC of the client sends out broadcast request to DHCP server, which returns with the IP address of the client along with the address of the TFTP server, and the location of the boot files on the TFTP server.

In detail :

1. Destination server is booted
2. NIC of the destination triggers DHCP request.
3. DHCP server intercepts the request and responds with the standard information(IP, subnet, gateway, etc). In addition it provides information about the location of a TFTP server and boot image(pxelinux.0)
4. When client receives this information, it contacts TFTP server for obtaining the boot image.
5. TFTP server sends the boot image(pxelinux.0) and client executes it.
6. By default, boot image searches the pxelinux.cfg directory in TFTP server.
7. The destination server downloads all the files it needs(kernel and root file system), and then loads them.

Make sure below packages are installed and services are started. [ Skipping package installation ]
1. tftp-server 
2. syslinux
3. dhcp

#grep disable /etc/xinetd.d/tftp
disable                 = no

#cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/
#cp /usr/share/syslinux/menu.c32 /var/lib/tftpboot/
#cp /usr/share/syslinux/memdisk /var/lib/tftpboot/
#cp /usr/share/syslinux/mboot.c32 /var/lib/tftpboot/
#cp /usr/share/syslinux/chain.c32 /var/lib/tftpboot/
#mkdir /var/lib/tftpboot/pxelinux.cfg
#mkdir -p /var/lib/tftpboot/images/centos/i386/6.3
#cp /var/ftp/pub/images/pxeboot/initrd.img /var/lib/tftpboot/images/centos/i386/6.3
#cp /var/ftp/pub/isolinux/vmlinuz /var/lib/tftpboot/images/centos/i386/6.3

DHCP configuration file is as below..

# cat /etc/dhcp/dhcpd.conf

option domain-name      "";
option domain-name-servers;
default-lease-time 600;
max-lease-time 7200;
#################The followings are mandatory to boot from PXE ###################
allow booting;
allow bootp;
option option-128 code 128 = string;
option option-129 code 129 = text;
filename "/pxelinux.0";
subnet netmask {
        range dynamic-bootp;
        option broadcast-address;
        option routers;

# cat /var/lib/tftpboot/pxelinux.cfg/default

default menu.c32
prompt 0
timeout 30


LABEL CentsOS_6.3_i386
    MENU LABEL CentOS 6.3 i386
    KERNEL images/centos/i386/6.3/vmlinuz
    APPEND initrd=images/centos/i386/6.3/initrd.img ks=ftp://<IPaddress of kickstart file server>/pub/ks.cfg ramdisk_size=100000

LABEL CentsOS_6.3_i386_Rescue
    MENU LABEL CentOS 6.3 i386_Rescue
    KERNEL images/centos/i386/6.3/vmlinuz
    APPEND initrd=images/centos/i386/6.3/initrd.img ramdisk_size=100000 repo=ftp://<IPaddress of kickstart file server>/pub lang=en_US.UTF-8 keymap=us rescue 

This completes all your configuration, before this works make sure all your services are running and persistant.

#service xinetd start
#chkconfig xinetd on
#service dhcpd start
#chkconfig dhcpd on

you need to check in what method are you trying to kickstart your installation based on which your services should be up.
#service vsftpd/httpd/nfs start
#chkconfig vsftpd/httpd/nfs on

NOTE: Make sure your first boot device is network.

I have chosen to change the password in the rescue environment as a test and is successful.