Thursday 3 November 2016

git branching and merge for puppetmaster configs modules and manifests - 4/4


As earlier discussed in the post, http://sunlnx.blogspot.in/2016/11/puppet-manifests-and-modules-in.html we shall add version controlling on /etc/puppet and since its important not to distub the main configuration, we shall create a branch out of it and once its all working fine, we shall merge it with master. 

what and why is branching ?

Branching is a process of creating a new pointer that allows you to branch off the code and work on the same code within the safe environment where you are free to mess up and you can discard those changes and you could merge if you feel its correct. git offers that we could switch code from one branch to other and also by merging you don't have any duplicate code. 

As earlier described in article(http://sunlnx.blogspot.in/2015/07/install-and-configure-git-centos-7.html), make sure you install/configure your git repository.

[root@puppetmaster puppet]# git branch
* master
[root@puppetmaster puppet]# git branch puppet-testing
[root@puppetmaster puppet]# git branch
* master
  puppet-testing
[root@puppetmaster puppet]#

[root@puppetmaster puppet]# git status
# On branch master
nothing to commit (working directory clean)
[root@puppetmaster puppet]#

[root@puppetmaster puppet]# git checkout puppet-testing
Switched to branch 'puppet-testing'
[root@puppetmaster puppet]# git status
# On branch puppet-testing
nothing to commit (working directory clean)
[root@puppetmaster puppet]#

[root@puppetmaster puppet]# git branch
  master
* puppet-testing
[root@puppetmaster puppet]#

Now you can do anything you wish to the branch, like deleting, modifying ...

[root@puppetmaster puppet]# cat > git-branching.test
this is just an file created in puppet-testing branch.
we shall see how to merge this into 'master' branch.
[root@puppetmaster puppet]#

[root@puppetmaster puppet]# ls -l git-branching.test
-rw-r--r-- 1 root root 108 Nov  3 04:58 git-branching.test
[root@puppetmaster puppet]#

[root@puppetmaster puppet]# git add *
[root@puppetmaster puppet]# git commit -m "Added git-branching to repos"
[puppet-testing 386d918] Added git-branching to repos
 1 files changed, 2 insertions(+), 0 deletions(-)
 create mode 100644 git-branching.test
[root@puppetmaster puppet]#

[root@puppetmaster puppet]# git push origin puppet-testing
Password:
Counting objects: 4, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 369 bytes, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local objects.
 * [new branch]      puppet-testing -> puppet-testing
[root@puppetmaster puppet]#

Refresh you repository in github and you can see branch puppet-testing. you could also create a new branch again from 'puppet-testing' and it can go on as much as you can. something great that's more flexible on git.

Now in order to merge your branch, switch to master branch 

[root@puppetmaster puppet]# git checkout master
Switched to branch 'master'
[root@puppetmaster puppet]#

[root@puppetmaster puppet]# git merge puppet-testing
Updating 285446b..386d918
Fast-forward
 git-branching.test |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
 create mode 100644 git-branching.test
[root@puppetmaster puppet]# git branch -d puppet-testing
Deleted branch puppet-testing (was 386d918).
[root@puppetmaster puppet]#

[root@puppetmaster puppet]# git push origin master
Password:
Total 0 (delta 0), reused 0 (delta 0)
   285446b..386d918  master -> master
[root@puppetmaster puppet]#

[root@puppetmaster puppet]# git push origin --delete puppet-testing
Password:
 - [deleted]         puppet-testing
[root@puppetmaster puppet]#

You can now refresh your github.com and see there won't be branch available. checkout in your master branch the file was merged to master..

Thanks for re-sharing !

Run puppet manifests and modules in different environments - 3/4

In our previous posts, we have seen on how to install and configure puppetmaster/puppetclients we shall now see on how to create the manifests on the different environments. Additionally, I have added the puppet configuration folder to be controlled using version controller 'git', where I shall explain in my next final post.

Puppet version 3.8.7, by default has no environmentpath enabled and it needs to be mentioned in config files.

Place these 3 lines below [main] section of /etc/puppet/puppet.conf, save and close.
Defined the directory and have asked puppet to first check environment path while on its execution.
    confdir = /etc/puppet
    environmentpath = $confdir/environments
    basemodulepath  = $confdir/modules:/usr/share/puppet/modules
[root@puppetmaster puppet]#service puppetmaster restart

Production Environment:

[root@puppetmaster puppet]# mkdir -p environments/{production,testing}

[root@puppetmaster ~]# cd /etc/puppet/environments/production
[root@puppetmaster production]# mkdir manifests modules

Define your environment.conf in the each of the folder.

[root@puppetmaster production]# cat environments.conf
modulepath = $confdir/environments/production/modules:$condfir/modules:/usr/share/puppet/modules
[root@puppetmaster production]#

[root@puppetmaster production]#mkdir -p modules/prodtest/{files,manifests}

Finally, once after you create your directories, the tree structure should be as below :

[root@puppetmaster production]# tree -F .
.
├── environments.conf
├── manifests/
│   └── node.pp
└── modules/
    └── prodtest/
        ├── files/
        │   └── prodtest.conf
        └── manifests/
            └── init.pp

5 directories, 4 files
[root@puppetmaster production]#

Create files accordingly..

[root@puppetmaster production]# cat modules/prodtest/manifests/init.pp
class prodtest {
file {'/tmp/production':
       path => '/tmp/production',
       ensure => present,
       mode => 640,
       source  => 'puppet:///modules/prodtest/prodtest.conf',
       }
}
[root@puppetmaster production]#

[root@puppetmaster production]# cat modules/prodtest/files/prodtest.conf
I am executing from the prodcution environment....
[root@puppetmaster production]#

[root@puppetmaster production]# cat manifests/node.pp
    include prodtest
}
[root@puppetmaster production]#

From the client, execute the below to retrive info..

[root@puppetclient ~]# puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for puppetclient.example.com
Info: Applying configuration version '1478153895'
Notice: /Stage[main]/Prodtest/File[/tmp/production]/ensure: defined content as '{md5}467d4f8c0dff1ec2799ee98637aff019'
Notice: Finished catalog run in 0.11 seconds
[root@puppetclient ~]#

[root@puppetclient ~]# ls -l /tmp/production
-rw-r----- 1 root root 51 Nov  3 06:18 /tmp/production

[root@puppetclient ~]# cat /tmp/production
I am executing from the prodcution environment....
[root@puppetclient ~]#

Now you could write all your classes from the module directory and start executing those manifests. 
punch "puppet manifests examples" in google and you will get lot of pages. I would leave it as an exercise for the reader. 

testing environment:

Just as in the production, I have create a separate environment for 'testing' ,create the directories and files as the same as what you did for production environment.

[root@puppetmaster environments]# tree -F testing/
testing/
├── environments.conf
├── manifests/
│   └── nodes.pp
└── modules/
    └── testing/
        ├── files/
        │   └── testing.conf
        └── manifests/
            └── init.pp

5 directories, 4 files
[root@puppetmaster environments]#

[root@puppetmaster testing]# cat environments.conf
modulepath = $confdir/environments/testing/modules:$condfir/modules:/usr/share/puppet/modules
[root@puppetmaster testing]#

[root@puppetmaster testing]# cat modules/testing/manifests/init.pp
class testing {
file { '/tmp/testing':
  ensure  => present,
  owner   => 'root',
  group   => 'root',
  mode    => '0777',
  source  => 'puppet:///modules/testing/testing.conf',
   }
}
[root@puppetmaster testing]#

[root@puppetmaster testing]# cat modules/testing/files/testing.conf
creating from testing environment....
Creating the test file for Puppet demonstration
[root@puppetmaster testing]#
 [root@puppetmaster testing]# cat manifests/nodes.pp
    include testing
}
[root@puppetmaster testing]#

On the client, execute mentioning the environment which puppet should pick, else by default it will pick from 'production'

[root@puppetclient ~]# puppet agent -t --environment testing
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for puppetclient.example.com
Info: Applying configuration version '1478154384'
Notice: /Stage[main]/Testing/File[/tmp/testing]/ensure: defined content as '{md5}8fffaf0c4ae55d51df4e926284eef521'
Notice: Finished catalog run in 0.10 seconds
[root@puppetclient ~]#

[root@puppetclient ~]#ls -l
-rwxrwxrwx 1 root root 86 Nov  3 06:26 /tmp/testing
[root@puppetclient ~]#

[root@puppetclient ~]# cat /tmp/testing
creating from testing environment....
Creating the test file for Puppet demonstration
[root@puppetclient ~]#


we shall see in our next post, on how to add '/etc/puppet' to version controlling and create branch for testing puppet and later merging so that it can be used for many in the development/testing teams.

Thanks for re-sharing !

Wednesday 2 November 2016

​Configure your own git server - Linux


In this post, I would explain on how to configure your own git server where you could manage your own code or your configuration files.  
I shall consider two servers one with hostname 'gitserver' other as a 'gitclient'.

Environment: Linux Centos 6
git version: 1.7.1

First, you would require to install 'git' on both the machines. 

#yum install -y git

Ensure you have created a password less authentication between 'gitserver' and 'gitclient'. incase any help needed refer - http://sunlnx.blogspot.in/2012/09/ssh-passwordless-linuxwindows.html 

Create a project directory for git on your 'gitserver'

[root@gitserver ~]# mkdir python.git
[root@gitserver ~]# cd python.git/
[root@gitserver python.git]# git init --bare
Initialized empty Git repository in /root/python.git/
[root@gitserver python.git]#

Login to the 'gitclient' to create a git empty repository.

[root@gitclient ~]# mkdir python
[root@gitclient ~]# cd python/
[root@gitclient python]# git init
Initialized empty Git repository in /root/python/.git/

[root@gitclient python]# cat > hello.py
#!/usr/bin/python
print "Hi .. I am from gitclient who will be commited to gitserver"
[root@gitclient python]# 

[root@gitclient python]# git add .
[root@gitclient python]# git commit -m "First git commit - Hello" -a
[master (root-commit) 5308ec1] First git commit - Hello
 Committer: root <root@gitclient.oratest.com>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly:

    git config --global user.name "Your Name"
    git config --global user.email you@example.com

If the identity used for this commit is wrong, you can fix it with:

    git commit --amend --author='Your Name <you@example.com>'

 1 files changed, 2 insertions(+), 0 deletions(-)
 create mode 100644 hello.py
[root@gitclient python]#

[root@gitclient python]# git remote add origin ssh://gitserver.oratest.com/root/python.git
[root@gitclient python]# git push origin master
Counting objects: 3, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 295 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
 * [new branch]      master -> master
[root@gitclient python]#

you can check from gitserver log regarding the commits.

[root@gitserver python.git]# git log
commit 5308ec10f87eab1c5bb474a5f42de26818915f7b
Date:   Wed Nov 2 13:57:48 2016 +0530

    First git commit - Hello
[root@gitserver python.git]#

If there are anyone who wants to add their code to this repository, they just would need to clone the repository to their local machine.

#git clone ssh://gitserver.oratest.com/root/python.git

they could now edit files, write commit change messages and then can push to the server. 

You can also add your repository in github.com, incase if you need any help refer http://sunlnx.blogspot.in/2015/07/install-and-configure-git-centos-7.html 

References:
Git cheat sheet

Thanks for re-sharing

Run puppetmaster/puppetclients using Dockers 2/4


In my earlier post, we already have set up puppet master and puppet client in the Guest OS, since its easily portable for deployment of applications I would create an docker image so that it would be easy for us to deploy the same in other machines to test in any environment without any overheads.

you can see the previous post on how to install puppet : http://sunlnx.blogspot.in/2016/11/installconfigure-puppet-serverpuppet.html

You first need to install docker packages 


Downloaded the docker images of Centos 6, which will pull the latest docker image.

root@ubuntu:~# docker pull centos:6
6: Pulling from library/centos

Digest: sha256:f3cc24b376d42d9fd9fa914335dddee46ed7b5a2e2dd1ee6df045ce437199d79
Status: Image is up to date for centos:6
root@ubuntu:~#

You need to fire up two containers from the downloaded docker image, one for the puppetmaster and the other for puppetclient. 

root@ubuntu:~# docker run -it --name puppetmaster --hostname puppetmaster.example.com centos:6 /bin/bash
root@puppetmaster /] 

root@ubuntu:~# docker run -it --name puppetclient --hostname puppetclient.example.com centos:6 /bin/bash
root@puppetmaster /] 

root@ubuntu:~# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
7d2c3af9ff52        76aa66802274        "/bin/bash"         24 hours ago        Up 3 hours                              puppetclient
113707c588ab        6f9a4933ce57        "/bin/bash"         2 days ago          Up 3 hours                              puppetmaster
root@ubuntu:~#


Once you have fired up your docker containers, make sure you have your both hosts puppetmaster and puppetclient are resolvable, either in DNS or in /etc/hosts.
you can install/configure puppet master and puppet client as in the article http://sunlnx.blogspot.in/2016/11/installconfigure-puppet-serverpuppet.html 

once they are completed, you can stop both the containers. 
root@ubuntu:~#docker stop puppetmaster
puppetmaster
root@ubuntu:~#docker stop puppetclient
puppetclient

root@ubuntu:~#docker ps
<display no running containers>

root@ubuntu:~#docker commit -m "puppetmaster" <CONTAINER ID> <dockerid>/puppetmaster:<tag>
root@ubuntu:~#docker commit -m "puppetclient" <CONTAINER ID> <dockerid>/puppetclient:<tag>

root@ubuntu:~#docker images
<you could now see your image installed with puppetmaster and puppetclient>

anyone who has no account for docker hub, you can create a new account. https://hub.docker.com/
already account exists, then you can push you docker image created puppetmaster/puppetclient to your account and can be download later in any of the platform installed with dockers to fire up two containers. 

root@ubuntu:~#docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username (dockerid):
.
.
<successful>

root@ubuntu:~#docker push <dockerid>/puppetmaster:latest
root@ubuntu:~#docker push <dockerid>/puppetclient:lastest

you have now successfully uploaded your images to your dockerid repository. 

you can download anytime if you require your docker images using below command and start to firing up your container.

root@ubuntu:~#docker run -it --name puppetmaster --hostname puppetmaster.example.com <dockerid>/puppetmaster /bin/bash
root@puppetmaster /] service puppetmaster start
root@puppetmaster /] echo "172.17.0.3 puppetclient.example.com" >>/etc/hosts

root@ubuntu:~#docker run -it --name puppetclient --hostname puppetclient.example.com <dockerid>/puppetclient /bin/bash
root@puppetclient /]echo "172.17.0.2 puppetmaster.example.com" >>/etc/hosts"

Try to check if they are able to ping resolve both the nodes and once done your puppetmaster/puppetclient with docker images are ready to use.

try executing "testing" part from particle http://sunlnx.blogspot.in/2016/11/installconfigure-puppet-serverpuppet.html to see it all works as expected.

Thanks for re-sharing

​Install/configure puppet server/puppet client Guest OS - Linux 1/4


1. In this post I only wish to tell how to install and configure puppet master & puppet client on the Guest OS.
2. In my next post, we shall install docker to run puppet master and puppet client .... 
3. we will try to write few manifests also shall configure different environments for running puppet....

Docker Installed OS: Ubuntu 16.04
Tested on: RHEL/CentOS/Ubuntu 
Puppet Version: 3.8.7 

Let me first let you know how to install puppet server/puppet client and what change needs to be modified ?
Make sure your hostnames are resolvable, you could either configure DNS or add the hostnames in you /etc/hosts file.

Hostname & Descriptions of few major tools :

puppetmaster.example.com: (puppet master) 
This would be centralized managemnt daemon, and each manahed node will run puppet agent. It would serve compiled configuration, files, templates,
and custom plugins to managed nodes.

puppetclient.example.com: (puppet agent)   
puppet agent runs on each managed node, which will wake up every 30 mins by default to check with pupper master, send the new information about the system facts, and receive 'compiled catalog' describing the desired system configuration. puppet agent is then reponsible for making the system match the compile catalog. If 'pluginsync' is enabled in node configuration, custom plugins stored on pupper master are transferred automatically. 
puppet master then determines what information a given managed node should see based on unique identifier "certname". 

puppet apply:
runs puppet locally, to test manifests, non-networked case. it will not contact puppet master server, otherwise it just 'puppet agent' 

puppet cert: 
when the client contacts the server it will generate a certificate which should be signed by master to secure connection. 'autosign=true' will sign automatically when the clients connects to master server. 

#puppet cert list --all

if 'autosign' option not enabled, then you might require to sign, 
#puppet cert sign puppetclient.example.com

puppetmaster Install/Configure : 

Download RPM from puppetlabs to install puppet server

I had an issue with the time sync between puppetmaster and puppetclients and hence has to install 'ntp' and configure. 
it is not required for the VM, however that has fixed my issues hence thought to mention over here.
[root@puppetmaster ~]#yum install ntp

append any entry in 'server' part of the ntp.conf file

[root@puppetmaster ~]#vim /etc/ntp.conf
server <puppetmasterserver IP>

[root@puppetmaster ~]#service ntpd restart

[root@puppetmaster ~]#yum clean all; yum install puppet-server

Change your config files accordingly to your IP address and the hostnames.

[root@puppetmaster ~]#cat > /etc/puppet/puppet.conf
[main]
    certname = puppetmaster.example.com
    logdir = /var/log/puppet
    vardir = /var/lib/puppet
    rundir = /var/run/puppet
    ssldir = /var/lib/puppet/ssl
    factpath = $vardir/lib/facter

# This section is used by the Puppet master and Puppet cert applications.
[master]
    dns_alt_names = puppetmaster.example.com
    ssl_client_header = SSL_CLIENT_S_DN
    ssl_client_verify_header = SSL_CLIENT_VERIFY
    autosign = true
[root@puppetmaster ~]#

[root@puppetmaster ~]# puppet cert list --all
+ "puppetclient.example.com" (SHA256) EC:72:61:11:EA:C9:65:B4:43:B0:C7:45:56:38:40:A3:B4:85:E3:D9:27:8A:BB:56:BF:62:81:57:1A:91:AE:E0
+ "puppetmaster.example.com" (SHA256) EB:6B:BA:B1:03:8D:3B:CD:76:5D:46:9F:B9:2E:0D:3E:DF:70:8C:DE:35:F3:7D:D7:1A:81:FA:BA:63:31:DC:55 (alt names: "DNS:puppetmaster.example.com")
[root@puppetmaster ~]#

[root@puppetmaster ~]#service puppetmaster start

Ensure you have your master server has port listening

[root@puppetmaster ~]# netstat -tupln | grep 8140
tcp        0      0 0.0.0.0:8140                0.0.0.0:*                   LISTEN      -
[root@puppetmaster ~]#

Testing :

[root@puppetmaster ~]#cd /etc/puppet/manifests
[root@puppetmaster /etc/puppet/manifests]#cat > site.pp
file {'masterserver':
    ensure => present,
    path => '/tmp/masterserver',
    mode => 644,
    owner => root,
    group => root,
    content => 'on succsful test I must be born in /tmp/ and reader should study this message
'
}
[root@puppetmaster /etc/puppet/manifests]#

[root@puppetmaster /etc/puppet/manifests]# puppet apply site.pp
Notice: Compiled catalog for puppetmaster.example.com in environment production in 0.07 seconds
Notice: /Stage[main]/Main/File[masterserver]/ensure: created
Notice: Finished catalog run in 0.02 seconds
[root@puppetmaster /etc/puppet/manifests]# 

[root@puppetmaster ]# ls -l /tmp/masterserver
-rw-r--r-- 1 root root 78 Nov  2 02:27 /tmp/masterserver
[root@puppetmaster ~]#

[root@puppetmaster ~]# cat /tmp/masterserver
on succsful test I must be born in /tmp/ and reader should study this message
[root@puppetmaster ~]#

puppet client Install/Configure :

Download RPM from puppetlabs to install puppet server

I had an issue with the time sync between puppet master and puppet clients and hence has to install 'ntp' and configure. 
it is not required for the VM, however that has fixed my issues hence thought to mention over here.

[root@puppetclient ~] scp root@puppetmaster.example.com:/etc/ntp.conf /etc/ntp.conf
[root@puppetclient ~] service ntpd restart

[root@puppetclient ~]#yum clean all; yum install puppet

[root@puppetclient ~]# cat /etc/puppet/puppet.conf
[main]
    logdir = /var/log/puppet
    vardir = /var/lib/puppet
    rundir = /var/run/puppet
    ssldir = /var/lib/puppet/ssl
    factpath = $vardir/lib/facter

# This section is used by the Puppet master and Puppet cert applications.
[agent]
    ssl_client_header = SSL_CLIENT_S_DN
    ssl_client_verify_header = SSL_CLIENT_VERIFY
    server = puppetmaster.example.com
[root@puppetclient ~]#

[root@puppetclient ~]# puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for puppetclient.example.com
Info: Applying configuration version '1478054988'
Notice: /Stage[main]/Main/File[masterserver]/ensure: created
Notice: Finished catalog run in 0.02 seconds
[root@puppetclient ~]#

[root@puppetclient ~]# ls -l /tmp/masterserver
-rw-r--r-- 1 root root 78 Nov  2 02:51 /tmp/masterserver
[root@puppetclient ~]#

[root@puppetclient ~]# cat /tmp/masterserver
on succsful test I must be born in /tmp/ and reader should study this message
[root@puppetclient ~]#

Your puppet master and puppet client are ready, enjoy  !

Thanks for re-sharing