Showing posts with label source code. Show all posts
Showing posts with label source code. Show all posts

Mar 20, 2014

Few best practices on setting up Puppet 3 master/agent environment

Puppet is a configurations management tool like Chef and CFEngine. This tool is to manage configurations of large dynamically changing infrastructures like clouds efficiently. Puppet 3 is the latest release from PuppetLabs but still some operating system distributions does not include those packages in their repositories. So we need to some manual things to install puppet 3.

In this post I will explain few best practices to follow when installing a puppet master - agent environment. I have configure puppet master and agent environments several times and came across with this sequence and I think this a good way of doing this. But please note this not "the" best way of doing it and not recommended to use it as it is in a production environment. And also this will not describe about best practises of writing puppet manifests/modules.

Set a domain name for the environment
First of all use a domain name for your environment. Think that you are going to set up a puppet environment for ABC company, you can set the domain for that as 'abc.com' or 'dc1.abc.com' (data center 1 of ABC company). If you are doing it for testing purposes its advisory to use 'example.com'. 'example.com' is a reserved domain name for documentation and example purposes and no one can register that domain, so it will avoid many DNS resolution issues.

Give a proper FQDN for each hosts hostname.
Set a fully qualified domain name (FQDN) to each and every host within the puppet environment including the puppet master node. It will reduce lots of SSL related issues. It is not enough to just to give a hostname because most systems adds a domain (via DHCP) that will introduce some issues. Run 'hostname' and ' hostname -f ' and see the difference.

Use 'puppet' as a prefix as the puppet masters hostname. So it would be like;

    puppet.abc.com or
    puppet.cd1.adc.com or
    puppet.example.com

And for the puppet agents;

    8976712.apache.abc.com or
    8976712.apache.dc1.abc.com or
    8976712.apache.example.com

Or

    8976712.node001.abc.com or
    8976712.node002.dc1.abc.com or
    8976712.node003.example.com

Use a UUID when creating the hostnames for puppet agents. Then give the service name (apache,mysql) or the node number (node002 - if using multiple services in a single server). That name must match the node definitions in the 'site.pp' (or 'nodes.pp').

Use the 'hostname' command and edit the '/etc/hostname' configurations file to change the hostname. You can do it like this, assuming that the host is '8976712.node001.abc.com'

# hostname 8976712.node001.abc.com
# echo '8976712.node001.abc.com' >/etc/hostname

Give and IP address to each FQDN.
It is a must to give an appropriate IP addresses to each hostname/FQDN. At least, the system should be able to refer to the '/etc/hosts' file and resolve the IP address of the relevant FQDN and should have following entries in the '/etc/hosts' file.

    127.0.0.1 localhost
    127.0.0.1 < local fqdn >
    < puppet master ip > < puppet master fqdn >

For an example, if you take '8976712.node001.abc.com' node, its '/etc/hosts' file should like this.

    127.0.0.1 localhost
    127.0.0.1 8976712.node001.abc.com
    192.168.1.100 puppet.abc.com

Check the system time and timezone information
Both puppet master and agents should have same system time and time zone on both systems. Use 'date' command to check the system time and time zone. Synchronize the system time with a well known time server. Commands are bit different from one distribution to another.

Download and install puppet repositories from PuppetLabs website
PuppetLabs provide an apt and a yum repository. Most distributions does not support puppet 3 for the moment therefore, we need to add those manually.

Please refer to "Using the Puppet Labs Package Repositories" article and install the appropriate repository for your system. Then update your repository lists.

Install puppet master 
After completing all above steps, then try to install puppet master using a package management system (apt/yum).

It's better to go ahead with default setting. But you need to do few changes to some configuration files to make it work as a master-agent environment puppet master server. Use a 'autosign.conf' file to automatically sign agents SSL requests. But avoid using ' * ' in that. Better to use it like this;

*.abc.com

It's better to add the 'server=puppet.< domain >' in the 'puppet.conf  's 'main' section. On Debian based distros change the 'start' option in to 'yes' to start the puppet master. After configuring all restart the puppet master service. Open port 8140 from the system firewall specially check that if you are using any RedHat distribution.

Track changes
Use a version controlling system like git or subversion to track changes to puppet manifests. Use branching, versioning/tagging features to do it effectively.

Install puppet agent
First of all it is better to have puppet master installed. Then check the hostname and DNS resolutions for the hostname and puppet master. Then try to install puppet agent using a package management system.

You have to do few changes to connect to the puppet master server. Edit the '/etc/puppet/puppet.conf ' and add 'server=puppet.< domain >' to the 'main' section. Change the 'start' option to 'yes' in '/etc/default/puppet' configuration file in debian based distros. Then restart the puppet agent.

Test the system
Add this into your puppet masters '/etc/puppet/manifests/site.pp' file.
node default {
    file { '/tmp/mytestfile.t':
        owner   => 'root',
        group   => 'root',
        content => "This file was created by puppet.\n",
        ensure  => present,
    }
Then run 'puppet agent -vt ' on the agent and check the '/tmp ' directory.

Automated script
I wrote a script to automate this and you can get it from here on github. It support Debian, RedHat and SLES distributions. If you have any issues please report those to this.

Jan 13, 2013

Why we use GitHub in our final project?

As an undergraduate we need to do a project to complete our degree. According to my institute that should be a group project. After few reviews and discussions we select to build a secure and redundant backup system for a enterprise network. In the initial planning stage we realize that this project will have a heavy codding part.

Although we had a heavy codding part, no one had time to do the source code management because all four members of the team did a job while doing the degree. From the design we divide the project into several modules and assign each module to a particular member. But we knew at some point we need to merge those modules into a single unit. With work experience we knew that though a member develop a module, in some occasions others also need to edit the same module to make it interactive with the main system. Then we need to manage those changes and revisions.

Now we need to use some sort of a system to do this source code management task. Most of companies we work use Subversion or SVN, CVS and few use Git to do this task. But there was problem. All these tools required a centralized location to store the source code (repository). (Git works in a slightly different way as it is a distributed version controlling system.) We didn't have a public IP to host a small server or a Amazon-EC2 free tier cloud instance to use as a centralized location.

I remember that I heard something called Github while working. I almost had an account on github.com but I haven't done anything with that. We did some basic testing and agreed to use github.com as our central repository. In GitHub you can select a free or a paid service. As our project is a free and a open source project we decide to go with the free option.

GitHub or github.com is a web base hosting service for Git repositories. Simply it host or store your source code managed by the Git system. Git is a distributed version controlling system designed to handle everything from small to very large projects with speed and efficiency. The Linux kernel development project which is the biggest community driven project in the world is also managed by Git. GitHub converts this power tool in to a very user friendly and interesting tool to use by beginners. Even without prior experience any IT person can use this web base service to so the majority of tasks that performed by Git.

So we create a temporary repository and did some basic testing. Then I initialize new repository for the project and add other members in to that. I was responsible of developing the system core module. I did the initial coding and push it to github. Most of the IDEs support to do this task via the IDE itself. Others take a clone of the repository and they build their modules on top of that. We commit each change we made. It helps us to revert the changes if there was an error after the change. As commits are locally to the user, after a successful change each person push their changes to the central repository (github). Time to time we take updates from the central repository to make sure that the local copy is up to date. In few scenarios we had conflicts but we manage those easily with the help of IDEs.

Via the GitHubs web GUI we easily identify the changes made by each member. It have some graphs to provide information about the project and each members involvements. Anyone other than the specified team members can get a clone of the project but they want be able to push to the main repository. Only thing they can do is to submit a pull request. As the repository owner he/she can view the changes and have the authority to allow or deny the request. With that flexibility we can get more help from others without getting any trouble.


For more info :

Linus Torvalds visits Google to share his thoughts on Git :
http://www.youtube.com/watch?v=4XpnKHJAok8

GitHub help : https://help.github.com

Our project on GitHub : https://github.com/thilinapiy/SRDBS