May 7, 2014

Run WSO2 products in a Docker container

Docker is an open-source project to easily create lightweight, portable, self-sufficient containers from any application. There are two ways to run docker container;

1. Run a pre-build docker image.
2. Build your own docker image and use it.

In the first option you can use a base image like Ubuntu, CentOS or an image built by someone else like thilina/ubuntu_puppetmaster. You can find these images index.docker.io

In the second option you can build the image using a "Dockerfile". In this approach we can do customizations to the container by editing this file.

When creating a docker container for WSO2 products option 2 is the best. I have wrote a sample Dockerfile on github. It describes how to build a Docker container for WSO2 API manager single node implementation. For the moment docker have some limitations like unable to edit the '/etc/hosts' file, etc. If you need to create a clusters of WSO2 products (an API manager cluster in this case) you need to do some additional things like setting up a DNS server, etc.

How to build an API manager docker container?


Get a git clone of the build repository.
git clone https://github.com/thilinapiy/dockerfiles
Download Oracle JDK 7 tar.gz (not JDK 8) and place it in '/dockerfiles/base/dist/'
mv /jdk-7u55-linux-x64.tar.gz /dockerfiles/base/dist/
Download WSO2 API manager and place that in '/dockerfiles/base/dist/'
mv /wso2am-1.6.0.zip /dockerfiles/base/dist/
Change directory to '/dockerfiles/base/'.
cd dockerfiles/base/
Run docker command to build image.
docker build -t apim_image .

How to start API manager from the build image?


Start in interactive mode
docker run -i -t --name apim_test apim_image
Start in daemon mode
docker run -d    --name apim_test apim_image
Other options that can use when starting a docker image
--dns  < dns server address >
--host < hostname of the container >

Major disadvantages in docker (for the moment)

  • Can't edit the '/etc/hosts' file in the container.
  • Can't edit the '/etc/hostname' file. --host option can use to set a hostname when starting.
  • Can't change DNS server settings in '/etc/resolve.conf'. --dns option can use to set DNS servers. Therefore, if you need to create a WSO2 product cluster you need to setup a  DNS server too.

Read more about WSO2 API manager : Getting Started with API Manager


May 6, 2014

Python 3 appindicator example script for Ubuntu 14.04

On Ubuntu 14.04 Python 3 is the default python version. Therefor If you try to run previous appindicator scripts on Ubuntu 14.04 those will not work. This script is done using Python 3 and relevant libraries.

Loading ....

Mar 20, 2014

Few best practices on setting up Puppet 3 master/agent environment

Puppet is a configurations management tool like Chef and CFEngine. This tool is to manage configurations of large dynamically changing infrastructures like clouds efficiently. Puppet 3 is the latest release from PuppetLabs but still some operating system distributions does not include those packages in their repositories. So we need to some manual things to install puppet 3.

In this post I will explain few best practices to follow when installing a puppet master - agent environment. I have configure puppet master and agent environments several times and came across with this sequence and I think this a good way of doing this. But please note this not "the" best way of doing it and not recommended to use it as it is in a production environment. And also this will not describe about best practises of writing puppet manifests/modules.

Set a domain name for the environment
First of all use a domain name for your environment. Think that you are going to set up a puppet environment for ABC company, you can set the domain for that as 'abc.com' or 'dc1.abc.com' (data center 1 of ABC company). If you are doing it for testing purposes its advisory to use 'example.com'. 'example.com' is a reserved domain name for documentation and example purposes and no one can register that domain, so it will avoid many DNS resolution issues.

Give a proper FQDN for each hosts hostname.
Set a fully qualified domain name (FQDN) to each and every host within the puppet environment including the puppet master node. It will reduce lots of SSL related issues. It is not enough to just to give a hostname because most systems adds a domain (via DHCP) that will introduce some issues. Run 'hostname' and ' hostname -f ' and see the difference.

Use 'puppet' as a prefix as the puppet masters hostname. So it would be like;

    puppet.abc.com or
    puppet.cd1.adc.com or
    puppet.example.com

And for the puppet agents;

    8976712.apache.abc.com or
    8976712.apache.dc1.abc.com or
    8976712.apache.example.com

Or

    8976712.node001.abc.com or
    8976712.node002.dc1.abc.com or
    8976712.node003.example.com

Use a UUID when creating the hostnames for puppet agents. Then give the service name (apache,mysql) or the node number (node002 - if using multiple services in a single server). That name must match the node definitions in the 'site.pp' (or 'nodes.pp').

Use the 'hostname' command and edit the '/etc/hostname' configurations file to change the hostname. You can do it like this, assuming that the host is '8976712.node001.abc.com'

# hostname 8976712.node001.abc.com
# echo '8976712.node001.abc.com' >/etc/hostname

Give and IP address to each FQDN.
It is a must to give an appropriate IP addresses to each hostname/FQDN. At least, the system should be able to refer to the '/etc/hosts' file and resolve the IP address of the relevant FQDN and should have following entries in the '/etc/hosts' file.

    127.0.0.1 localhost
    127.0.0.1 < local fqdn >
    < puppet master ip > < puppet master fqdn >

For an example, if you take '8976712.node001.abc.com' node, its '/etc/hosts' file should like this.

    127.0.0.1 localhost
    127.0.0.1 8976712.node001.abc.com
    192.168.1.100 puppet.abc.com

Check the system time and timezone information
Both puppet master and agents should have same system time and time zone on both systems. Use 'date' command to check the system time and time zone. Synchronize the system time with a well known time server. Commands are bit different from one distribution to another.

Download and install puppet repositories from PuppetLabs website
PuppetLabs provide an apt and a yum repository. Most distributions does not support puppet 3 for the moment therefore, we need to add those manually.

Please refer to "Using the Puppet Labs Package Repositories" article and install the appropriate repository for your system. Then update your repository lists.

Install puppet master 
After completing all above steps, then try to install puppet master using a package management system (apt/yum).

It's better to go ahead with default setting. But you need to do few changes to some configuration files to make it work as a master-agent environment puppet master server. Use a 'autosign.conf' file to automatically sign agents SSL requests. But avoid using ' * ' in that. Better to use it like this;

*.abc.com

It's better to add the 'server=puppet.< domain >' in the 'puppet.conf  's 'main' section. On Debian based distros change the 'start' option in to 'yes' to start the puppet master. After configuring all restart the puppet master service. Open port 8140 from the system firewall specially check that if you are using any RedHat distribution.

Track changes
Use a version controlling system like git or subversion to track changes to puppet manifests. Use branching, versioning/tagging features to do it effectively.

Install puppet agent
First of all it is better to have puppet master installed. Then check the hostname and DNS resolutions for the hostname and puppet master. Then try to install puppet agent using a package management system.

You have to do few changes to connect to the puppet master server. Edit the '/etc/puppet/puppet.conf ' and add 'server=puppet.< domain >' to the 'main' section. Change the 'start' option to 'yes' in '/etc/default/puppet' configuration file in debian based distros. Then restart the puppet agent.

Test the system
Add this into your puppet masters '/etc/puppet/manifests/site.pp' file.
node default {
    file { '/tmp/mytestfile.t':
        owner   => 'root',
        group   => 'root',
        content => "This file was created by puppet.\n",
        ensure  => present,
    }
Then run 'puppet agent -vt ' on the agent and check the '/tmp ' directory.

Automated script
I wrote a script to automate this and you can get it from here on github. It support Debian, RedHat and SLES distributions. If you have any issues please report those to this.

Feb 4, 2014

KVM with Virt-Manager as a virtualization tool for Linux

I have use several operating system (OS) level virtualization tools like VMware Workstation, VMware Player, Oracle VirtualBox, Microsoft VirtualPC and KVM for many years.

Overall VMware Workstation is the best tool for me. But to use that we need to purchase a license.  As an alternative to VMware Workstation we can use VMware player (a stripdown version of WMware Workstation) for free but only for non-commercial use. Also you can't run multiple guest operating systems concurrently using that.

As an alternative to VMware products most of linux people use Oracle VirtualBox. I had some issues when I try to NAT a virtual instance (guest OS) on Ubuntu 12.10 (host OS) machine. As a solution for this most blogs forums suggest to change the virtual network option into Bridge. But most networks (including my home wifi network) doesn't allow this option because we do MAC address filtering.

Obviously you can not install Microsoft VirtualPC on a Linux host (even with vine). Truly I haven't use Xen, so I can't give any opinions on that.

kernel-based virtual machine (KVM) is another tool that we can use to do OS level virtualization. I will explain how to install KVM and Virt-Manager the graphical user interface which can use to interact with KVM on Ubuntu (Check this to install KVM on CentOS).

Installing KVM and Virt-manager on Ubuntu


Update your repository list
sudo apt-get update
Install packages and dependencies
sudo apt-get install kvm virt-manager
After completing the installation you can search for "Virtual machine Manager" on the search of Ubuntu Unity. Give the sudo password in the popup window.

Or else you can use the following command to start the virt-manager GUI from the terminal.
sudo virt-manager
In the first attempt it will prompt the following user interface. Use the default settings and click on "connect".

KVM Virt-Manager add connection

Installing guest operating system in KVM


Open virt-manager by clicking on the icon from unity search or using command. Then click on the left most icon of the GUI as follows. It will open a wizard to create a new virtual machine.

Create new virtual machine


Give a proper name, that name will appear on the virt-manager virtual machine list.

There are several ways to install an operating system and this tool also support few of them too. Usually we use a boot-able CD/DVD to install an operating system to a new machine/laptop. Also we can directly give the ISO image of an operating system.  Therefore, I will go ahead with the "Local installation media" option and click on "Forward"


In this I'm going to give an Ubuntu 12.04 desktop ISO image. Select the "Use ISO image" option and click on "Browse".


It will open another window which will list down all the image files of your virtual machines. Click on the bottom "Browse Local" button and browse to the ISO image that you need to install. Give the OS type and version in relevant fields. Then continue the wizard.


You can download those ISO images from relevant websites other than Windows.

In this step you need to set the guest machines memory and CPU. As I'm going to install a Ubuntu Desktop with the user interface (UI), I'm going to give 1024 MB of RAM and a single CPU will work for this.


Now we need to set the disk space. It is enough to give 8-10GB of disk space for a virtual (guest) machine.

Specially, remove the default check on the "Allocate entire disk now" option. If you do so, KVM will not allocate full 10GB (or what you set) from your host machines hard disk. It will only use the real data capacity used in the installation and only when you add data to the guest system it will grow. So this save lot of disk space.


In the last step you will get a confirmation page.

Few best practices when using KVM

  • To run virtual machine manager you need sudo permission. So you can create an alias for this.
alias virt-manager='sudo virt-manager'
sudo visudo
  • Add the line with your username
username ALL= NOPASSWD: ALL
  • Try to use text only installations of operating systems. It will reduce resources (RAM/disk space) usage. Most server edition operating systems by default install the text only (run level 3) environment.
  • Use base images
Create some base operating system installations in the system. If you need a virtual machine you can get a clone of the base installation and use. In this figure I have create three base images (virtual machines)
a CentOS, RHEL and a Ubuntu.

Other three machines are clones of a Ubuntu base machine which I used to simulate a Ubuntu base network. After the simulation I can delete those virtual machines with used virtual hard disk (.img file).


Install your favorite commonly used tools like vim/emacs, tree, htop, telnet, git, subversion, oracle JDK, links, debug tools and custom scripts.

Install additional repositories like EPEL, RPMForge,  repos on RedHat base distributions. Puppet repositories on all distributions.

So how to clone a virtual machine? Right click on the virtual machine (should be on power off state) and select "Clone" option. It will give the following window. Give a proper name for that and continue it. It will take few minutes to copy and the time will depend on your base image size.


When you need delete a virtual machine, select and right click on that and select "Delete" option. It will prompt another window as follows. Select the "Delete associated storage files" option and it will enable the list of storages which you need to delete in order to save your disk space. Keep in mind not to delete any iso images if those appear on this list.