Oct 14, 2017

Watch Kubernetes pod events stop/start

Run kube proxy to use curl without authentication

kubectl proxy

Run the curl to watch the events and filter it with jq for pod starts and stops.

curl -s | jq --raw-output \ 'if .object.reason == "Started" then . elif .object.reason == "Killing" then . else empty end | [.object.firstTimestamp, .object.reason, .object.metadata.namespace, .object.metadata.name] | @csv'

Output will be

"2017-10-14T02:51:31Z","Killing","default","echo-1788535470-v3089.14ed5012be9b81d2" "2017-10-14T02:17:12Z","Started","default","hello-minikube-180744149-7sbj2.14ed4e335345db3d" "2017-10-14T02:17:15Z","Started","kube-system","default-http-backend-27b99.14ed4e341737f843" "2017-10-14T02:17:11Z","Started","kube-system","kube-addon-manager-minikube.14ed4e33195f434a" "2017-10-14T02:17:14Z","Started","kube-system","kube-dns-910330662-78fqv.14ed4e33d9790ee6" "2017-10-14T02:17:15Z","Started","kube-system","kube-dns-910330662-78fqv.14ed4e33e88e1b68" "2017-10-14T02:17:15Z","Started","kube-system","kube-dns-910330662-78fqv.14ed4e3404bfcffc" "2017-10-14T02:17:13Z","Started","kube-system","kube-state-metrics-3741290554-5cv05.14ed4e337556350c" "2017-10-14T02:17:13Z","Started","kube-system","kube-state-metrics-3741290554-5cv05.14ed4e33804c8647" "2017-10-14T02:17:15Z","Started","kube-system","kubernetes-dashboard-8991s.14ed4e340386d190" "2017-10-14T02:54:57Z","Killing","kube-system","kubernetes-dashboard-8991s.14ed5042a8fa1c81" "2017-10-14T02:54:58Z","Started","kube-system","kubernetes-dashboard-xd7h5.14ed5042d33aa3c7" "2017-10-14T02:17:16Z","Started","kube-system","nginx-ingress-controller-9qn5l.14ed4e3426ecdaa8" "2017-10-14T02:55:23Z","Killing","kube-system","nginx-ingress-controller-9qn5l.14ed50489b820cce" "2017-10-14T02:55:37Z","Started","kube-system","nginx-ingress-controller-rf6j3.14ed504c01cf90ea" "2017-10-14T02:17:13Z","Started","kube-system","prometheus-3898748193-jgxzk.14ed4e339109bcb4" "2017-10-14T02:17:14Z","Started","kube-system","prometheus-3898748193-jgxzk.14ed4e33af0e9433"

Aug 29, 2017

Generate a SANs certificate

We are going to use openssl to generate a certificate with subject alternative names. When we use SANs in a certificate we can use the same certificate to front several websites with different domain names.

First we need to generate a private key. Since we are going to use this in a web server like Nginx or apache I'm not going to encrypt the private key with a passphrase.

openssl genrsa -out thilina.org.key 2048

Then we need to have a configurations file to add those alternative names into the certificate signing request (CSR).


[ req ]
default_bits       = 2048
distinguished_name = req_distinguished_name
req_extensions     = req_ext

[ req_distinguished_name ]
countryName         = Country Name (2 letter code)
stateOrProvinceName = State or Province Name (full name)
localityName        = Locality Name (eg, city)
organizationName    = Organization Name (eg, company)
commonName          = Common Name (e.g. server FQDN or YOUR name)

[ req_ext ]
subjectAltName = @alt_names


Now I'm going to generate the CSR in a single command.

openssl req -new -key thilina.org.key -sha256 -nodes -out thilina.org.csr \
  -subj "/C=LK/ST=Colombo/L=Colombo/O=Thilina Piyasundara/OU=Home/CN=thilina.org" \
  -config san.conf

Print and verify the CSR

openssl req -in thilina.org.csr -text -noout

Certificate Request:
        Version: 1 (0x0)
        Subject: C = LK, ST = Colombo, L = Colombo, O = Thilina Piyasundara, OU = Home, CN = thilina.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Exponent: 65537 (0x10001)
        Requested Extensions:
            X509v3 Subject Alternative Name:
                DNS:thilina.org, DNS:api.thilina.org, DNS:gateway.thilina.org
    Signature Algorithm: sha256WithRSAEncryption

Validate the key, csr and certificates are matching.

openssl rsa -noout -modulus -in domain.key | openssl md5
openssl x509 -noout -modulus -in domain.crt | openssl md5
openssl req -noout -modulus -in domain.csr | openssl md5

Feb 7, 2017

Running your spring-boot app in Bitesize

First of all we have to have the spring-boot code in a git(svn) repo. I have create a sample spring-boot application using maven archetypes. You can find the code in;


Compile the code and generate the package using following command;
# cd SpringBoot
# mvn clean package
This will build the app and create a jar file called 'SpringBoot-1.0.0.jar'.

We can run the application with following command and it will start it in port 8080.
# java -jar target/SpringBoot-1.0.0.jar
Now we switch to the next part. In this we need to update the bitesize files according to our needs.


First we'll update the 'build.bitesize' file. In this we need to update the projects and name accordingly and give the source code repo url and related details as in all other projects. But if you look at the shell commands you can see that I have modified few of those. I have add the 'mvn clean package' command and change the 'cp' command to copy the build jar to '/app' directory. Then it will build the deb as previous.
project: spring-dev
  - name: spring-app
    os: linux
      git: git@github.com:thilinapiy/SpringBoot.git
      branch: master
      - shell: sudo mkdir -p /app
      - shell: sudo mvn clean package
      - shell: sudo cp -rf target/*.jar /app
      - shell: sudo /usr/local/bin/fpm -s dir -n spring-app --iteration $(date "+%Y%m%d%H%M%S") -t deb /app
      - location: "*.deb"
Then we'll check the 'application.bitesize' file. I have change the 'runtime' to an ububtu-jdk8. Then change the command to run the jar.
project: spring-dev
  - name: spring-app
    runtime: ubuntu-jdk8:1.0
    version: "0.1.0"
      - name: spring-app 
        type: debian-package
          build: spring-app
        version: 1.0
    command: "java -jar /app/SpringBoot-1.0.0.jar"
In the 'environments.bitesize' I have update the port to 8080.
project: spring-dev
  - name: production
    namespace: spring-dev
      method: rolling-upgrade
      - name: spring-app
        external_url: spring.dev-bite.io
        port: 8080 
        ssl: "false"
        replicas: 2
In the stackstorm create_ns option give the correct manspace and the repo-url.
Reference : http://docs.prsn.io//deployment-pipeline/readme.html

Feb 4, 2017

Granting dbadmin privileges to a user in MongoDB cluster

We need to grant 'dbadmin' privileges to a user called 'store_db_user' to their mondo database in a 4 node cluster.

First we need to connect to the primary database of the cluster with super.

# mongo -u supperuser -p password -h node1.mongo.local

If you connect to the primary replica it will change the shell prompt to something like this;


Then you can list down the databases using following command.

mongoreplicas:PRIMARY>show dbs
admin     0.078GB
local     2.077GB
store_db  0.078GB

Then switch to the relevant database;

mongoreplicas:PRIMARY>use store_db

And grant permissions;

    { role: "dbOwner", db: "store_db" },

Exit from the admin user and login to the cluster as the database user.

# mongo -u store_db_user -p store_passwd -h node1.mongo.local/store_db

Validate the change.

mongoreplicas:PRIMARY>show users
	"_id" : "store_db.store_db_user",
	"user" : "store_db_user",
	"db" : "store_db",
	"roles" : [
			"role" : "dbOwner",
			"db" : "store_db"
			"role" : "readWrite",
			"db" : "store_db"

Apr 27, 2016

Running your WordPress blog on WSO2 App Cloud

WSO2 App Cloud is now supporting Docker base PHP applications. In this blog post I will describe how to install a WordPress blog in this environment. In order to setup a WordPress environment we need to have two things;

  1. Web server with PHP support
  2. MySQL database

If we have both of these we can start setting up WordPress. In WSO2 App Cloud we can use the PHP application as the WordPress hosting web server which is a PHP enabled Apache web server docker image. Also it provides a database service where you can easily create and manage MySQL databases via the App Cloud user interface (UI).

For the moment WSO2 App Cloud is on beta therefore these docker images will have only 12h of lifetime with no data persistence in the file storage level. Data on MySQL databases will be safe unless you override. If you need more help don't hesitate to contact Cloud support.

Creating PHP application

Signup or signin to WSO2 App Cloud via http://wso2.com/cloud. Then click on the "App Cloud beta" section.
Then it will redirect you to the App Cloud user interface. Click on 'Add New Application' button on the left hand corner.
This will prompt you to several available applications. Select 'PHP Web Application' box and continue.
Then it will prompt you a wizard. In that give a proper name and a version to your application. Name and version will be use to generate the domain name for your application.

There are several options that you can use to upload PHP content to this application. For the moment I will download the wordpress-X.X.X.zip file from the wordpress site and upload it to application.
In the below sections of the UI you can set the run time and container specification. Give the highest Apache version as the runtime and use minimal container speck as wordpress does not require much processing and memory.
If the things are all set and the file upload is complete click on 'Create' button. You will get the following status pop-up when you click the create button and it will redirect you to the application when its complete.
In the application UI note the URL. Now you can click on the 'Launch App' button so that it will redirect you to your PHP application.
Newly installed WordPress site will be like this.
Now we need to provide database details to it. Therefore, we need to create database and a user.

Creating database

Go back to the Application UI and click on 'Back to listing' button.
In that UI you can see a button in the top left hand corner called 'Create database'. Click on that.
In the create database UI give a database name, database user name and a password . Password need to pass the password policy so you can click on 'Generate password' to generate a secure password easily. By the way of you use generate password option make sure you copy the generated password before you proceed with database creation. Otherwise you may need to reset the password.

Also note that database name and database user name will append tenant domain and random string accordingly to the end of both. Therefore, those fields will only get few number of input characters.
If all set then click on 'Create database' button to proceed. After successfully creating the database it will redirect you to a database management user interface like following.
Now you can use those details to login to the newly create mysql database as follows;
$ mysql -h mysql.storage.cloud.wso2.com -p'' -u
eg :-
$ mysql -h mysql.storage.cloud.wso2.com -p'XXXXXXXXX' -u admin_LeWvxS3l wpdb_thilina 
Configuring WordPress

If the database creation is successful and you can login to it without any issue we can use those details to configure WordPress.

Go back to the WordPress UI and click on 'let's go' button. It will prompt to a database configuration wizard. Fill those fields with the details that we got from the previous section.
If WordPress application can successfully establish a connection with the database using your inputs it will prompt you to a UI as follows.
On that click on 'Run the install'. Then WordPress will start populating database tables and insert initial data to the given database.

When its complete it will ask for some basic configurations like the site title, admin user name and passwords.
Click on 'Install WordPress' after filling those information. Then it will redirect you to the WordPress admin console login page. Login to that using the username and password gave in the previous section.
So now WordPress is ready to use. But the existing URL is not very attractive. If you have a domain you can use it as the base URL of this application.

Setting custom domain (Optional)

IN the application UI click on the top left three lines button shown in the following image.
It will show some advance configuration that we can use. In that list select the last one 'Custom URL' option.
It will prompt you following user interface. Enter the domain name that you are willing to use.
But before you validate make sure you add a DNS CNAME to that domain pointing to you application launch URL.

Following is the wizard that I got when adding the CNAME via Godaddy. This user interface and adding CNAME options will be different for you DNS provider.
You can validate the CNAME by running 'dig' command in Linux or nslookup in windows.
If the CNAME is working click on 'Update'.
 If that is successful you will get the above notification and if you access that domain name it will show your newly created WordPress blog.

Apr 2, 2016

Add Let's Encrypt free SSL certificates to WSO2 API Cloud

Let's encrypt is a free and open certificate authority runs for the public benefit. This service is provided by the Internet Security Research Group and there are lots of companies working with them to make the Internet secure. People who have a domain name can get free SSL certificate for their websites using this service for three months. I they need to use for more than that three months we need to renew the certificate and its also for free. But the best thing is that this certificate is accepted by most of the new web browsers and systems by default. So you don't need to add CA certs to you browsers any more.

In this article I will explain how we can use that service to get a free SSL certificate and add that to WSO2 API Cloud. So that you can have your own API store like;


In order to do that you need to have following things in hand.
  • Domain name.
  • Rights to add/delete/modify DNS A records and CNAMEs.
  • Publicly accessible webserver with root access or a home router with port forwarding capabilities. 

Step 1

If you have a publicly accessible webserver you can skip this step.If you don't have a publicly accessible webserver you can make your home PC/Laptop a temporary webserver if you can do port forwarding/NATing in you home router. I will show how I did that with my ADSL router. You can get help on port forwarding information by referring to this website http://portforward.com.

a. Add a port forwarding rule in your home router.

Get your local (laptop) IP (by running ifconfig/ip addr) and put that as the backend server in your router for. Set the WAN port as 80 and LAN port as 80.

After adding the rule it will be like this.

b. Start a webserver in your laptop. We can use the simple Python server for this. Make sure to check the IPTable rules/Firewall rules.

mkdir /tmp/www
cd /tmp/www/
echo 'This is my home PC :)' > index.html
sudo python3 -m http.server 80

c. Get the public IP of your router. Go to this link : http://checkip.dyndns.org it will give the public IP address. This IP is changing time-to-time so no worries.

d. Try to access that IP from a browser.
If it is giving the expected output you have a publicly accessible webserver.

Step 2

Now we need to update a DNS entry. My expectation is to have a single SSL certificate for both domains 'store.thilina.piyasundara.org' and 'api.thilina.piyasundara.org'.

a. Go to your DNS provides console and add an A record for both domain names to point to the public IP of your webserver (or the IP that we got from the previous step).

b. Try to access both via a browser and if its giving the expected out put you can proceed to the next step.

Step 3

I'm follow the instruction in the 'let's encrypt' guide. As I'm using the python server I need to use the 'certonly' option when running the command to generate the certs.

a. Get the git clone of the letsencrypt project.

git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt

b. Run cert generation command. (this requires root/sudo access)

./letsencrypt-auto certonly --webroot -w /tmp/www/ -d store.thilina.piyasundara.org -d api.thilina.piyasundara.org

If this succeed you can find the SSL keys and certs in '/etc/letsencrypt/live/store.thilina.piyasundara.org' location.

Step 4

Check the content of the certs. (Be root before you try to 'ls' that directory)

openssl x509 -in cert.pem -text -noout

Step 5

Create an API in WSO2 API Cloud if you don't have one. Else start on adding a custom domain to your tenant.

a. Remove both A records and add CNAME records to those two domains. Both should point to the domain 'customdns.api.cloud.wso2.com'.

b. Now click on the 'Configure' option in the top options bar and select the 'Custom URL' option.

c. Make ready you SSL certs. Copy 'cert.pem', 'chain.pem' and 'privkey.pem' to you home directory.

d. Modify API store domain. Click on the modify button, add the domain name click on verify. It will take few seconds. If that succeed you have correctly configured the CNAME to point to WSO2 cloud.

e. Add cert files to the API Cloud. The order should be the certificate (cert.pem), private key (privatekey.pem) and the CAs chain file (chain.pem). Again it will take sometime to verify uploaded details.

f. Update the gateway domain same as the previous.

Now if you go the API Store it will show something like this.

g. Same way you can use the gateway domain when you need to invoke APIs.

curl -X GET --header 'Accept: application/json' --header 'Authorization: Bearer ' 'https://gateway.api.cloud.wso2.com:8243/t/thilina/gituser/1.0.0/thilinapiy'

Now you don't need '-k' option. If not make sure you operating system (CA list) is up to date.

Step 6

Make sure to remove port forwarding in you home router if you use that and any changes that you make while obtaining the SSL certificates.