Nov 11, 2017

Run a Mongo cluster with authentication on kubernetes using StatefulSets.

With StatefulSets running a mongo cluster with persistent storage is easy. Kubernetes blog post in [1] explains how to do that. But when we try to enable authentication there is a small problem with the above solution. Problem is we need to start the cluster without replica set (--replSet) option and add the admin user and a key file, then restart with the replica set and the key file.

In order to do that we need to modify the above sample code to add a pod life cycle - post start command.

    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo:3.4.9
          command:
          - /bin/sh
          - -c
          - >
            if [ -f /data/db/admin-user.lock ]; then
              mongod --replSet rs0 --clusterAuthMode keyFile --keyFile /etc/secrets/mongo.key --setParameter authenticationMechanisms=SCRAM-SHA-1;
            else
              mongod --auth;
            fi;
          lifecycle:
            postStart:
              exec:
                command:
                - /bin/sh
                - -c
                - >
                  if [ ! -f /data/db/admin-user.lock ]; then
                    sleep 5;
                    touch /data/db/admin-user.lock
                    if [ "$HOSTNAME" = "mongo-0" ]; then
                      mongo --eval 'db = db.getSiblingDB("admin"); db.createUser({ user: "admin", pwd: "password", roles: [{ role: "root", db: "admin" }]});';
                    fi;
                    mongod --shutdown;
                  fi;
          ports:
            - containerPort: 27017
If you look at the command section of the above statefulSet part you can see I'm checking for a lock file. If that file exists I will start the cluster with replica set option. Else I will start the cluster with just auth.

Then you can notice the postStart section. In post start section we can define what to do at the start of a container in a pod. You can refer to this post in [2] to get an idea of pod lifecycle. In that it will again check for the lock file. If the file does not exists it will create the file and if the hostname is equal to mongo-o it will add the mongo admin user to the cluster. Then it will stop the node. Stopping the node will result the container to restart. Since we are using persistent storage the initially created lock file will exist. Since the lock file is there command will start the cluster with replica set option and postStart will pass its loop.

Running the mongo cluster

Generate a random key to enable keyFile authentication replication as described in this post.

openssl rand -base64 741 > mongodb-keyfile
Create a secret using that random string.
kubectl create secret generic mongo-key --from-file=mongodb-keyfile
Create statefulSet and headless service.
kubectl create -f  https://gist.githubusercontent.com/thilinapiy/0c5abc2c0c28efe1bbe2165b0d8dc115/raw/d3d0e64dfd35158907d076422c362f289d124dfc/mongo-statefulset.yaml
Scale-up if you need mote high availability.
kubectl scale --replicas=3 statefulset mongo
Let me know if you need further help.

1. http://blog.kubernetes.io/2017/01/running-mongodb-on-kubernetes-with-statefulsets.html
2. https://blog.openshift.com/kubernetes-pods-life
3. https://docs.mongodb.com/manual/tutorial/enforce-keyfile-access-control-in-existing-replica-set/

Oct 14, 2017

Watch Kubernetes pod events stop/start


Run kube proxy to use curl without authentication

kubectl proxy

Run the curl to watch the events and filter it with jq for pod starts and stops.

curl -s 127.0.0.1:8001/api/v1/watch/events | jq --raw-output \ 'if .object.reason == "Started" then . elif .object.reason == "Killing" then . else empty end | [.object.firstTimestamp, .object.reason, .object.metadata.namespace, .object.metadata.name] | @csv'

Output will be

"2017-10-14T02:51:31Z","Killing","default","echo-1788535470-v3089.14ed5012be9b81d2" "2017-10-14T02:17:12Z","Started","default","hello-minikube-180744149-7sbj2.14ed4e335345db3d" "2017-10-14T02:17:15Z","Started","kube-system","default-http-backend-27b99.14ed4e341737f843" "2017-10-14T02:17:11Z","Started","kube-system","kube-addon-manager-minikube.14ed4e33195f434a" "2017-10-14T02:17:14Z","Started","kube-system","kube-dns-910330662-78fqv.14ed4e33d9790ee6" "2017-10-14T02:17:15Z","Started","kube-system","kube-dns-910330662-78fqv.14ed4e33e88e1b68" "2017-10-14T02:17:15Z","Started","kube-system","kube-dns-910330662-78fqv.14ed4e3404bfcffc" "2017-10-14T02:17:13Z","Started","kube-system","kube-state-metrics-3741290554-5cv05.14ed4e337556350c" "2017-10-14T02:17:13Z","Started","kube-system","kube-state-metrics-3741290554-5cv05.14ed4e33804c8647" "2017-10-14T02:17:15Z","Started","kube-system","kubernetes-dashboard-8991s.14ed4e340386d190" "2017-10-14T02:54:57Z","Killing","kube-system","kubernetes-dashboard-8991s.14ed5042a8fa1c81" "2017-10-14T02:54:58Z","Started","kube-system","kubernetes-dashboard-xd7h5.14ed5042d33aa3c7" "2017-10-14T02:17:16Z","Started","kube-system","nginx-ingress-controller-9qn5l.14ed4e3426ecdaa8" "2017-10-14T02:55:23Z","Killing","kube-system","nginx-ingress-controller-9qn5l.14ed50489b820cce" "2017-10-14T02:55:37Z","Started","kube-system","nginx-ingress-controller-rf6j3.14ed504c01cf90ea" "2017-10-14T02:17:13Z","Started","kube-system","prometheus-3898748193-jgxzk.14ed4e339109bcb4" "2017-10-14T02:17:14Z","Started","kube-system","prometheus-3898748193-jgxzk.14ed4e33af0e9433"

Aug 29, 2017

Generate a SANs certificate

We are going to use openssl to generate a certificate with subject alternative names. When we use SANs in a certificate we can use the same certificate to front several websites with different domain names.

First we need to generate a private key. Since we are going to use this in a web server like Nginx or apache I'm not going to encrypt the private key with a passphrase.


openssl genrsa -out thilina.org.key 2048


Then we need to have a configurations file to add those alternative names into the certificate signing request (CSR).

sans.conf

[ req ]
default_bits       = 2048
distinguished_name = req_distinguished_name
req_extensions     = req_ext

[ req_distinguished_name ]
countryName         = Country Name (2 letter code)
stateOrProvinceName = State or Province Name (full name)
localityName        = Locality Name (eg, city)
organizationName    = Organization Name (eg, company)
commonName          = Common Name (e.g. server FQDN or YOUR name)

[ req_ext ]
subjectAltName = @alt_names

[alt_names]
DNS.1=thilina.org
DNS.2=api.thilina.org
DNS.3=gateway.thilina.org


Now I'm going to generate the CSR in a single command.


openssl req -new -key thilina.org.key -sha256 -nodes -out thilina.org.csr \
  -subj "/C=LK/ST=Colombo/L=Colombo/O=Thilina Piyasundara/OU=Home/CN=thilina.org" \
  -config san.conf


Print and verify the CSR


openssl req -in thilina.org.csr -text -noout



Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: C = LK, ST = Colombo, L = Colombo, O = Thilina Piyasundara, OU = Home, CN = thilina.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d0:13:91:5d:62:7c:4f:57:6d:4c:79:85:59:d8:
                    c5:ae:50:41:cc:db:fe:b4:75:fc:c1:73:e7:a7:ac:
                    89:36:3b:26:08:0f:33:b0:96:5c:29:a1:ee:9a:14:
                    13:4b:5b:43:74:74:a2:fd:97:2b:2b:bd:2a:b8:e6:
                    22:d2:01:15:f3:7f:e9:d8:c9:d4:65:04:5a:ef:f0:
                    03:41:63:56:39:eb:5f:e5:90:de:33:b7:bb:60:0e:
                    e3:70:79:60:8f:cb:a9:71:3b:e3:0a:b1:17:47:aa:
                    41:08:b5:44:5e:1a:a1:fa:a2:ce:ed:18:c5:a3:b0:
                    6f:0f:57:ca:ae:28:7f:91:49:14:6b:94:4c:3c:33:
                    fb:27:ed:77:37:a7:d6:54:4e:a7:6e:bc:c9:a2:a1:
                    b5:f2:f0:aa:76:64:04:83:96:92:03:36:4c:3e:14:
                    0e:97:a6:79:9e:23:c1:2a:c4:7a:3d:6e:f3:1c:40:
                    e3:d1:61:f2:56:51:8f:0f:04:76:62:ea:b0:1f:94:
                    e8:a8:8b:54:d6:08:5a:79:a6:a4:a0:00:fb:5f:c3:
                    d5:d4:50:ea:15:12:ea:9b:10:cc:9a:d9:32:6e:48:
                    93:30:4b:e7:2e:fe:a9:a0:31:16:61:24:3f:29:54:
                    2a:25:da:d2:b3:6a:d9:d5:a9:51:ee:d3:bb:b9:83:
                    86:59
                Exponent: 65537 (0x10001)
        Attributes:
        Requested Extensions:
            X509v3 Subject Alternative Name:
                DNS:thilina.org, DNS:api.thilina.org, DNS:gateway.thilina.org
    Signature Algorithm: sha256WithRSAEncryption
         96:44:43:98:60:76:49:ad:8b:01:65:20:f1:ca:4a:47:84:67:
         dc:77:f0:2e:bb:30:68:8b:2f:79:c4:4c:10:91:ec:70:fe:73:
         9c:3e:f4:69:18:8c:34:f6:85:05:26:b1:2a:35:38:f5:93:59:
         c2:a4:07:83:73:79:88:9b:ff:17:99:66:34:58:21:bc:de:8e:
         65:b9:50:bb:18:52:53:9b:ed:a3:4e:c7:55:73:2e:42:47:dc:
         94:4d:fb:cc:ba:b1:7a:57:a6:f9:fa:27:a2:54:aa:cd:f6:79:
         3d:b7:0a:82:a3:18:41:ec:f5:db:cc:05:6a:43:64:d7:4a:00:
         fe:a3:89:f9:25:f3:79:55:f9:79:3a:b2:96:5e:9d:67:f5:c7:
         e4:ab:fc:da:cb:df:f5:76:36:44:fe:d2:87:3a:d7:a2:a9:2e:
         fc:7f:ba:a6:12:44:70:e0:c4:42:57:01:1e:51:0a:d4:2e:33:
         e2:63:20:c2:9a:07:1b:78:e8:fb:42:b5:e5:85:00:b1:2c:25:
         d8:ad:43:af:6a:01:09:59:7e:d0:af:dd:72:f3:93:18:30:38:
         c2:b0:6c:8e:88:79:4e:16:fe:e3:87:46:c2:eb:f3:2e:2b:aa:
         a7:a9:76:1d:fd:8b:d9:d9:1c:a3:1c:21:db:af:b0:0b:7e:15:
         37:37:0f:25



Validate the key, csr and certificates are matching.

openssl rsa -noout -modulus -in domain.key | openssl md5
openssl x509 -noout -modulus -in domain.crt | openssl md5
openssl req -noout -modulus -in domain.csr | openssl md5


Feb 7, 2017

Running your spring-boot app in Bitesize

First of all we have to have the spring-boot code in a git(svn) repo. I have create a sample spring-boot application using maven archetypes. You can find the code in;

https://github.com/thilinapiy/SpringBoot

Compile the code and generate the package using following command;
# cd SpringBoot
# mvn clean package
This will build the app and create a jar file called 'SpringBoot-1.0.0.jar'.

We can run the application with following command and it will start it in port 8080.
# java -jar target/SpringBoot-1.0.0.jar
Now we switch to the next part. In this we need to update the bitesize files according to our needs.

https://github.com/thilinapiy/bitesize-spring

First we'll update the 'build.bitesize' file. In this we need to update the projects and name accordingly and give the source code repo url and related details as in all other projects. But if you look at the shell commands you can see that I have modified few of those. I have add the 'mvn clean package' command and change the 'cp' command to copy the build jar to '/app' directory. Then it will build the deb as previous.
project: spring-dev
components:
  - name: spring-app
    os: linux
    repository:
      git: git@github.com:thilinapiy/SpringBoot.git
      branch: master
    build:
      - shell: sudo mkdir -p /app
      - shell: sudo mvn clean package
      - shell: sudo cp -rf target/*.jar /app
      - shell: sudo /usr/local/bin/fpm -s dir -n spring-app --iteration $(date "+%Y%m%d%H%M%S") -t deb /app
    artifacts:
      - location: "*.deb"
Then we'll check the 'application.bitesize' file. I have change the 'runtime' to an ububtu-jdk8. Then change the command to run the jar.
project: spring-dev
applications:
  - name: spring-app
    runtime: ubuntu-jdk8:1.0
    version: "0.1.0"
    dependencies:
      - name: spring-app 
        type: debian-package
        origin:
          build: spring-app
        version: 1.0
    command: "java -jar /app/SpringBoot-1.0.0.jar"
In the 'environments.bitesize' I have update the port to 8080.
project: spring-dev
environments:
  - name: production
    namespace: spring-dev
    deployment:
      method: rolling-upgrade
    services:
      - name: spring-app
        external_url: spring.dev-bite.io
        port: 8080 
        ssl: "false"
        replicas: 2
In the stackstorm create_ns option give the correct manspace and the repo-url.
Reference : http://docs.prsn.io//deployment-pipeline/readme.html

Feb 4, 2017

Granting dbadmin privileges to a user in MongoDB cluster

We need to grant 'dbadmin' privileges to a user called 'store_db_user' to their mondo database in a 4 node cluster.

First we need to connect to the primary database of the cluster with super.

# mongo -u supperuser -p password -h node1.mongo.local

If you connect to the primary replica it will change the shell prompt to something like this;

mongoreplicas:PRIMARY>

Then you can list down the databases using following command.

mongoreplicas:PRIMARY>show dbs
admin     0.078GB
local     2.077GB
store_db  0.078GB

Then switch to the relevant database;

mongoreplicas:PRIMARY>use store_db

And grant permissions;

mongoreplicas:PRIMARY>db.grantRolesToUser(
  "store_db_user",
  [
    { role: "dbOwner", db: "store_db" },
  ]
)

Exit from the admin user and login to the cluster as the database user.

# mongo -u store_db_user -p store_passwd -h node1.mongo.local/store_db

Validate the change.

mongoreplicas:PRIMARY>show users
{
	"_id" : "store_db.store_db_user",
	"user" : "store_db_user",
	"db" : "store_db",
	"roles" : [
		{
			"role" : "dbOwner",
			"db" : "store_db"
		},
		{
			"role" : "readWrite",
			"db" : "store_db"
		}
	]
}

Apr 27, 2016

Running your WordPress blog on WSO2 App Cloud

WSO2 App Cloud is now supporting Docker base PHP applications. In this blog post I will describe how to install a WordPress blog in this environment. In order to setup a WordPress environment we need to have two things;

  1. Web server with PHP support
  2. MySQL database

If we have both of these we can start setting up WordPress. In WSO2 App Cloud we can use the PHP application as the WordPress hosting web server which is a PHP enabled Apache web server docker image. Also it provides a database service where you can easily create and manage MySQL databases via the App Cloud user interface (UI).

Note:- 
For the moment WSO2 App Cloud is on beta therefore these docker images will have only 12h of lifetime with no data persistence in the file storage level. Data on MySQL databases will be safe unless you override. If you need more help don't hesitate to contact Cloud support.

Creating PHP application

Signup or signin to WSO2 App Cloud via http://wso2.com/cloud. Then click on the "App Cloud beta" section.
Then it will redirect you to the App Cloud user interface. Click on 'Add New Application' button on the left hand corner.
This will prompt you to several available applications. Select 'PHP Web Application' box and continue.
Then it will prompt you a wizard. In that give a proper name and a version to your application. Name and version will be use to generate the domain name for your application.

There are several options that you can use to upload PHP content to this application. For the moment I will download the wordpress-X.X.X.zip file from the wordpress site and upload it to application.
In the below sections of the UI you can set the run time and container specification. Give the highest Apache version as the runtime and use minimal container speck as wordpress does not require much processing and memory.
If the things are all set and the file upload is complete click on 'Create' button. You will get the following status pop-up when you click the create button and it will redirect you to the application when its complete.
In the application UI note the URL. Now you can click on the 'Launch App' button so that it will redirect you to your PHP application.
Newly installed WordPress site will be like this.
Now we need to provide database details to it. Therefore, we need to create database and a user.

Creating database

Go back to the Application UI and click on 'Back to listing' button.
In that UI you can see a button in the top left hand corner called 'Create database'. Click on that.
In the create database UI give a database name, database user name and a password . Password need to pass the password policy so you can click on 'Generate password' to generate a secure password easily. By the way of you use generate password option make sure you copy the generated password before you proceed with database creation. Otherwise you may need to reset the password.

Also note that database name and database user name will append tenant domain and random string accordingly to the end of both. Therefore, those fields will only get few number of input characters.
If all set then click on 'Create database' button to proceed. After successfully creating the database it will redirect you to a database management user interface like following.
Now you can use those details to login to the newly create mysql database as follows;
$ mysql -h mysql.storage.cloud.wso2.com -p'' -u
eg :-
$ mysql -h mysql.storage.cloud.wso2.com -p'XXXXXXXXX' -u admin_LeWvxS3l wpdb_thilina 
Configuring WordPress

If the database creation is successful and you can login to it without any issue we can use those details to configure WordPress.

Go back to the WordPress UI and click on 'let's go' button. It will prompt to a database configuration wizard. Fill those fields with the details that we got from the previous section.
If WordPress application can successfully establish a connection with the database using your inputs it will prompt you to a UI as follows.
On that click on 'Run the install'. Then WordPress will start populating database tables and insert initial data to the given database.

When its complete it will ask for some basic configurations like the site title, admin user name and passwords.
Click on 'Install WordPress' after filling those information. Then it will redirect you to the WordPress admin console login page. Login to that using the username and password gave in the previous section.
So now WordPress is ready to use. But the existing URL is not very attractive. If you have a domain you can use it as the base URL of this application.

Setting custom domain (Optional)

IN the application UI click on the top left three lines button shown in the following image.
It will show some advance configuration that we can use. In that list select the last one 'Custom URL' option.
It will prompt you following user interface. Enter the domain name that you are willing to use.
But before you validate make sure you add a DNS CNAME to that domain pointing to you application launch URL.

Following is the wizard that I got when adding the CNAME via Godaddy. This user interface and adding CNAME options will be different for you DNS provider.
You can validate the CNAME by running 'dig' command in Linux or nslookup in windows.
If the CNAME is working click on 'Update'.
 If that is successful you will get the above notification and if you access that domain name it will show your newly created WordPress blog.