Nov 11, 2017

Run a Mongo cluster with authentication on kubernetes using StatefulSets.

With StatefulSets running a mongo cluster with persistent storage is easy. Kubernetes blog post in [1] explains how to do that. But when we try to enable authentication there is a small problem with the above solution. Problem is we need to start the cluster without replica set (--replSet) option and add the admin user and a key file, then restart with the replica set and the key file.

In order to do that we need to modify the above sample code to add a pod life cycle - post start command.

    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo:3.4.9
          command:
          - /bin/sh
          - -c
          - >
            if [ -f /data/db/admin-user.lock ]; then
              mongod --replSet rs0 --clusterAuthMode keyFile --keyFile /etc/secrets/mongo.key --setParameter authenticationMechanisms=SCRAM-SHA-1;
            else
              mongod --auth;
            fi;
          lifecycle:
            postStart:
              exec:
                command:
                - /bin/sh
                - -c
                - >
                  if [ ! -f /data/db/admin-user.lock ]; then
                    sleep 5;
                    touch /data/db/admin-user.lock
                    if [ "$HOSTNAME" = "mongo-0" ]; then
                      mongo --eval 'db = db.getSiblingDB("admin"); db.createUser({ user: "admin", pwd: "password", roles: [{ role: "root", db: "admin" }]});';
                    fi;
                    mongod --shutdown;
                  fi;
          ports:
            - containerPort: 27017
If you look at the command section of the above statefulSet part you can see I'm checking for a lock file. If that file exists I will start the cluster with replica set option. Else I will start the cluster with just auth.

Then you can notice the postStart section. In post start section we can define what to do at the start of a container in a pod. You can refer to this post in [2] to get an idea of pod lifecycle. In that it will again check for the lock file. If the file does not exists it will create the file and if the hostname is equal to mongo-o it will add the mongo admin user to the cluster. Then it will stop the node. Stopping the node will result the container to restart. Since we are using persistent storage the initially created lock file will exist. Since the lock file is there command will start the cluster with replica set option and postStart will pass its loop.

Running the mongo cluster

Generate a random key to enable keyFile authentication replication as described in this post.

openssl rand -base64 741 > mongodb-keyfile
Create a secret using that random string.
kubectl create secret generic mongo-key --from-file=mongodb-keyfile
Create statefulSet and headless service.
kubectl create -f  https://gist.githubusercontent.com/thilinapiy/0c5abc2c0c28efe1bbe2165b0d8dc115/raw/d3d0e64dfd35158907d076422c362f289d124dfc/mongo-statefulset.yaml
Scale-up if you need mote high availability.
kubectl scale --replicas=3 statefulset mongo
Let me know if you need further help.

1. http://blog.kubernetes.io/2017/01/running-mongodb-on-kubernetes-with-statefulsets.html
2. https://blog.openshift.com/kubernetes-pods-life
3. https://docs.mongodb.com/manual/tutorial/enforce-keyfile-access-control-in-existing-replica-set/

Oct 14, 2017

Watch Kubernetes pod events stop/start


Run kube proxy to use curl without authentication

kubectl proxy

Run the curl to watch the events and filter it with jq for pod starts and stops.

curl -s 127.0.0.1:8001/api/v1/watch/events | jq --raw-output \ 'if .object.reason == "Started" then . elif .object.reason == "Killing" then . else empty end | [.object.firstTimestamp, .object.reason, .object.metadata.namespace, .object.metadata.name] | @csv'

Output will be

"2017-10-14T02:51:31Z","Killing","default","echo-1788535470-v3089.14ed5012be9b81d2" "2017-10-14T02:17:12Z","Started","default","hello-minikube-180744149-7sbj2.14ed4e335345db3d" "2017-10-14T02:17:15Z","Started","kube-system","default-http-backend-27b99.14ed4e341737f843" "2017-10-14T02:17:11Z","Started","kube-system","kube-addon-manager-minikube.14ed4e33195f434a" "2017-10-14T02:17:14Z","Started","kube-system","kube-dns-910330662-78fqv.14ed4e33d9790ee6" "2017-10-14T02:17:15Z","Started","kube-system","kube-dns-910330662-78fqv.14ed4e33e88e1b68" "2017-10-14T02:17:15Z","Started","kube-system","kube-dns-910330662-78fqv.14ed4e3404bfcffc" "2017-10-14T02:17:13Z","Started","kube-system","kube-state-metrics-3741290554-5cv05.14ed4e337556350c" "2017-10-14T02:17:13Z","Started","kube-system","kube-state-metrics-3741290554-5cv05.14ed4e33804c8647" "2017-10-14T02:17:15Z","Started","kube-system","kubernetes-dashboard-8991s.14ed4e340386d190" "2017-10-14T02:54:57Z","Killing","kube-system","kubernetes-dashboard-8991s.14ed5042a8fa1c81" "2017-10-14T02:54:58Z","Started","kube-system","kubernetes-dashboard-xd7h5.14ed5042d33aa3c7" "2017-10-14T02:17:16Z","Started","kube-system","nginx-ingress-controller-9qn5l.14ed4e3426ecdaa8" "2017-10-14T02:55:23Z","Killing","kube-system","nginx-ingress-controller-9qn5l.14ed50489b820cce" "2017-10-14T02:55:37Z","Started","kube-system","nginx-ingress-controller-rf6j3.14ed504c01cf90ea" "2017-10-14T02:17:13Z","Started","kube-system","prometheus-3898748193-jgxzk.14ed4e339109bcb4" "2017-10-14T02:17:14Z","Started","kube-system","prometheus-3898748193-jgxzk.14ed4e33af0e9433"

Aug 29, 2017

Generate a SANs certificate

We are going to use openssl to generate a certificate with subject alternative names. When we use SANs in a certificate we can use the same certificate to front several websites with different domain names.

First we need to generate a private key. Since we are going to use this in a web server like Nginx or apache I'm not going to encrypt the private key with a passphrase.


openssl genrsa -out thilina.org.key 2048


Then we need to have a configurations file to add those alternative names into the certificate signing request (CSR).

sans.conf

[ req ]
default_bits       = 2048
distinguished_name = req_distinguished_name
req_extensions     = req_ext

[ req_distinguished_name ]
countryName         = Country Name (2 letter code)
stateOrProvinceName = State or Province Name (full name)
localityName        = Locality Name (eg, city)
organizationName    = Organization Name (eg, company)
commonName          = Common Name (e.g. server FQDN or YOUR name)

[ req_ext ]
subjectAltName = @alt_names

[alt_names]
DNS.1=thilina.org
DNS.2=api.thilina.org
DNS.3=gateway.thilina.org


Now I'm going to generate the CSR in a single command.


openssl req -new -key thilina.org.key -sha256 -nodes -out thilina.org.csr \
  -subj "/C=LK/ST=Colombo/L=Colombo/O=Thilina Piyasundara/OU=Home/CN=thilina.org" \
  -config san.conf


Print and verify the CSR


openssl req -in thilina.org.csr -text -noout



Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: C = LK, ST = Colombo, L = Colombo, O = Thilina Piyasundara, OU = Home, CN = thilina.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d0:13:91:5d:62:7c:4f:57:6d:4c:79:85:59:d8:
                    c5:ae:50:41:cc:db:fe:b4:75:fc:c1:73:e7:a7:ac:
                    89:36:3b:26:08:0f:33:b0:96:5c:29:a1:ee:9a:14:
                    13:4b:5b:43:74:74:a2:fd:97:2b:2b:bd:2a:b8:e6:
                    22:d2:01:15:f3:7f:e9:d8:c9:d4:65:04:5a:ef:f0:
                    03:41:63:56:39:eb:5f:e5:90:de:33:b7:bb:60:0e:
                    e3:70:79:60:8f:cb:a9:71:3b:e3:0a:b1:17:47:aa:
                    41:08:b5:44:5e:1a:a1:fa:a2:ce:ed:18:c5:a3:b0:
                    6f:0f:57:ca:ae:28:7f:91:49:14:6b:94:4c:3c:33:
                    fb:27:ed:77:37:a7:d6:54:4e:a7:6e:bc:c9:a2:a1:
                    b5:f2:f0:aa:76:64:04:83:96:92:03:36:4c:3e:14:
                    0e:97:a6:79:9e:23:c1:2a:c4:7a:3d:6e:f3:1c:40:
                    e3:d1:61:f2:56:51:8f:0f:04:76:62:ea:b0:1f:94:
                    e8:a8:8b:54:d6:08:5a:79:a6:a4:a0:00:fb:5f:c3:
                    d5:d4:50:ea:15:12:ea:9b:10:cc:9a:d9:32:6e:48:
                    93:30:4b:e7:2e:fe:a9:a0:31:16:61:24:3f:29:54:
                    2a:25:da:d2:b3:6a:d9:d5:a9:51:ee:d3:bb:b9:83:
                    86:59
                Exponent: 65537 (0x10001)
        Attributes:
        Requested Extensions:
            X509v3 Subject Alternative Name:
                DNS:thilina.org, DNS:api.thilina.org, DNS:gateway.thilina.org
    Signature Algorithm: sha256WithRSAEncryption
         96:44:43:98:60:76:49:ad:8b:01:65:20:f1:ca:4a:47:84:67:
         dc:77:f0:2e:bb:30:68:8b:2f:79:c4:4c:10:91:ec:70:fe:73:
         9c:3e:f4:69:18:8c:34:f6:85:05:26:b1:2a:35:38:f5:93:59:
         c2:a4:07:83:73:79:88:9b:ff:17:99:66:34:58:21:bc:de:8e:
         65:b9:50:bb:18:52:53:9b:ed:a3:4e:c7:55:73:2e:42:47:dc:
         94:4d:fb:cc:ba:b1:7a:57:a6:f9:fa:27:a2:54:aa:cd:f6:79:
         3d:b7:0a:82:a3:18:41:ec:f5:db:cc:05:6a:43:64:d7:4a:00:
         fe:a3:89:f9:25:f3:79:55:f9:79:3a:b2:96:5e:9d:67:f5:c7:
         e4:ab:fc:da:cb:df:f5:76:36:44:fe:d2:87:3a:d7:a2:a9:2e:
         fc:7f:ba:a6:12:44:70:e0:c4:42:57:01:1e:51:0a:d4:2e:33:
         e2:63:20:c2:9a:07:1b:78:e8:fb:42:b5:e5:85:00:b1:2c:25:
         d8:ad:43:af:6a:01:09:59:7e:d0:af:dd:72:f3:93:18:30:38:
         c2:b0:6c:8e:88:79:4e:16:fe:e3:87:46:c2:eb:f3:2e:2b:aa:
         a7:a9:76:1d:fd:8b:d9:d9:1c:a3:1c:21:db:af:b0:0b:7e:15:
         37:37:0f:25



Validate the key, csr and certificates are matching.

openssl rsa -noout -modulus -in domain.key | openssl md5
openssl x509 -noout -modulus -in domain.crt | openssl md5
openssl req -noout -modulus -in domain.csr | openssl md5


Feb 7, 2017

Running your spring-boot app in Bitesize

First of all we have to have the spring-boot code in a git(svn) repo. I have create a sample spring-boot application using maven archetypes. You can find the code in;

https://github.com/thilinapiy/SpringBoot

Compile the code and generate the package using following command;
# cd SpringBoot
# mvn clean package
This will build the app and create a jar file called 'SpringBoot-1.0.0.jar'.

We can run the application with following command and it will start it in port 8080.
# java -jar target/SpringBoot-1.0.0.jar
Now we switch to the next part. In this we need to update the bitesize files according to our needs.

https://github.com/thilinapiy/bitesize-spring

First we'll update the 'build.bitesize' file. In this we need to update the projects and name accordingly and give the source code repo url and related details as in all other projects. But if you look at the shell commands you can see that I have modified few of those. I have add the 'mvn clean package' command and change the 'cp' command to copy the build jar to '/app' directory. Then it will build the deb as previous.
project: spring-dev
components:
  - name: spring-app
    os: linux
    repository:
      git: git@github.com:thilinapiy/SpringBoot.git
      branch: master
    build:
      - shell: sudo mkdir -p /app
      - shell: sudo mvn clean package
      - shell: sudo cp -rf target/*.jar /app
      - shell: sudo /usr/local/bin/fpm -s dir -n spring-app --iteration $(date "+%Y%m%d%H%M%S") -t deb /app
    artifacts:
      - location: "*.deb"
Then we'll check the 'application.bitesize' file. I have change the 'runtime' to an ububtu-jdk8. Then change the command to run the jar.
project: spring-dev
applications:
  - name: spring-app
    runtime: ubuntu-jdk8:1.0
    version: "0.1.0"
    dependencies:
      - name: spring-app 
        type: debian-package
        origin:
          build: spring-app
        version: 1.0
    command: "java -jar /app/SpringBoot-1.0.0.jar"
In the 'environments.bitesize' I have update the port to 8080.
project: spring-dev
environments:
  - name: production
    namespace: spring-dev
    deployment:
      method: rolling-upgrade
    services:
      - name: spring-app
        external_url: spring.dev-bite.io
        port: 8080 
        ssl: "false"
        replicas: 2
In the stackstorm create_ns option give the correct manspace and the repo-url.
Reference : http://docs.prsn.io//deployment-pipeline/readme.html

Feb 4, 2017

Granting dbadmin privileges to a user in MongoDB cluster

We need to grant 'dbadmin' privileges to a user called 'store_db_user' to their mondo database in a 4 node cluster.

First we need to connect to the primary database of the cluster with super.

# mongo -u supperuser -p password -h node1.mongo.local

If you connect to the primary replica it will change the shell prompt to something like this;

mongoreplicas:PRIMARY>

Then you can list down the databases using following command.

mongoreplicas:PRIMARY>show dbs
admin     0.078GB
local     2.077GB
store_db  0.078GB

Then switch to the relevant database;

mongoreplicas:PRIMARY>use store_db

And grant permissions;

mongoreplicas:PRIMARY>db.grantRolesToUser(
  "store_db_user",
  [
    { role: "dbOwner", db: "store_db" },
  ]
)

Exit from the admin user and login to the cluster as the database user.

# mongo -u store_db_user -p store_passwd -h node1.mongo.local/store_db

Validate the change.

mongoreplicas:PRIMARY>show users
{
	"_id" : "store_db.store_db_user",
	"user" : "store_db_user",
	"db" : "store_db",
	"roles" : [
		{
			"role" : "dbOwner",
			"db" : "store_db"
		},
		{
			"role" : "readWrite",
			"db" : "store_db"
		}
	]
}