The following diagram depicts the deployment architecture of the Custos. The Custos services are deployed on a three-node K8 cluster and exposed to the external traffic via a K8 Ingress-Controller. The Ingress controller is fronted by Nginx reverse proxy and the whole cluster is protected through a firewall.

The following sections describe the fresh deployment of the Custos services in K8 Cluster.

Set up K8 Cluster

Prerequisites

Three ubuntu VMs
Setup SSH Keys on the local machine and remote ubuntu VMs [1]
Setup Ansible on local machine [2]
Basic understanding of Docker and its concepts.

Step 1

We need to create an Ansible playbook to execute the K8 deployment setup. Hence, create a local working directory and create a hosts file that contains IPs of Master and Worker nodes of the K8 cluster.

[masters]
custos-kube-master ansible_host=IP of Master ansible_user=root

[workers]
custos-kube-worker1 ansible_host=IP of Worker ansible_user=root
custos-kube-worker2 ansible_host=IP of Worker ansible_user=root

[all:vars]
ansible_python_interpreter=/usr/bin/python3

Step 2

Next, configure access privileges for non-root users. To this end, create a file with the following content.

- hosts: all
become: yes
tasks:
- name: create the 'ubuntu' user
user: name=ubuntu append=yes state=present createhome=yes shell=/bin/bash

- name: allow 'ubuntu' to have passwordless sudo
lineinfile:
dest: /etc/sudoers
line: 'ubuntu ALL=(ALL) NOPASSWD: ALL'
validate: 'visudo -cf %s'

- name: set up authorized keys for the ubuntu user
authorized_key: user=ubuntu key="{{item}}"
with_file:
- ~/.ssh/id_rsa.pub

This will create non-root user "ubuntu" on remote servers and configure to access them using SSH public keys. Next, execute the above script using

ansible-playbook -i hosts config.yml

Step 3

Next, install K8 dependencies using the following Ansible scripts. This installs Docker, Transports, Kubelet, Kubeadm and Kubectl on Ubuntu machines.

- hosts: all
become: yes
tasks:
- name: install Docker
apt:
name: docker.io
state: present
update_cache: true

- name: install APT Transport HTTPS
apt:
name: apt-transport-https
state: present

- name: add Kubernetes apt-key
apt_key:
url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
state: present

- name: add Kubernetes' APT repository
apt_repository:
repo: deb http://apt.kubernetes.io/ kubernetes-xenial main
state: present
filename: 'kubernetes'

- name: install kubelet
apt:
name: kubelet=1.14.0-00
state: present
update_cache: true

- name: install kubeadm
apt:
name: kubeadm=1.14.0-00
state: present

- hosts: master
become: yes
tasks:
- name: install kubectl
apt:
name: kubectl=1.14.0-00
state: present
force: yes

Step 4

Next, we can set-up the master node using the following script.

- hosts: masters
become: yes
tasks:
- name: initialize the cluster
shell: kubeadm init --pod-network-cidr=10.244.0.0/16 >> cluster_initialized.txt
args:
chdir: $HOME
creates: cluster_initialized.txt

- name: create .kube directory
become: yes
become_user: ubuntu
file:
path: $HOME/.kube
state: directory
mode: 0755

- name: copy admin.conf to user's kube config
copy:
src: /etc/kubernetes/admin.conf
dest: /home/ubuntu/.kube/config
remote_src: yes
owner: ubuntu

- name: install Pod network
become: yes
become_user: ubuntu
shell: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml >> pod_network_setup.txt
args:
chdir: $HOME
creates: pod_network_setup.txt

Step 5

Next, we can set-up the worker nodes using the following script

- hosts: masters
become: yes
gather_facts: false
tasks:
- name: get join command
shell: kubeadm token create --print-join-command
register: join_command_raw

- name: set join command
set_fact:
join_command: "{{ join_command_raw.stdout_lines[0] }}"

- hosts: workers
become: yes
tasks:
- name: join cluster
shell: "{{ hostvars['custos-kube-master'].join_command }} >> node_joined.txt"
args:
chdir: $HOME
creates: node_joined.txt

Successful execution of the above scripts will create the K8 Cluster.

Install Helm

We need to install Helm to deploy the Custos services on the K8 Cluster. Currently, we are using version 2[3]. Helm V2 requires Tiller service to be deployed on K8 Cluster to work, but, V3 directly accesses the Cluster.

Install Dependencies and Third Party Services

The following services and components are required by the Custos services and K8 Cluster as dependencies.

Install Nginx Ingress Controller

The Nginx ingress controller is the entry point of the K8 Cluster. The open-source Nginx controller distributes traffic among its own namespace services and is restricted to its own space. Hence, we require separate ingress controllers for Custos, Linkerd, Keycloak, and, Vault namespaces. We have used the helm chart[4] to deploy Ingress-Controllers. By default, the Ingress controller requires the auto-provisioning of a load balancer and deployments that do not support load-balancers should use NodePort service type or proxy to bridge the external environment and the cluster.

Install MySQL Database

MySQL database is required by the Custos services to store data. We have used bitnami/MySQL DB[5]. This will create one Master Pod and two Slave pods. All services are connected to the Master pod for reading and writing data. Slave pods are used as replicators. Moreover, PVs should be created for data persistence. The following configuration creates PV in the K8 cluster, and before creating the PVs storage mount points should be created in each node.

PV configuration

apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql
labels:
type: local
spec:
storageClassName: ""
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/bitnami/mysql"

Install Cert-Manager

The Cert-Manager component is used for the auto renewal of Let’s Encrypt certificates[6]. This will install Cert-Manager into the K8 Cluster. Nextly, Certificate Issuer is needed to be configured. For that, we have used the following configuration.

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-production
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
Email: abc@gmail.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource that will be used to store the account's private key.
name: acme-prod-private-key
# Add a single challenge solver, HTTP01 using nginx
solvers:
- http01:
ingress:
class: nginx

According to the above configuration, Let’ Encrypt will use the HTTP challenge. So port 80 should be opened and traffic should be directed to K8 Cluster. If NodePort is used, ensure that traffic is reachable from outside through the standard ports. (configure reverse proxy)

Install Logging Stack

We have used Flutend, ElasticSearch, and Kibana to pull logs from all nodes. Flutend pulls all logs from nodes and stores them in ES storage. Kibana is pointed to ES as a dashboard and ES fetches all logs collected by Flutend. This logging stack is installed in kube-logging namespace.

Install Linkerd Service Mesh

We are using Linkerd Service Mesh[8] to enable SSL for internal inter-service communication and as a dashboard for services. The Following configuration will expose the Linkerd dashboard to outside environment.

apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: web-ingress-auth
namespace: linkerd
data:
auth: YWRtaW46JGFwcjEkbjdDdTZnSGwkRTQ3b2dmN0NPOE5SWWpFakJPa1dNLgoK
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: web-ingress
namespace: linkerd
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/upstream-vhost: linkerd-web.linkerd.svc.cluster.local:8084
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Origin "";
proxy_hide_header l5d-remote-ip;
proxy_hide_header l5d-server-id;
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: web-ingress-auth
nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
cert-manager.io/cluster-issuer: letsencrypt-production
spec:
rules:
- host: custos.scigap.org
http:
paths:
- backend:
serviceName: linkerd-web
servicePort: 8084
tls:
- hosts:
- custos.scigap.org
secretName: tls-secret

Install Keycloak

The Keycloak[9] is used as our main IDP implementation. The following diagram depicts the architecture of the Keycloak and the Keycloak pods are connected to PostgreSQL Database for persistence.

Install Vault

The HashiCorp Vault is used as our secret storage. It ensures the security of secrets. The following diagram depicts the architecture of the Vault deployment.

We have used [11,12,13,14] for deployment instructions.

Install Custos Services

The Custos source code is hosted openly at github[16]. Active developments are going on the development branch and the master branch contains the latest released source code.

Prerequisites

Install Java11 and Maven 3.6.X

Checkout source code

git clone https://github.com/apache/airavata-custos.git

Build Source code

mvn clean install

This will build the source code and create docker images, helm charts to be deployed in K8 cluster and Java artifacts. Helm charts are created at the "target/helm" path. To publish docker images to the docker repository the use following command.

mvn dockerfile:push

Use the below commands to install the Custos service using helm charts.

helm install --name service_name --namespace custos chart_name

To upgrade an existing service

helm upgrade --name service_name --namespace custos chart_name

Troubleshooting

The main areas that you might need to troubleshoot are the Custos services and Databases. Troubleshooting Custos services are very easy, you can check the logs fom Kibana server related to the Custos namespace, or you can directly log into the Pod relevant to the service and check for console logs.

Troubleshooting Databases

First, check logs from the Kibana dashboard, or login into Master and Slave nodes and check for console logs. Most probably it should print errors.

Steps to replace database or migrate a database

First login to Slave node and stop Slave replication thread using

mysql > STOP SLAVE;

Make a backup of existing databases located at DB mount point (e.g /bitnami/mysql)
Remove old data.
Install a new database
Create databases required by the Custos services
Reinstall Custos core services, this will create relevant tables on databases.
Log in to the master pod, that wants to import previous data and follow these steps

mysql > ALTER TABLE table_name DISCARD TABLESPACE;