The following diagram depicts the deployment architecture highlevel components of the message flow of the Custos. The Custos services are deployed on a three5-node K8 cluster and exposed to the external traffic via a K8 Ingress-Controller. The Ingress-Controller is fronted by Nginx Reverse Proxy and the whole cluster is protected through a Firewall.
The following sections describe the fresh deployment of the Custos services on a K8 Cluster.
Set up K8 Cluster
Prerequisites
- Three ubuntu VMs
- Setup SSH Keys on the local machine and remote ubuntu VMs [1]
- Setup Ansible on local machine [2]
- Basic understanding of Docker and its concepts.
Step 1
We need to create an Ansible playbook to execute the K8 deployment setup. Hence, create a local working directory and create a hosts file that contains IPs of Master and Worker nodes of the K8 Cluster.
...
Next, configure access privileges for non-root users. To this end, create a file with the following content.
...
K8 Cluster Deployment Architecture
Custos K8 cluster is a rancher bootstrapped cluster. And has following configurations.
- Two master nodes
- Four worker nodes (including master which has two worker nodes)
Data Replicas
Custos has three types of databases
- MySQl database for Custos microsevices
- PostgreSQL database for Keycloak services.
- Harshicorp database for Consul services.
- External Kafka service for event based message delivering.
Each database is mounted to volumes as shown in the above figure and has master and secondary deployments to make them highly available. In addition, underlying docker services are taken away from root data structure and mounted to a separate volume.
External Data Backups (Pending)
Although, we have a replica of data volumes internally, we take backups of K8 cluster and data volumes to external location to bootstrap new cluster from a backup in any case if underlying infrastructure is not available.We use
velero to implement automatic backups along with restic plugin.
The following sections describe the fresh deployment of the Custos services on a K8 Cluster.
Set up K8 Cluster
Prerequisites
- Seven ubuntu VMs
- Setup SSH Keys on the local machine and remote ubuntu VMs [1]
- Basic understanding of Docker, Kubernetes and its concepts.
Step 1 : Installing Rancher
- Install Rancher 2.5.5 or above on one of the VMs.
Step 2 : Bootstrap K8 Cluster on Bare Metals Servers With Rancher
Step 3
This will create non-root user "ubuntu" on remote servers and configure to access them using SSH public keys. Next, execute the above script using
ansible-playbook -i hosts config.yml
Step 3
Next, install K8 dependencies using the following Ansible scripts. This installs Docker, Transports, Kubelet, Kubeadm, and Kubectl on Ubuntu VMs.
...
Step 4
Next, we can set-up the master node using the following script.
...