Kubernetes : Deploy a production ready cluster with Ansible.

7 min readMay 25, 2020

Kubesray is a group of Ansible playbooks for deployment a Kubernetes cluster in most metal and most clouds.

Official site: https://kubespray.io/

As described by project git repository :

  • Can be deployed on AWS, GCE, Azure, OpenStack, vSphere, Packet (bare metal), Oracle Cloud Infrastructure (Experimental), or Baremetal
  • Highly available cluster
  • Composable (Choice of the network plugin for instance)
  • Supports most popular Linux distributions
  • Continuous integration tests

My intention for this lab is show a deployment on premisses environment, so if you found this post directly without had a Kubernetes background maybe this overview can help you.

Hardware requirements

For Kubernetes production ready deployments is require :

These specifications apply the simplest possible cluster setup :

  • 4 GB or more of RAM per machine (any less will leave little room for your apps)
  • 2 CPUs or more (less than this will cause a deployment error)

It’s important to had these specification on hardware because it will check during the Ansible playbooks execution.

For large deployments consider read this link and study the necessity .


In my Kubespray deployment I will install :

  • 3 Masters
  • 3 Workers
  • 3 Etcd Hosts
  • External Load balancer with HAProxy for Kubernetes API

As described by official Kubernetes document that’s necessary provision servers with the following requirements. The follow steps will configure step by step during this deployment.

Ansible host installation

Before start the requirements create the ssh keypair

ssh-keygen -t rsaGenerating public/private rsa key pair.
Enter file in which to save the key (/home/demo/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/demo/.ssh/id_rsa.
Your public key has been saved in /home/demo/.ssh/id_rsa.pub.
The key fingerprint is:
4a:dd:0a:c6:35:4e:3f:ed:27:38:8c:74:44:4d:93:67 demo@a
The key's randomart image is:
+--[ RSA 2048]----+
| .oo. |
| . o.E |
| + . o |
| . = = . |
| = S = . |
| o + = + |
| . o + o . |
| . o |
| |

Just copy the key for all hosts

ssh-copy-id root@host

Install this packages

yum install -y python3-pip python-netaddr

Clone the kube spray repository

cd /opt/
git clone https://github.com/kubernetes-sigs/kubespray
cd kubespray

Install the requirements with pip

sudo pip3 install -r requirements.txt

A sample inventory could be found in inventory/sample/

I created my kubespray inventory from inventory/sample/inventory.ini file

Change all information for your necessity :

  • ansible_host : configure for public ip
  • ip : variable for bind kubernetes services
  • etcd_member_name : you should set etcd_member_name for etcd cluster.

An inventory template .

etcd1 ansible_host=publicip ip=privateip etcd_member_name=etcd1
etcd2 ansible_host=publicip ip=privateip etcd_member_name=etcd2
etcd3 ansible_host=publicip ip=privateip etcd_member_name=etcd3
master1 ansible_host=publicip ip=privateip
master2 ansible_host=publicip ip=privateip
master3 ansible_host=publicip ip=privateip
worker1 ansible_host=publicip ip=privateip
worker2 ansible_host=publicip ip=privateip
worker3 ansible_host=publicip ip=privateip

If you don’t have 2 ips for public and private configurations use this example:

host ansible_host= ip=


For a manual installation this document can help you.

I install the haproxy with this ansible playbook installation.

Create a directory called /opt/playbooks

mkdir /opt/playbooks

Create the /opt/playbooks/haproxy.yml file

- hosts: loadbalancer,masters
gather_facts: yes
become: true
- name: Install Haproxy packages
name: haproxy
state: present
when: "'loadbalancer' in group_names"
- name: Haproxy conf template
src: ./haproxy.conf.j2
dest: /etc/haproxy/haproxy.cfg
mode: 0644
when: "'loadbalancer' in group_names"
- name: Semanage allows http 6443 port
ports: "{{ item }}"
proto: tcp
setype: http_port_t
state: present
when: "'loadbalancer' in group_names"
- 6443
- 9000
- name: Start Haproxy
name: haproxy
enabled: yes
state: restarted
when: "'loadbalancer' in group_names"

Create the /opt/playbooks/haproxy.cfg.j2 file

log local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
stats socket /var/lib/haproxy/stats
log global
option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen stats :9000
stats enable
stats realm Haproxy\ Statistics
stats uri /haproxy_stats
stats auth admin:password
stats refresh 30
mode http
frontend main *:{{ balancer_listen_port|default('6443') }}
default_backend {{ balancer_name | default('mgmt6443') }}
option tcplog
backend {{ balancer_name | default('mgmt6443') }}
balance source
mode tcp
# MASTERS 6443
{% for host in groups.masters %}
server {{ host }} {{ hostvars[host].ansible_default_ipv4.address }}:6443 check
{% endfor %}

Setup the inventory for this playbook

vi /etc/ansible/hosts[loadbalancer]

Execute the playbook

ansible-playbook haproxy.yml

Access the Haproxy web console

http://<LOAD BALANCER NAME>:9000/haproxy_stats

Kube Spray setup

Go to kubespray directory

cd /opt/kubespray

Check all addons options and uncomment that.

  • dashboard_enabled : Consider to disable the Kubernetes default dashboard for security reasons and use a reliable configuration after this deployment.
  • local_volume_provisioner_enabled and cert_manager_enabled : Enable the local volume provisioner so persistent volumes can be used and the cert manager to later be able to automatically provision SSL certs using Let’s Encrypt.
vi inventory/mycluster/group_vars/k8s-cluster/addons.yml...
dashboard_enabled: false
local_volume_provisioner_enabled: true
cert_manager_enabled: true

Check all deployment options like what’s sdn is used for deployment.

  • kube_network_plugin : Choose the Sdn plugin, normally I use calico.
  • kubeconfig_localhost: Made kubespray generate a kubeconfig file on the computer used to run Kubespray.
vi inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml...
kube_network_plugin: calico
kubeconfig_localhost: true

Checking the external haproxy setup and upstream dns server options.

vi inventory/mycluster/group_vars/all/all.yml## External LB example config
apiserver_loadbalancer_domain_name: "loadbalancer.local.lab"
port: 6443
#Upstream dns servers

Run this command to installing the cluster by ansible

ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml

I had a problem during my deployment and I find an issue was opened for this error:

Docker client version 1.40 is too new. Maximum supported API version is 1.39

To handle with this situation I downgraded the docker ce cli package :

yum downgrade docker-ce-cli-19.03.7-3.el7.x86_64  -y

After my downgrade of docker ce cli package in master and worker nodes re-run the playbook again :

ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml -b -v --private-key=~/.ssh/id_rsa -K

The ouput from a sucessful deployment :

Access the HAProxy web console and you watch the traffic over frontend for the api server :

If did you have any trouble with a deployment that can be remove at this moment without any application in production.

ansible-playbook -i inventory/mycluster/inventory.ini reset.yml

Deployment tests

Install the Kubernetes client

yum install -y kubernetes-client

Then export a KUBECONFIG environment variable pointing to the admin.conf file that was generated by Kubespray and use kubectl to verify your cluster nodes are up and running:

export KUBECONFIG=/opt/kubespray/inventory/mycluster/artifacts/admin.conf

Cluster information

[root@host  ~]# kubectl cluster-info
Kubernetes master is running at https://loadbalancer.local.lab:6443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.p

Check the cluster configuration :

[root@host  ~]# kubectl config view
apiVersion: v1
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://loadbalancer.local.lab:6443
name: cluster.local
- context:
cluster: cluster.local
user: kubernetes-admin
name: kubernetes-admin@cluster.local
current-context: kubernetes-admin@cluster.local
kind: Config
preferences: {}
- name: kubernetes-admin
client-certificate-data: REDACTED
client-key-data: REDACTED

Launch a POD and a service to conclude the tests.


kubectl apply -f https://k8s.io/examples/application/deployment.yaml


kubectl expose deployment/nginx-deployment --type="NodePort" --port 80

Testing with curl from internal ip

[root@master1 ~]# kubectl get service nginx-deployment
nginx-deployment NodePort <none> 80:31660/TCP 4m7s
[root@master1 ~]# curl
<!DOCTYPE html>
<title>Welcome to nginx!</title>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>

Testing an external request for the master ip with pod port exposed


My test:

Launching the Dashboard

If you were forgot to disabled it during the deployment just run this command to delete

kubectl delete deployments kubernetes-dashboard -n kube-system

Deploy the dashbord

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc2/aio/deploy/recommended.yaml

In the default namespace, create a service account.

kubectl create serviceaccount fajlinux -n default

Create cluster binding rules for the newly created service account.

kubectl create clusterrolebinding fajlinux-admin -n default --cluster-role=cluster-admin --serviceaccount=default:fajlinux

Run the kubectl command below to generate the token

kubectl get secret $(kubectl get serviceaccount fajlinux -o jsonpath="{.secrets[0].name}") -o jsonpath="{.data.token}" | base64 --decode

Get the token generated by output from above command and login with Token option.

In my next post I would to post about a load balancer solution in baremetal and non cloud environments .