CIS and Kubernetes - Part 1: Install Kubernetes and Calico

Welcome to this series to see how to:

  • Install Kubernetes and Calico (Part 1)
  • Deploy F5 Container Ingress Services (F5 CIS) to tie applications lifecycle to our application services (Part 2)


Here is the setup of our lab environment:


 


BIG-IP Version: 15.0.1

Kubernetes component: Ubuntu 18.04 LTM



We consider that your BIG-IPs are already setup and running:

  • Licensed and setup as a cluster
  • The networking setup is already done



Part 1: Install Kubernetes and Calico


Setup our systems before installing kubernetes


Step1: Update our systems and install docker


To run containers in Pods, Kubernetes uses a container runtime. We will use docker and follow the recommendation provided here


As root on ALL Kubernetes components (Master and Node):


# Install packages to allow apt to use a repository over HTTPS
apt-get -y update && apt-get install -y apt-transport-https ca-certificates curl software-properties-common

# Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

# Add Docker apt repository.
add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"

# Install Docker CE.
apt-get -y update && apt-get install -y docker-ce=18.06.2~ce~3-0~ubuntu

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker


We may do a quick test to ensure docker run as expected:


docker run hello-world


Step2: Setup Kubernetes tools (kubeadm, kubelet and kubectl)


To setup Kubernetes, we will leverage the following tools:

kubeadm: the command to bootstrap the cluster.

kubelet: the component that runs on all of the machines in your cluster and does things like starting pods and containers.

kubectl: the command line util to talk to your cluster.

As root on ALL Kubernetes components (Master and Node):



curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get -y update


We can review which version of kubernetes is supported with F5 Container Ingress Services here

At the time of this article, the latest supported version is v1.13.4. We'll make sure to install this specific version with our following step



apt-get install -qy kubelet=1.13.4-00 kubeadm=1.13.4-00 kubectl=1.13.4-00 kubernetes-cni=0.6.0-00
apt-mark hold kubelet kubeadm kubectl


Install Kubernetes


Step1: Setup Kubernetes with kubeadm


We will follow the steps provided in the documentation here

As root on the MASTER node (make sure to update the api server address to reflect your master node IP):


kubeadm init --apiserver-advertise-address=10.1.20.20 --pod-network-cidr=192.168.0.0/16



Note: SAVE somewhere the kubeadm join command. It is needed to "assimilate" the node later. In my example, it looks like the following (YOURS WILL BE DIFFERENT):


kubeadm join 10.1.20.20:6443 --token rlbc20.va65z7eauz89mmuv --discovery-token-ca-cert-hash sha256:42eca5bf49c645ff143f972f6bc88a59468a30276f907bf40da3bcf5127c0375


Now you should NOT be ROOT anymore. Go back to your non root user. Since i use Ubuntu, i'll use the default "ubuntu" user


Run the following commands as highlighted in the screenshot above:


mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config


Step2: Install the networking component of Kubernetes


The last step is to setup the network related to our k8s infrastructure. In our kubeadm init command, we used --pod-network-cidr=192.168.0.0/16 in order to be able to setup next on network leveraging Calico as documented here


kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml


You may monitor the deployment by running the command:


kubectl get pods --all-namespaces



After some time (<1 min), everything should have a "Running" status.

Make sure that CoreDNS started also properly. If everything is up and running, we have our master setup properly and can go to the node to setup k8s on it. 



Step3: Add the Node to our Kubernetes Cluster


Now that the master is setup properly, we can assimilate the node. You need to retrieve the "kubeadmin join …" command that you received at the end of the "kubeadm init …" cmd. You must run the following command as ROOT on the Kubernetes NODE (remember that you got a different hash and token, the command below is an example): 


kubeadm join 10.1.20.20:6443 --token rlbc20.va65z7eauz89mmuv --discovery-token-ca-cert-hash sha256:42eca5bf49c645ff143f972f6bc88a59468a30276f907bf40da3bcf5127c0375



We can check the status of our node by running the following command on our MASTER (ubuntu user)


kubectl get nodes


Both component should have a "Ready" status.

Last step is to setup Calico between our BIG-IPs and our Kubernetes cluster

 

Setup Calico


We need to setup Calico on our BIG-IPs and k8S components. We will setup our environment with the following AS Number: 64512


Step1: BIG-IPs Calico setup


F5 has documented this procedure here


We will use our self IPs on the internal network. Therefore we need to make sure of the following:

  • The self IP has a portlock down setup to "Allow All"
  • Or add a TCP custom port to the self IP: TCP port 179


You need to allow BGP on the default route domain 0 on your BIG-IPs. Connect to the BIG-IP GUI on go into Network > Route domain. Click on Route Domain "0" and allow BGP



Click on "Update"


Once this is done, connect via SSH and get into a bash shell on both BIG-IPs

Run the following commands: 


#access the IMI Shell
imish

#Switch to enable mode
enable

#Enter configuration mode
config terminal
#Setup route bgp with AS Number 64512
router bgp 64512

#Create BGP Peer group
neighbor calico-k8s peer-group

#assign peer group as BGP neighbors
neighbor calico-k8s remote-as 64512

#we need to add all the peers: the other BIG-IP, our k8s components
neighbor 10.1.20.20 peer-group calico-k8s
neighbor 10.1.20.21 peer-group calico-k8s

#on BIG-IP1, run
neighbor 10.1.20.12 peer-group calico-k8s

#on BIG-IP2, run
neighbor  10.1.20.11 peer-group calico-k8s

#save configuration
write

#exit
end



You can review your setup with the command


show ip bgp neighbors


Note: your other BIG-IP should be identified with a router ID and have a BGP state of "Active". The k8s node won't have a router ID since BGP hasn't already been setup on those nodes. 



Keep your BIG-IP SSH sessions open. We'll re-use the imish terminal once our k8s components have Calico setup


Step2: Kubernetes Calico setup


On the MASTER node (not as root), we need to retrieve the calicoctl binary


curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.10.0/calicoctl
chmod +x calicoctl
sudo mv calicoctl /usr/local/bin


We need to setup calicoctl as explained here


sudo mkdir /etc/calico


Create a file /etc/calico/calicoctl.cfg with your preferred editor (you'll need sudo privilegies). This file should contain the following


apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
  datastoreType: "kubernetes"
  kubeconfig: "/home/ubuntu/config"


Note: you may have to change the path specified by the kubeconfig parameter based on the user you use to do kubectl command


To make sure that calicoctl is properly setup, run the command


calicoctl get nodes


You should get a list of your Kubernetes nodes



Now we can work on our Calico/BGP configuration as documented here


On the MASTER node:


cat << EOF | calicoctl create -f -
 apiVersion: projectcalico.org/v3
 kind: BGPConfiguration
 metadata:
   name: default
 spec:
   logSeverityScreen: Info
   nodeToNodeMeshEnabled: true
   asNumber: 64512
EOF


Note: Because we setup nodeToNodeMeshEnabled to True, the k8s node will receive the same config


We may now setup our BIG-IP BGP peers. Replace the peerIP Value with the IP of your BIG-IPs


cat << EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-global-bigip1
spec:
  peerIP: 10.1.20.11
  asNumber: 64512
EOF


cat << EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-global-bigip2
spec:
  peerIP: 10.1.20.12
  asNumber: 64512
EOF


Review your setup with the command: 


calicoctl get bgpPeer


If you go back to your BIG-IP SSH connections, you may check that your Kubernetes nodes have a router ID now in your BGP configuration:

 


imish
show ip bgp neighbors


Summary


So far we have:

  • Setup Kubernetes
  • Setup Calico between our BIG-IPs and our Kubernetes cluster


In the next article, we will setup F5 container Ingress Services (F5 CIS)

Published Nov 04, 2019
Version 1.0

Was this article helpful?

1 Comment

  • Nice article,  and when Create a file /etc/calico/calicoctl.cfg

      kubeconfig: "/home/ubuntu/config"
    

    should modify to

      kubeconfig: "/home/ubuntu/.kube/config"