How to run Kubernetes on Container Linux

, ,

This article will guide you setting up Container Linux (CoreOS) for Kubernetes.

Introduction

Kubernetes can run on almost any host supporting Docker, even on devices like the Raspberry PI. Personally I like Container Linux because it’s lightweight and practically maintenance free after setup. (compared to classic OS’es like Ubuntu or Debian). It’s an OS created specifically for container workloads and it has been a huge contributor to the succes of Kuberentes with techniques like “etcd”.

Hardware

Let’s run this cluster on at least 2 nodes, make sure they have at least 3G RAM each, both should have at least 20G disks and have basic network connectivity, make sure both nodes can connect to the internet. VMware is nice for this setup, it’s what I used in this example, but you can also run this on Digital Ocean or any other cloud service that offers Container Linux images.

The great thing about Kubernetes is that it doesn’t care about the hardware, it could be AWS combined with a Raspberry PI and VMware. In this article we focus on x64 architecture since Container Linux doesn’t run on ARM.

Starting Container Linux

There are a few ways to install Container Linux, but you’ll need to set a password at least to be able to login. I run Container Linux booted via iPXE with an ignition file. It allows me to configure basic features like the hostname and openssh public keys etc. I think you can paste this file on a webserver and refer to it using boot options when booting from ISO.

The ignition file sets up authentication, but also some other small settings recommended for Kubernetes.

{
  "ignition": { "version": "2.2.0" },
  "storage": {
    "files": [
        {
              "filesystem": "root",
              "path": "/etc/hostname",
              "mode": 420,
              "contents": {
                        "source": "data:,core1"
                }
        },
        {
        "filesystem": "root",
        "group": {},
        "path": "/etc/sysctl.d/local.conf",
        "user": {},
        "contents": {
          "source": "data:,net.netfilter.nf_conntrack_max%3D131072",
          "verification": {}
        },
        "mode": 420
        }
        ]
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "passwordHash": "$6--REDACTED--0 (create a password hash using Linux for example)",
        "sshAuthorizedKeys": [
          "ssh-rsa VL--REDACTED--Z0z="
        ]
      }
    ]
  }
}

Ignition files have many options and technically it could perform all steps described in this article, but I like to show the steps required so you will better understand the components that are required.

Ignition files can be validated via the Container Linux website.

Installing Container Linux

The first thing you want to do is install Container Linux on the disk. In this example I install the integrated VMware drivers, note that the ignition file is used to set authentication options.

sudo coreos-install -d /dev/sda -o vmware_raw -i yourcustomignfile.ign -C stable

Reboot and you should see a boot menu appear shortly, indicating that installation was successful. After booting from disk change the hostname and configure the network, it is very important understand that both the hostname but also the IP address will be bound to the trust certificates used in Kubernetes! (I prefer reserved dhcp leases)

Both hostname and IP address will be used for certificate trust, changing them later is a pain!

I use the following names since I regard the master as a normal node.

  • kube1.domain.fqn
  • kube2.domain.fqn
  • etc.

Prepare the node for Kubernetes

The following steps describe the components required for a Kubernetes node, perform these steps as root and don’t bother about sudo.

Every node must have the kubelet service running, it is the heart of Kubernetes. It bootstraps the cluster and does low level network setup. It does all the low level work and depends on tools we need to install first. Since Kubernetes runs in Docker, it’s both the chicken and the egg basically.

All tools will run from /opt, Container Linux respects this path when upgrading.

Prepare the paths and set the release version

# set an ENV for later use, do NOT forget this part ;)
RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
CNI_VERSION="v0.7.4"

# directories
mkdir -p /opt/bin /opt/cni/bin /etc/systemd/system/kubelet.service.d

Install CNI, the Container Network Interface

First thing we need is CNI or Container Network Interface, it enables the kubelet service to setup and manage the Kubernetes network. Make sure CNI is installed in /opt/cni/bin and the files are executable before proceeding.

curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-amd64-${CNI_VERSION}.tgz" | tar -C /opt/cni/bin -xz

Install low level Kubernetes tools

Next up are tools like crictl, kubelet, kubeadm and kubectl, these tools have no external dependencies making installation easy.

kubeadm: needed for low level node administration
kubelet: low level kubernetes bootstrapper
kubectl: user interface for Kubernetes
crictl: cri-o tool, part of cri-o containers

curl -L https://github.com/kubernetes-incubator/cri-tools/releases/download/${RELEASE}/crictl-${RELEASE}-linux-amd64.tar.gz | tar -C /opt/bin -xz

cd /opt/bin
curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl}
chmod +x /opt/bin/{kubeadm,kubelet,kubectl,crictl}

That’s basically it, now all we need to do is install and enable the kubelet service.

Kubelet is installed in /opt on Container Linux, we need to change that in the service file.

curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/kubelet.service" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service

# this is the systemd way of changing a service, it's elegant
mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/10-kubeadm.conf" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# Update kubelet service to allow Container Linux volumeplugins
echo "KUBELET_EXTRA_ARGS=--volume-plugin-dir=/var/lib/kubelet/volumeplugins" > /etc/default/kubelet
# Almost done, enable services and reboot the node
systemctl enable docker
systemctl enable kubelet

Your node is ready

After this you can start installing Kubernetes with the kubeadm tool, this will be described in another blog.

Stay tuned..

/by