Kubernetes is declarative, in other words, every component is described in the kube-apiserver as manifests. It makes sense to store this configuration in a version control system like Git. This idea brought us Gitops, where every change is done via a git push command. Everything as code is the new normal.

This article describes the advantages of Gitops with ArgoCD

When first starting with Kubernetes you’ll learn to apply manifests using kubectl. A very popular approach is to use an URL as a manifest, this allows very complex configurations to be installed with a single command. Everything required for the installation is included in this URL, it’s almost magic.

Installation is fine, but what about managing the state of the cluster? How do you keep track of versions etc.?

The problem with static manifests

Sooner or later you’ll realise that static manifests are hard to manage, how do you know that the installation is still in the original state? What if we add stuff that is not in the manifest? Recept for disaster right?

Wouldn’t it be nice if the manifest in the URL would be checked with the state of the cluster every 5 minutes?

ArgoCD manages Kubernetes manifests

With ArgoCD you can configure a Git repository as the source for your manifests. ArgoCD will notice changes in this Git repository and it can be configured to perform actions when this occurs. Better yet, a new cluster could use this same Git repository and configuration would take minimal effort.

Git as a central point of truth for your Kubernetes manifests is rock solid

We appreciate the Gitops workflow for this reason, it’s very easy to create a new cluster that conforms to your configuration and keeping it in sync with changes is done automatically. Combine this with Helm and you’ll start to see the added value.

But Mike, are there any Gitops challenges?

ArgoCD Gitops
ArgoCD logo

Glad you ask! There are some challenges when working with Gitops. Mostly are related to storing secrets in Git. Since Kubernetes ‘secrets’ are not encrypted, you’ll need to use inline encryption like SOPS, this requires an additional step in the workflow.

Another weak point is the fact that it’s currently hard to define manifests with parameters. Imagine you want to apply a manifest 4 times with 4 different values, this would require that you define values in each manifest. (Flux does this a little bit better than ArgoCD)

Flux or ArgoCD?

Another fine Gitops tool is Flux. Flux however, comes without a web interface and is perhaps a bit more strict in terms of Gitops principles (installing Flux is done via Gitops). For this reason we recommend ArgoCD, it will get you started with Gitops in no-time!

Stay tuned

Next post will contain a hands-on session with ArgoCD.

This article will guide you setting up Container Linux (CoreOS) for Kubernetes.


Kubernetes can run on almost any host supporting Docker, even on devices like the Raspberry PI. Personally I like Container Linux because it’s lightweight and practically maintenance free after setup. (compared to classic OS’es like Ubuntu or Debian). It’s an OS created specifically for container workloads and it has been a huge contributor to the succes of Kuberentes with techniques like “etcd”.


Let’s run this cluster on at least 2 nodes, make sure they have at least 3G RAM each, both should have at least 20G disks and have basic network connectivity, make sure both nodes can connect to the internet. VMware is nice for this setup, it’s what I used in this example, but you can also run this on Digital Ocean or any other cloud service that offers Container Linux images.

The great thing about Kubernetes is that it doesn’t care about the hardware, it could be AWS combined with a Raspberry PI and VMware. In this article we focus on x64 architecture since Container Linux doesn’t run on ARM.

Starting Container Linux

There are a few ways to install Container Linux, but you’ll need to set a password at least to be able to login. I run Container Linux booted via iPXE with an ignition file. It allows me to configure basic features like the hostname and openssh public keys etc. I think you can paste this file on a webserver and refer to it using boot options when booting from ISO.

The ignition file sets up authentication, but also some other small settings recommended for Kubernetes.

  "ignition": { "version": "2.2.0" },
  "storage": {
    "files": [
              "filesystem": "root",
              "path": "/etc/hostname",
              "mode": 420,
              "contents": {
                        "source": "data:,core1"
        "filesystem": "root",
        "group": {},
        "path": "/etc/sysctl.d/local.conf",
        "user": {},
        "contents": {
          "source": "data:,net.netfilter.nf_conntrack_max%3D131072",
          "verification": {}
        "mode": 420
  "passwd": {
    "users": [
        "name": "core",
        "passwordHash": "$6--REDACTED--0 (create a password hash using Linux for example)",
        "sshAuthorizedKeys": [
          "ssh-rsa VL--REDACTED--Z0z="

Ignition files have many options and technically it could perform all steps described in this article, but I like to show the steps required so you will better understand the components that are required.

Ignition files can be validated via the Container Linux website.

Installing Container Linux

The first thing you want to do is install Container Linux on the disk. In this example I install the integrated VMware drivers, note that the ignition file is used to set authentication options.

sudo coreos-install -d /dev/sda -o vmware_raw -i yourcustomignfile.ign -C stable

Reboot and you should see a boot menu appear shortly, indicating that installation was successful. After booting from disk change the hostname and configure the network, it is very important understand that both the hostname but also the IP address will be bound to the trust certificates used in Kubernetes! (I prefer reserved dhcp leases)

Both hostname and IP address will be used for certificate trust, changing them later is a pain!

I use the following names since I regard the master as a normal node.

  • kube1.domain.fqn
  • kube2.domain.fqn
  • etc.

Prepare the node for Kubernetes

The following steps describe the components required for a Kubernetes node, perform these steps as root and don’t bother about sudo.

Every node must have the kubelet service running, it is the heart of Kubernetes. It bootstraps the cluster and does low level network setup. It does all the low level work and depends on tools we need to install first. Since Kubernetes runs in Docker, it’s both the chicken and the egg basically.

All tools will run from /opt, Container Linux respects this path when upgrading.

Prepare the paths and set the release version

# set an ENV for later use, do NOT forget this part ;)
RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"

# directories
mkdir -p /opt/bin /opt/cni/bin /etc/systemd/system/kubelet.service.d

Install CNI, the Container Network Interface

First thing we need is CNI or Container Network Interface, it enables the kubelet service to setup and manage the Kubernetes network. Make sure CNI is installed in /opt/cni/bin and the files are executable before proceeding.

curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-amd64-${CNI_VERSION}.tgz" | tar -C /opt/cni/bin -xz

Install low level Kubernetes tools

Next up are tools like crictl, kubelet, kubeadm and kubectl, these tools have no external dependencies making installation easy.

kubeadm: needed for low level node administration
kubelet: low level kubernetes bootstrapper
kubectl: user interface for Kubernetes
crictl: cri-o tool, part of cri-o containers

curl -L https://github.com/kubernetes-incubator/cri-tools/releases/download/${RELEASE}/crictl-${RELEASE}-linux-amd64.tar.gz | tar -C /opt/bin -xz

cd /opt/bin
curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl}
chmod +x /opt/bin/{kubeadm,kubelet,kubectl,crictl}

That’s basically it, now all we need to do is install and enable the kubelet service.

Kubelet is installed in /opt on Container Linux, we need to change that in the service file.

curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/kubelet.service" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service

# this is the systemd way of changing a service, it's elegant
mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/10-kubeadm.conf" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# Update kubelet service to allow Container Linux volumeplugins
echo "KUBELET_EXTRA_ARGS=--volume-plugin-dir=/var/lib/kubelet/volumeplugins" > /etc/default/kubelet
# Almost done, enable services and reboot the node
systemctl enable docker
systemctl enable kubelet

Your node is ready

After this you can start installing Kubernetes with the kubeadm tool, this will be described in another blog.

Stay tuned..