This document describes how to install and use Traefik 2 as an edge router on Kubernetes combined with Network Policies for enhanced security.
This is part 1 of the series, it will focus on installing and understanding Traefik 2.
Introduction
We love Traefik, it’s an amazing edge router that does so many things right. Gone are the days of static configuration files and manual reloads on changes. Traefik 2 brings many nice features like HTTP2 and TLS TCP support with Letsencrypt, a new dashboard and a much more extensible way of configuration. It is one of the best tools to have as a hosting company. Traefik you rock!
This document describes a working setup for Traefik 2 with Kubernetes, I will explain basic configuration options to get you started.
Installation
Installation of Traefik 2 is pretty simple but the documentation is not very helpful at the moment, it is overwhelming with options that might misguide you. Don’t worry, this howto describes all steps needed for a working setup. Once you’ll understand the basics things should be easy to get going.
Step 1) prepare CRD, RBAC and a ServiceAccount for Traefik 2
Ready for Kubernetes bingo! Here we go..
CRD stands for Custom Resource Definition, a term used by Kubernetes to describe a way to extend features in Kubernetes. Traefik 2 requires CRD objects for configuration and this is a much better approach compared to the original Traefik where you would have to “embed” config options using Ingress annotations. Every CRD has it’s own scope that allows an admin to set options without interfering with other CRD’s.
RBAC stands for Role Based Access Control, a system to describe permissions. In short RBAC increases security by limiting access based on roles. Traefik 2 needs permissions to read services and pods for discovery etc.
A ServiceAccount is required to bind the permissions defined in RBAC to Traefik via a deployment.
Save the contents below to a file named traefik2-crd-rbac.yaml and apply it to your cluster. (you don’t need to specify a namespace since it’s a cluster resource.) As an alert admin you should not trust this site but refer to the source, https://docs.traefik.io/user-guides/crd-acme/.
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutes.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRoute
plural: ingressroutes
singular: ingressroute
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutetcps.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRouteTCP
plural: ingressroutetcps
singular: ingressroutetcp
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: middlewares.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: Middleware
plural: middlewares
singular: middleware
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: tlsoptions.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: TLSOption
plural: tlsoptions
singular: tlsoption
scope: Namespaced
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.containo.us
resources:
- middlewares
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- ingressroutes
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- ingressroutetcps
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- tlsoptions
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: traefik-ingress
Step 2) Install Traefik 2 as a deployment
Save the contents below to a file named traefik2.yaml and apply it. It will create a namespace and deploy Traefik in a dedicated namespace. Deploying Traefik to it’s own namespace is a good security practise that allows fine grained Network Policies in a later stage. Note that this deployment has one replica for testing purpose, in production you would have to optimize this settings with affinity etc.
Make sure you read the deployment and change the email address!
apiVersion: v1
kind: Namespace
metadata:
labels:
ingress-controller: traefik
name: traefik-ingress
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
labels:
app: traefik
name: traefik
namespace: traefik-ingress
spec:
replicas: 1
selector:
matchLabels:
app: traefik
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: traefik
spec:
containers:
- name: traefik
image: traefik:v2.0
imagePullPolicy: Always
args:
- --api.debug=true
- --api.insecure=true
- --log.level=debug
- --entrypoints.http.Address=:80
- --entrypoints.https.Address=:443
- --entrypoints.traefik.Address=:8080
- --entrypoints.mqtt.Address=:8883
- --providers.kubernetescrd
- --ping=true
- --certificatesresolvers.default.acme.tlschallenge=true
- --certificatesresolvers.default.acme.email={{EMAIL}}
- --certificatesresolvers.default.acme.storage=acme.json
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
- containerPort: 8080
name: admin
protocol: TCP
- containerPort: 8883
name: mqtt
protocol: TCP
livenessProbe:
failureThreshold: 2
httpGet:
path: /ping
port: admin
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: admin
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 250m
memory: 128Mi
requests:
cpu: 100m
memory: 64Mi
securityContext:
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALL
restartPolicy: Always
serviceAccount: traefik-ingress-controller
serviceAccountName: traefik-ingress-controller
We try to secure this deployment as much as possible by using a securitycontext setting, unfortunately it is not possible to run Traefik as a normal user AFAIK.
Step 3) Understand what’s happening
We created a dedicated namespace “traefik-ingress” to allow Network Policies in a later stage. (make sure to label this namespace, since they are required for Network Policies.) Note that this deployment makes use of the serviceaccount that we created earlier to gain permissions required for operation. We also define a minimal set of options as arguments, this tells Traefik where to look for configuration options.
In this example we create a Traefik 2 deployment in the namespace “traefik-ingress” with the arguments to create several entrypoints.
let’s look at the arguments defined.
- –api.debug=true, allow this to see logging turn off in production obviously
- –log.level=debug, show debug output, disable in production
- –api.insecure=true, we want to allow a dashboard without a SSL certificate
- –entrypoints.http.Address=:80, http entrypoint on po80
- –entrypoints.https.Address=:443, https entrypoint on port 443
- –entrypoints.traefik.Address=:8080, traefik dashboard entrypoint on port 8080
- –entrypoints.mqtt.Address=:8883, mqtt entrypoint on port 8883
- –providers.kubernetescrd, look for Kubernetes configuration options
- –ping=true, respond to readiness and liveness probes on the admin interface
- –certificatesresolvers.default.acme.tlschallenge=true, use TLS validation for Letsencrypt, make sure incoming port 443 is reachable to Traefik!
- –certificatesresolvers.default.acme.email={{EMAIL}}, enter your Letsencrypt email here
- –certificatesresolvers.default.acme.storage=acme.json, file used for letsencrypt certificates
The original documentation was unclear to me how to change the Traefik dashboard port from 8080 to something else. The dashboard port for Traefik is an entrypoint named “traefik” which by default binds to port 8080, now you know, spread the word :)
Step 4) Configure the Traefik 2 Service
Save the contents below to a file named traefik2-svc.yaml and apply it. This will create the services required for Traefik, as you can see it creates several services for HTTP, HTTPS, Admin and a mqtt port we will use later to demonstrate TLS options.
apiVersion: v1
kind: Service
metadata:
annotations:
name: traefik
namespace: traefik-ingress
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
- name: admin
port: 8080
protocol: TCP
targetPort: 8080
- name: https
port: 443
protocol: TCP
targetPort: 443
- name: mqtt
port: 8883
protocol: TCP
targetPort: 8883
selector:
app: traefik
type: LoadBalancer
Using Traefik 2
You should now see a Traefik 2 pod running on your cluster in the desired namespace, let’s check the logs and the service.
$ kubectl -n traefik-ingress logs -f -l app=traefik
$ kubectl -n traefik-ingress get svc -o yaml
Before proceeding we can test the Traefik 2 dashboard which is available on port 8080, since we allowed insecure API it’s available on plain HTTP. Go ahead and open http://{{EXTERNAL-IP}}:8080, where {{EXTERNAL-IP}} is found with;
kubectl -n traefik-ingress get svc traefik
The service ports are used for the IngressRoute objects we are going to define, it might be a good idea to disable the admin service on a public connection! Think about this..
Testing a simple HTTP service
Once the dashboard is working we can proceed with the next step. Create a simple Nginx deployment in the default namespace and create a HTTP service for this deployment on port 80 which we later will config for Traefik 2 using the IngressRoute object.
kubectl create deployment nginx --image=nginx
# create a service for this deployment
kubectl create service clusterip nginx --tcp=80:80
Expose a service via a IngressRoute
To be able to access the nginx deployment, we need to create a IngressRoute object. An IngressRoute object is part of the Traefik 2 CRD’s so this will only work on a Kubernetes setup with Traefik 2 installed. A simple example shows the minimal config required for this task. In this case we setup a simple IngressRoute that responds to the hostname “test.k8s” on every plain HTTP request.
Before proceeding, make sure to setup a hostname matching EXTERNAL-IP in your DNS or local hostfile.
Traefik 2 is watching every namespace for IngressRoute objects, creating this object immediately instructs Traefik to take action, no reloads needed!
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: ingressroute
spec:
entryPoints:
- http
routes:
- match: Host(`test.k8s`)
kind: Rule
services:
- name: nginx
port: 80
Test the IngressRoute
We created a nginx deployment and exposed it’s service via an IngressRoute, you should now be able to access this service. Let’s see
curl http://test.k8s
This should show the default Nginx page, indicating a working setup. Every request will go through Traefik 2, increasing the replica count in Kubernetes will automatically update Traefik with the corresponding configuration without any manual action. Now that’s neat!
Take a look at the dashboard for some interesting information on pods and endpoints.
Middlewares, TLS, Network Policies and more..
That’s it, you now have a working setup using Traefik 2 and Nginx. As you can see the IngressRoute object is the most important object to define access to your services. In the next part I will show you the various options to modify requests using Middlewares and secure a namespace using Network Policies. We will also address IngressRouteTCP which allows TLS encrypted TCP streams.
Stay tuned..
