Combining k3d and GitHub Actions for easy, quick and cheap E2E testing

The Old Way

During my first year of Software Engineering, I understood the importance of two things:

1. E2E tests running with CI. They save a ton of manual effort testing the software, increase development velocity by inspiring confidence for shipping the code and usually testing a few common scenarios end-to-end will catch most bugs.
2. Quick iteration cycles. They allow for quickly spotting mistakes and arriving at the target solution.

A year back, when I was developing the Cassandra Operator at Rook, doing E2E testing with the tools available left a lot to be desired:

• Provisioning a Kubernetes cluster was done with an Ansible script, which started Virtual Machines for K8s Nodes. Minikube doesn’t support multiple Nodes, so it wasn’t an option. This method had a very high overhead and a very slow startup time.
• Because of the high overhead, we were also limited in the types of scenarios we could test. For example, I couldn’t test scale ups and scale downs of Cassandra, because of the limited CI infrastructure that quickly got depleted by the VM overhead.
• The Jenkins CI we used required a lot of manual setup and maintenance.

The New Generation of Tools

Luckily, a lot has changed in one year. The new generation of tools allow us to run E2E tests fast and easily develop CI pipelines for our project. Let’s take a look at the tools we’ll use:

k3d

k3d is the fastest and lightest way to deploy a Kubernetes cluster. It can deploy a multi-node Kubernetes cluster and is blazing fast.

To further drive this point home, spinning up a new 3-node Kubernetes cluster takes about 11 seconds on my 10-year old Toshiba laptop:

time k3d create cluster e2e-test-cluster \
  --workers 3 \
  --wait --timeout=5m \
  --update-kubeconfig4.10s user 2.34s system 58% cpu 11.056 total

k3d achieves this speed by doing two things:

• It deploys k3s, a trimmed-down but compliant Kubernetes distribution.
• It uses a container for each Node, not a Virtual Machine. A DinD-like solution is used.

GitHub Actions

GitHub Actions allow us to build workflows that are executed when certain events happen in a GitHub repo, e.g., when a pull request is created. They are a CI/CD system that is closely integrated with GitHub, so it is very easy to get started. An example workflow looks like the following:https://medium.com/media/16e0a6f4bc5fe627fa70e28e9c421ce5

In addition, they are completely free for public repositories, with hefty usage limits and machine specs (2-core, 7GB). This makes them ideal for open source repos.

A Real-Life Example

At Arrikto, we have an open-source project called oidc-authservice, which is essentially an authenticating proxy addon for the Envoy Proxy. The AuthService plays a small part in a complicated distributed system, so testing it end-to-end is no easy matter. We need to install:

• Istio
• Dex, an OIDC Provider
• A resource server (NGINX)

in addition to the AuthService, in order to test a full login flow.

This is a complicated setup, but it’s where Kubernetes really shines. To test our software end-to-end, we have to define our setup as Kubernetes manifests. Then write our E2E test, which will:

1. Apply our manifests to a Kubernetes cluster.
2. Wait for everything to become ready.
3. Test the login flow end-to-end. Since access to an in-cluster service is required, we can use “kubectl port-forward”.

You can find the full test code here along with instructions on how to run it.

Notice that this test can run on any Kubernetes cluster, the only thing it expects is to find Kubernetes client credentials inside the environment it runs. However, for our purposes, we will use k3d to make it run super-fast.

After writing our E2E test, it’s time to make it part of our CI, so that it runs whenever a pull request is created. For that purpose, we write the following workflow:

Simplified GitHub workflow for CI

It downloads the necessary tools using a simple Python script, tests the code, builds and pushes an image and then performs an E2E test with that image, using k3d. All of this is done in just under 5 minutes, at zero dollar cost.

Conclusion

Whether you are developing an operator for a distributed system or a small part in a sea of interconnected components, end-to-end testing is always a great idea.

We saw how k3d and GitHub Actions can be combined to create fast and lightweight E2E tests and integrate them easily in CI. They are a powerful addition to the Kubernetes (and not only) developer’s toolchain and enable open-source projects to easily start using CI at no cost.

Kudos to Darren Shepherd and the Rancher team for the amazing k3s and k3d tools!

PS. A GitHub Action for starting a k3d cluster would be an awesome next step.