Wireguard on K8s cluster with Helm

Posted on 26 Jan 2023

I wish more companies were adopting Wireguard. It's fast, simple and makes the job of setting up VPN networking a breeze. Akin to SSH protocol, minus bells and whistles and configuration burden. A couple of years ago I've covered the Wireguard on AWS topic, which works well for most of the use cases, however, this time we're going to talk about shipping Wireguard support into a Kubernetes cluster via Helm, hosted on DigitalOcean. Same setup will apply to pretty much any other cluster.

Networking

There are a couple of reasons why we'd want to reach out for a tool like Wireguard, (or any other VPN product for that matter):

  • Eliminate external access to cloud resources (VPCs, Databases, etc) where possible.
  • Secure private networking and DNS resolution.

First of all, if you've ever had to deal with resources hosted in cloud providers, like instances and databases, you're probably familiar with a concept of VPC and Security Groups. These things allow you to control access surface (aka firewall), and will require hole-punching for any external connections: you might want to connect to your database from a development machine, which requires your public IP address to be whitelisted.

If you're connected to your VPC via VPN, you don't have to worry about granting access to random IP addresses, and can limit access to VPN node(s) instead. As a bonus, we're able to use the internal DNS and connect to private resources by names instead of private IP addresses. This is a win-win scenario. Now, onto k8s.

Kubernetes

Regardless of your cloud provider DNS setup, Kubernetes (k8s) makes use of it's own DNS service running internally in the cluster. For example, if we deploy a web application myapp running on HTTP port 8080 in the default namespace, any pod or service in the cluster can directly communicate with it via http://myapp.default.svc.cluster.local:8080 endpoint.

This works great until you need to expose the service to external clients, for whatever reason. Sure, for quick access kubectl port-forward might be good enough, but anything else will require extra setup (load balancer, SSL termination, dns mapping, etc), and is probably an overkill unless you plan on publicly launching the service.

Given we're setting up Wireguard in k8s cluster, anyone with VPN access should be able to reach the private k8s DNS endpoints from their machines, programmatically or via browser. We could also add an extra layer of authentication, such as Google OAuth, but it's optional.

Cluster Setup

All you need is a working k8s cluster, helm cli and wg-tools (for key generation) installed locally. This post covers DOKS product, but getting things to work on other k8s flavors should be very similar. I really like to build stuff, so at some point I've come up with a helm-wireguard repo designed to:

  • Quickly setup Wireguard in any k8s cluster via Helm chart.
  • Allow running multiple instances of Wireguard, one per node.
  • Run a stand-alone DNS server for VPN clients.
  • Expose Prometheus Wireguard metrics endpoint for basic VPN usage monitoring.

A fair disclaimer is in order:

  • Helm-wireguard repo builds sosedoff/wireguard Docker image.
  • Target image is based on debian:bullseye-slim distribution.
  • Final image uses the standard Wireguard tooling.
  • Image includes a HTTP server to expose Wireguard health check and Prometheus metrics, built in Go.

First, we'll need to install the chart repo:

$ helm repo add wireguard https://raw.githubusercontent.com/sosedoff/helm-wireguard/main/repo/
$ helm repo update

Once complete, let's create a values.yaml file for the Helm chart:

network: 10.10.0.1/24
privateKey: <WG_PRIVATE_KEY>
publicKey: <WG_PUBLIC_KEY>

# By default we will use DaemonSet type of deployment to run wireguard server peer
# on each K8s node. To switch to regular deployment uncomment:
# replicas: 1
# deployKind: Deployment

loadbalancer:
  annotations:
    # These annotations are specific to DigitalOcean but should be similar in other cloud providers.
    service.beta.kubernetes.io/do-loadbalancer-name: "k8s-wireguard"
    service.beta.kubernetes.io/do-loadbalancer-size-unit: "1"
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-port: "8080"
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-protocol: "http"
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-path: "/health"

# A list of peers with access to our VPN gateway
peers:
  user1:
    privateKey: PEER1_PRIVATE_KEY
    publicKey: PEER1_PUB_KEY
    src: 10.10.0.2/32
  user2:
    privateKey: PEER2_PRIVATE_KEY
    publicKey: PEER2_PUB_KEY
    src: 10.10.0.3/32

You could probably leave the network field unchanged as it won't be overlapping with any default VPCs ranges, or adjust it if necessary. As for the privateKey and publicKey values, we'll have to generate those (or you can supply existing ones if you have em):

# Generate the server peer keys
$ wg genkey > privatekey           # make a private key
$ wg pubkey < privatekey > pubkey  # generate a public key

# Copy these values into the config
$ cat privatekey # should replace WG_PRIVATE_KEY in values.yaml
$ cat pubkey     # should replace WG_PUBLIC_KEY in values.yaml

Use the same process to generate key pairs for peers in the values.yaml file.

Now, it's time to create k8s resources. Fire up Helm command:

$ helm upgrade wireguard wireguard/wireguard \
  --install \
  --atomic \
  --create-namespace \
  --namespace wireguard \
  --values ./values.yaml

We tell Helm to create a wireguard namespace and install all k8s resources using that. This whole procedure might take some time, approx 5 mins or so, and uses --atomic flag, which means that if the run fails, helm will roll back to a previous state and delete all existing resources. Add a --debug for the debug output.

Once the command is done running, we can check if things are running as expected:

$ kubectl get pods -n wireguard
NAME                         READY   STATUS    RESTARTS   AGE
wireguard-794ddd6b55-tlmk2   1/1     Running   0          7d7h

$ kubectl logs wireguard-794ddd6b55-tlmk2 -n wireguard
entrypoint: enabling ip forwarding
net.ipv4.ip_forward = 1
net.ipv4.conf.all.forwarding = 1
net.ipv6.conf.all.forwarding = 1
2023/01/19 22:46:55 enabling wireguard interface wg0
2023/01/19 22:46:55 starting coredns
2023/01/19 22:46:55 [wg0] starting monitor
2023/01/19 22:46:55 starting prometheus metrics at: :9090
2023/01/19 22:46:55 starting web endpoint on port 8080
.:53

Most of the resources created are being k8s-native, like DaemonSet, Secret and others, however, the most important bit is the loadbalancer setting in the config. The LoadBalancer configuration will be picked up by DOKS agent (runs by default), and will trigger creation of a DO load balancer named k8s-wireguard.

Load balancer will allow traffic into k8s cluster node(s) and provide a static IP address. If you don't want to deal with IP addresses in Wireguard client configs, make sure to create a public DNS record like clustername.wg.mycorp.com. Load balancer settings are controlled entirely by annotations, so check em out.

To get the IP address of the loadbalancer, run:

$ kubectl get services -n wireguard
NAME           TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)                              AGE
wireguard      ClusterIP      10.245.218.171   <none>            8080/TCP,51820/UDP,9090/TCP,53/UDP   7d7h
wireguard-lb   LoadBalancer   10.245.59.156    XXX.XXX.XXX.XXX   8080:30535/TCP,51820:32703/UDP       7d7h
# ^ load balancer resource created by DigitalOcean agent

External IP address is listed as XXX.XXX.XXX.XXX in our example. With the server part configuration complete, let's move onto the client setup.

Client Setup

Wireguard provides a set of official software packages for Linux/MacOS/Windows, so pick one that works for you, the configuration portion will be exactly the same for all of them.

Let's create a new Wireguard client config clustername-doks.conf:

[Interface]
PrivateKey = <YOUR_CLIENT_PRIVATE_KEY>
Address = 10.10.0.2/32
DNS = 10.10.0.1

[Peer]
PublicKey = <WG_SERVER_PUBKEY>
AllowedIPs = 10.10.0.0/16
Endpoint = XXX.XXX.XXX.XXX:51820
PersistentKeepalive = 15

Where YOUR_CLIENT_PRIVATE_KEY is the client peer private key you've generated earlier, as part of user1 or user2 in values.yaml file. The WG_SERVER_PUBKEY is the public key of the server peer, we've saved it locally in pubkey file. The XXX.XXX.XXX.XXX part is the public IPv4 of the DOKS loadbalancer obtained in the previous section, it should stay permanent unless the loadbalancer is destroyed. Import this config into the Wireguard application and start the tunnel.

Testing

With all our components in place we can finally put everything to test. It's important to mention that VPN we've configured does not route all traffic through the k8s cluster by default. This is controller by AllowedIPs in the client config, and essentially tells Wireguard client to only send packets destined to 10.10.0.0/16 subnet. To force everything through the VPN, swap the value to 0.0.0.0/0 instead.

In our current setup, DNS setting is changed to 10.10.0.1, which is a CoreDNS server run by helm-wireguard package. All that service does is forward DNS queries to the /etc/resolv.conf locally, effectively forwarding the queries to the k8s cluster DNS server. This allows us to resolve any fqdn in k8s cluster locally.

The helm-wireguard package exposes a few endpoints: one for general health checks by a loadbalancer, another for Prometheus metrics:

  • http://wireguard.wireguard.svc.cluster.local:8080/health
  • http://wireguard.wireguard.svc.cluster.local:9090/metrics

You should be able to open these up in a browser or use curl in terminal. Given everything is setup correctly, these endpoint should return a successful response.

In case if that does not work, use dig command to see if DNS resolution is being handled by another service. A successful query might looks something like:

$ dig wireguard.wireguard.svc.cluster.local
;; ANSWER SECTION:
wireguard.wireguard.svc.cluster.local. 5 IN A   10.245.218.171
;; Query time: 41 msec
;; SERVER: 10.10.0.1#53(10.10.0.1)

Note the responding server 10.10.0.1. This is correct and exactly what's being provided to the Wireguard client via DNS = 10.10.0.1 setting. Another local service might override DNS configuration, so you could try to run the query against the k8s DNS:

$ dig @10.10.0.1 wireguard.wireguard.svc.cluster.local

If that works and resolves the requested DNS entry correctly, you should start looking into how your system is setup, particularly around network / wifi setup.

With that being said, we are able to achieve our goal of accessing private k8s resources directly via VPN. The DNS server runs in k8s and should be able to resolve private DigitalOcean resource endpoints (Redis, Postgres, etc) as well as any private domains. But what's also important is the fact that we're now able to access said resources directly, given they allow connections from within the VPC itself.

Wrap

We've covered a lot of ground but k8s is a beast on its own, not to mention Wireguard. Setting everything up correctly might be tricky depending on how both k8s cluster and DO VPCs are configured, and might require a whole lot of troubleshooting. As a result, you'll be able to run a Wireguard VPN on DigitalOcean in a split-tunnel fashion and securely connect to any private resources hosted on DigitalOcean.

Project is hosted on Github, feel free to check it out!