Wireguard on K8s cluster with Helm
I wish more companies were adopting Wireguard. It's fast, simple and makes the job of setting up VPN networking a breeze. Akin to SSH protocol, minus bells and whistles and configuration burden. A couple of years ago I've covered the Wireguard on AWS topic, which works well for most of the use cases, however, this time we're going to talk about shipping Wireguard support into a Kubernetes cluster via Helm, hosted on DigitalOcean. Same setup will apply to pretty much any other cluster.
There are a couple of reasons why we'd want to reach out for a tool like Wireguard, (or any other VPN product for that matter):
- Eliminate external access to cloud resources (VPCs, Databases, etc) where possible.
- Secure private networking and DNS resolution.
First of all, if you've ever had to deal with resources hosted in cloud providers, like instances and databases, you're probably familiar with a concept of VPC and Security Groups. These things allow you to control access surface (aka firewall), and will require hole-punching for any external connections: you might want to connect to your database from a development machine, which requires your public IP address to be whitelisted.
If you're connected to your VPC via VPN, you don't have to worry about granting access to random IP addresses, and can limit access to VPN node(s) instead. As a bonus, we're able to use the internal DNS and connect to private resources by names instead of private IP addresses. This is a win-win scenario. Now, onto k8s.
Regardless of your cloud provider DNS setup, Kubernetes (k8s) makes use of it's own
DNS service running internally in the cluster. For example, if we deploy a web
myapp running on HTTP port
8080 in the
default namespace, any
pod or service in the cluster can directly communicate with it via
This works great until you need to expose the service to external clients, for
whatever reason. Sure, for quick access
kubectl port-forward might be good enough,
but anything else will require extra setup (load balancer, SSL termination, dns mapping, etc),
and is probably an overkill unless you plan on publicly launching the service.
Given we're setting up Wireguard in k8s cluster, anyone with VPN access should be able to reach the private k8s DNS endpoints from their machines, programmatically or via browser. We could also add an extra layer of authentication, such as Google OAuth, but it's optional.
All you need is a working k8s cluster, helm cli and wg-tools (for key generation) installed locally. This post covers DOKS product, but getting things to work on other k8s flavors should be very similar. I really like to build stuff, so at some point I've come up with a helm-wireguard repo designed to:
- Quickly setup Wireguard in any k8s cluster via Helm chart.
- Allow running multiple instances of Wireguard, one per node.
- Run a stand-alone DNS server for VPN clients.
- Expose Prometheus Wireguard metrics endpoint for basic VPN usage monitoring.
A fair disclaimer is in order:
- Helm-wireguard repo builds
- Target image is based on
- Final image uses the standard Wireguard tooling.
- Image includes a HTTP server to expose Wireguard health check and Prometheus metrics, built in Go.
First, we'll need to install the chart repo:
$ helm repo add wireguard https://raw.githubusercontent.com/sosedoff/helm-wireguard/main/repo/ $ helm repo update
Once complete, let's create a
values.yaml file for the Helm chart:
network: 10.10.0.1/24 privateKey: <WG_PRIVATE_KEY> publicKey: <WG_PUBLIC_KEY> # By default we will use DaemonSet type of deployment to run wireguard server peer # on each K8s node. To switch to regular deployment uncomment: # replicas: 1 # deployKind: Deployment loadbalancer: annotations: # These annotations are specific to DigitalOcean but should be similar in other cloud providers. service.beta.kubernetes.io/do-loadbalancer-name: "k8s-wireguard" service.beta.kubernetes.io/do-loadbalancer-size-unit: "1" service.beta.kubernetes.io/do-loadbalancer-healthcheck-port: "8080" service.beta.kubernetes.io/do-loadbalancer-healthcheck-protocol: "http" service.beta.kubernetes.io/do-loadbalancer-healthcheck-path: "/health" # A list of peers with access to our VPN gateway peers: user1: privateKey: PEER1_PRIVATE_KEY publicKey: PEER1_PUB_KEY src: 10.10.0.2/32 user2: privateKey: PEER2_PRIVATE_KEY publicKey: PEER2_PUB_KEY src: 10.10.0.3/32
You could probably leave the
network field unchanged as it won't be overlapping
with any default VPCs ranges, or adjust it if necessary. As for the
publicKey values, we'll have to generate those (or you can supply existing ones if you have em):
# Generate the server peer keys $ wg genkey > privatekey # make a private key $ wg pubkey < privatekey > pubkey # generate a public key # Copy these values into the config $ cat privatekey # should replace WG_PRIVATE_KEY in values.yaml $ cat pubkey # should replace WG_PUBLIC_KEY in values.yaml
Use the same process to generate key pairs for
peers in the
Now, it's time to create k8s resources. Fire up Helm command:
$ helm upgrade wireguard wireguard/wireguard \ --install \ --atomic \ --create-namespace \ --namespace wireguard \ --values ./values.yaml
We tell Helm to create a
wireguard namespace and install all k8s resources using
that. This whole procedure might take some time, approx 5 mins or so, and uses
--atomic flag, which means that if the run fails, helm will roll back to a previous
state and delete all existing resources. Add a
--debug for the debug output.
Once the command is done running, we can check if things are running as expected:
$ kubectl get pods -n wireguard NAME READY STATUS RESTARTS AGE wireguard-794ddd6b55-tlmk2 1/1 Running 0 7d7h $ kubectl logs wireguard-794ddd6b55-tlmk2 -n wireguard entrypoint: enabling ip forwarding net.ipv4.ip_forward = 1 net.ipv4.conf.all.forwarding = 1 net.ipv6.conf.all.forwarding = 1 2023/01/19 22:46:55 enabling wireguard interface wg0 2023/01/19 22:46:55 starting coredns 2023/01/19 22:46:55 [wg0] starting monitor 2023/01/19 22:46:55 starting prometheus metrics at: :9090 2023/01/19 22:46:55 starting web endpoint on port 8080 .:53
Most of the resources created are being k8s-native, like
others, however, the most important bit is the
loadbalancer setting in the config.
LoadBalancer configuration will be picked up by DOKS agent (runs by default),
and will trigger creation of a DO load balancer named
Load balancer will allow traffic into k8s cluster node(s) and provide a static IP address.
If you don't want to deal with IP addresses in Wireguard client configs, make
sure to create a public DNS record like
clustername.wg.mycorp.com. Load balancer
settings are controlled entirely by annotations, so check em out.
To get the IP address of the loadbalancer, run:
$ kubectl get services -n wireguard NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE wireguard ClusterIP 10.245.218.171 <none> 8080/TCP,51820/UDP,9090/TCP,53/UDP 7d7h wireguard-lb LoadBalancer 10.245.59.156 XXX.XXX.XXX.XXX 8080:30535/TCP,51820:32703/UDP 7d7h # ^ load balancer resource created by DigitalOcean agent
External IP address is listed as
XXX.XXX.XXX.XXX in our example. With the server part
configuration complete, let's move onto the client setup.
Wireguard provides a set of official software packages for Linux/MacOS/Windows, so pick one that works for you, the configuration portion will be exactly the same for all of them.
Let's create a new Wireguard client config
[Interface] PrivateKey = <YOUR_CLIENT_PRIVATE_KEY> Address = 10.10.0.2/32 DNS = 10.10.0.1 [Peer] PublicKey = <WG_SERVER_PUBKEY> AllowedIPs = 10.10.0.0/16 Endpoint = XXX.XXX.XXX.XXX:51820 PersistentKeepalive = 15
YOUR_CLIENT_PRIVATE_KEY is the client peer private key you've generated
earlier, as part of
values.yaml file. The
the public key of the server peer, we've saved it locally in
pubkey file. The
XXX.XXX.XXX.XXX part is the public IPv4 of the DOKS loadbalancer obtained in the
previous section, it should stay permanent unless the loadbalancer is destroyed.
Import this config into the Wireguard application and start the tunnel.
With all our components in place we can finally put everything to test. It's important
to mention that VPN we've configured does not route all traffic through the k8s cluster by default.
This is controller by
AllowedIPs in the client config, and essentially tells Wireguard
client to only send packets destined to
10.10.0.0/16 subnet. To force everything
through the VPN, swap the value to
In our current setup, DNS setting is changed to
10.10.0.1, which is a CoreDNS
server run by
helm-wireguard package. All that service does is forward DNS queries
/etc/resolv.conf locally, effectively forwarding the queries to the k8s cluster
DNS server. This allows us to resolve any
fqdn in k8s cluster locally.
helm-wireguard package exposes a few endpoints: one for general health checks
by a loadbalancer, another for Prometheus metrics:
You should be able to open these up in a browser or use
curl in terminal. Given
everything is setup correctly, these endpoint should return a successful response.
In case if that does not work, use
dig command to see if DNS resolution is being
handled by another service. A successful query might looks something like:
$ dig wireguard.wireguard.svc.cluster.local ;; ANSWER SECTION: wireguard.wireguard.svc.cluster.local. 5 IN A 10.245.218.171 ;; Query time: 41 msec ;; SERVER: 10.10.0.1#53(10.10.0.1)
Note the responding server
10.10.0.1. This is correct and exactly what's being provided
to the Wireguard client via
DNS = 10.10.0.1 setting. Another local service might
override DNS configuration, so you could try to run the query against the k8s DNS:
$ dig @10.10.0.1 wireguard.wireguard.svc.cluster.local
If that works and resolves the requested DNS entry correctly, you should start looking into how your system is setup, particularly around network / wifi setup.
With that being said, we are able to achieve our goal of accessing private k8s resources directly via VPN. The DNS server runs in k8s and should be able to resolve private DigitalOcean resource endpoints (Redis, Postgres, etc) as well as any private domains. But what's also important is the fact that we're now able to access said resources directly, given they allow connections from within the VPC itself.
We've covered a lot of ground but k8s is a beast on its own, not to mention Wireguard. Setting everything up correctly might be tricky depending on how both k8s cluster and DO VPCs are configured, and might require a whole lot of troubleshooting. As a result, you'll be able to run a Wireguard VPN on DigitalOcean in a split-tunnel fashion and securely connect to any private resources hosted on DigitalOcean.
Project is hosted on Github, feel free to check it out!