Auto provisioning Let’s Encrypt wildcard certificates with cert-manager on GKE
Google Cloud Platform offers Google Managed Certificates for External Load Balancers and GKE Ingress but does not currently support the ability to manage wildcard certs. If you are looking to have Google manage your *.example.com certificate then you are out of luck until the feature becomes available.
This guide outlines how to use cert-manager on GKE to automatically provision a wildcard certificate when your Ingress resource gets created. Cert-manager will perform domain verification, request the certificate from Let’s Encrypt, and handle auto-renewing your certificate before the 90 day expiration date.
- Requires GKE version v1.16.0 or greater
- Requires a running GKE cluster
I will be doing these steps in Cloud Shell. Cloud shell can be accessed through the GCP console by clicking the Cloud Shell icon on the upper right corner of the console as detailed in the screenshot below.
Once the cloud shell loads up you need to get credentials to connect to your running GKE cluster.
gcloud container clusters get-credentials <cluster_name> — zone <cluster_zone>
Create a namespace where cert-manager will live.
kubectl create ns cert-manager
Create a ClusterRoleBinding to give yourself permissions to deploy cert-manager on the cluster
kubectl create clusterrolebinding cluster-admin-binding — clusterrole=cluster-admin — user=$(gcloud config get-value core/account)
Install the latest cert-manager into your cluster. (to find out the latest release of cert-manager go to the cert-manager Releases Page)
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/<release>/cert-manager.yaml
After a few minutes you should see it running on your cluster (yours will have different names)
kubectl get pods --namespace cert-managerNAME READY STATUS RESTARTS AGE
cert-manager-8454bdfcbb-grgqt 1/1 Running 0 105m
cert-manager-cainjector-56b88d57c7-t2pvf 1/1 Running 0 105m
cert-manager-webhook-5c49f9dc98–994mt 1/1 Running 0 105m
Create the DNS zone if not created yet:
For this guide I created a new zone for my domain in Google Cloud DNS. It is possible that you already have a zone somewhere for your domain and if so you can skip this part. You need a DNS zone to validate you are the owner of the domain for certificate verification. Getting wildcard certificates from Let’s Encrypt requires DNS validation.
To create a public zone on Cloud DNS follow this guide.
Create a GCP service account and download the key:
(You only need to do this if you are using Google Cloud DNS for validation)
You need to create a GCP service account that has access to add a TXT record to your DNS zone. The ClusterIssuer will use it to access Cloud DNS from the K8s cluster and automatically create a TXT record in order to validate you own the domain.
# put your project name into an environment variable
export PROJECT_NAME=[YOUR_PROJECT_NAME]# create service account
gcloud projects add-iam-policy-binding $PROJECT_NAME --member=serviceAccount:svc-cert-manager@$PROJECT_NAME.iam.gserviceaccount.com --role=roles/dns.admin# create and download service account key
gcloud iam service-accounts keys create ./credentials.json --iam-account=k8s-cert-manager@$PROJECT_NAME.iam.gserviceaccount.com --project=$PROJECT_NAME# give the service account dns admin permissions
gcloud projects add-iam-policy-binding $PROJECT_NAME --member=serviceAccount:k8s-cert-manager@$PROJECT_NAME.iam.gserviceaccount.com --role=roles/dns.admin
Note: The use of the dns.admin role in this example role is for convenience. If you want to ensure cert-manager runs under a least privilege service account, you will need to create a custom role with the following permissions:
Create a kubernetes secret from your newly created service account credentials:
I am using Google Cloud DNS to validate my certs but if you are using Cloudflare or Route53 or something else you need to follow the instructions to add API tokens to secrets instead of the service account.
This secret is used by the ClusterIssuer to add a TXT record to Google Cloud DNS for cert validation.
kubectl create secret generic dns-service-account \--from-file=./credentials.json \--namespace=cert-manager
Create the ClusterIssuer:
The ClusterIssuer is the part of cert-manager that tells cert-manager where to get the cert. For this we are using Let’s Encrypt. It also contains the information about what DNS provider you are using and how to authenticate to it. In this example we are using Cloud DNS but there are other solvers that can be plugged into this including Cloudflare and Route53.
cat <<EOF > clusterissuer-letsencrypt.yaml
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
# Secret resource that will be used to store the private key.
# Do not manually create secret, it will be created for you.
# Add a challenge solver, ACME DNS-01 as required for wildcards
# Google Cloud DNS - change if using other provider
# Secret from the google service account key
# The project in which to update the DNS zone
project: <your project id>
Now apply the ClusterIssuer
kubectl apply -f clusterissuer-letsencrypt.yaml
And you can check and see if it successfully registered you to let’s encrypt by describing.
kubectl describe clusterissuer letsencryptStatus:
Last Registered Email: firstname.lastname@example.org
Last Transition Time: 2021-02-06T00:44:25Z
Message: The ACME account was registered with the ACME server
Create your deployment and service:
Now you will create your deployment and service for your application.
For this example I deploy a simple hello-app
kubectl create deployment hello-server --image=gcr.io/google-samples/hello-app:1.0
And expose a service of type NodePort
kubectl expose deployment hello-server --type NodePort --port 80 --target-port 8080
Create your GKE Ingress and get your managed wildcard cert at the same time:
The best part of cert-manager is it automatically adds a TXT record to your DNS and validates and issues a cert from Let’s Encrypt when you create an Ingress resource in GKE. It also auto-renews the certificate for you. Wildcard domain certificates (those covering *.yourdomain.com) can only be requested using DNS validation.
# add an annotation indicating the issuer to use.
# the name of your ingress
- host: "*.example.com"
tls: # placing a host in the TLS config will create a cert
secretName: example-cert #created cert gets put into this secret
It can take over an hour to get your cert. To check the status of the certificate do
kubectl get certNAME READY SECRET AGE
example-cert True example-cert 5h50m
If READY shows False you probably have to wait longer. You can check the status of the cert process for errors by checking the logs of the cert-manager container.
Get the name of the cert-manager pod
kubectl get pods -n cert-managerNAME READY STATUS RESTARTS AGE
cert-manager-8454bdfcbb-grgqt 1/1 Running 0 8h
Check the logs of the cert-manager pod
kubectl logs cert-manager-8454bdfcbb-grgqt -n cert-managerI0206 00:47:01.585423 1 conditions.go:162] Found status change for Certificate "example-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-02-06 00:47:01.585413459 +0000 UTC m=+11290.701379832
You will see errors here if something went wrong. You can also check your DNS provider to see if the TXT record got inserted. If it did then the validation is still going and you need to wait a bit longer.
After a while you should get a public IP from the Load Balancer
kubectl get ingressNAME HOSTS ADDRESS PORTS AGE
myIngress *.example.com <PUBLIC_IP> 80, 443 5h56m
Change your DNS to point to the new Public IP created by the Ingress
You can then change your DNS to point to this new IP address.
You might want to edit your local hosts file first to make sure things work okay before moving your DNS over.