Auto provisioning Let’s Encrypt wildcard certificates with cert-manager on GKE

Google Cloud Platform offers Google Managed Certificates for External Load Balancers and GKE Ingress but does not currently support the ability to manage wildcard certs. If you are looking to have Google manage your *.example.com certificate then you are out of luck until the feature becomes available.

This guide outlines how to use cert-manager on GKE to automatically provision a wildcard certificate when your Ingress resource gets created. Cert-manager will perform domain verification, request the certificate from Let’s Encrypt, and handle auto-renewing your certificate before the 90 day expiration date.

Prerequisites:

• Requires GKE version v1.16.0 or greater
• Requires a running GKE cluster

I will be doing these steps in Cloud Shell. Cloud shell can be accessed through the GCP console by clicking the Cloud Shell icon on the upper right corner of the console as detailed in the screenshot below.

How to activate Cloud Shell in the GCP console

Cert-manager installation:

Once the cloud shell loads up you need to get credentials to connect to your running GKE cluster.

gcloud container clusters get-credentials <cluster_name> — zone <cluster_zone>

Create a namespace where cert-manager will live.

kubectl create ns cert-manager

Create a ClusterRoleBinding to give yourself permissions to deploy cert-manager on the cluster

kubectl create clusterrolebinding cluster-admin-binding — clusterrole=cluster-admin — user=$(gcloud config get-value core/account)

Install the latest cert-manager into your cluster. (to find out the latest release of cert-manager go to the cert-manager Releases Page)

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/<release>/cert-manager.yaml

After a few minutes you should see it running on your cluster (yours will have different names)

kubectl get pods --namespace cert-manager
NAME                                     READY STATUS RESTARTS AGE
cert-manager-8454bdfcbb-grgqt             1/1  Running    0   105m
cert-manager-cainjector-56b88d57c7-t2pvf  1/1  Running    0   105m
cert-manager-webhook-5c49f9dc98–994mt     1/1  Running    0   105m

Create the DNS zone if not created yet:

For this guide I created a new zone for my domain in Google Cloud DNS. It is possible that you already have a zone somewhere for your domain and if so you can skip this part. You need a DNS zone to validate you are the owner of the domain for certificate verification. Getting wildcard certificates from Let’s Encrypt requires DNS validation.

To create a public zone on Cloud DNS follow this guide.

Create a GCP service account and download the key:

(You only need to do this if you are using Google Cloud DNS for validation)

You need to create a GCP service account that has access to add a TXT record to your DNS zone. The ClusterIssuer will use it to access Cloud DNS from the K8s cluster and automatically create a TXT record in order to validate you own the domain.

# put your project name into an environment variable
export PROJECT_NAME=[YOUR_PROJECT_NAME]
# create service account
gcloud projects add-iam-policy-binding $PROJECT_NAME --member=serviceAccount:svc-cert-manager@$PROJECT_NAME.iam.gserviceaccount.com --role=roles/dns.admin
# create and download service account key
gcloud iam service-accounts keys create ./credentials.json --iam-account=k8s-cert-manager@$PROJECT_NAME.iam.gserviceaccount.com --project=$PROJECT_NAME
# give the service account dns admin permissions
gcloud projects add-iam-policy-binding $PROJECT_NAME   --member=serviceAccount:k8s-cert-manager@$PROJECT_NAME.iam.gserviceaccount.com   --role=roles/dns.admin

Note: The use of the dns.admin role in this example role is for convenience. If you want to ensure cert-manager runs under a least privilege service account, you will need to create a custom role with the following permissions:

• dns.resourceRecordSets.*
• dns.changes.*
• dns.managedZones.list

Create a kubernetes secret from your newly created service account credentials:

I am using Google Cloud DNS to validate my certs but if you are using Cloudflare or Route53 or something else you need to follow the instructions to add API tokens to secrets instead of the service account.

This secret is used by the ClusterIssuer to add a TXT record to Google Cloud DNS for cert validation.

kubectl create secret generic dns-service-account \
--from-file=./credentials.json \
--namespace=cert-manager

Create the ClusterIssuer:

The ClusterIssuer is the part of cert-manager that tells cert-manager where to get the cert. For this we are using Let’s Encrypt. It also contains the information about what DNS provider you are using and how to authenticate to it. In this example we are using Cloud DNS but there are other solvers that can be plugged into this including Cloudflare and Route53.

cat <<EOF > clusterissuer-letsencrypt.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
  namespace: cert-manager
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: <your_email@example.com>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the private key.
      # Do not manually create secret, it will be created for you.
      name: letsencrypt
    # Add a challenge solver, ACME DNS-01 as required for wildcards
    solvers:
    - dns01:
        # Google Cloud DNS - change if using other provider
        cloudDNS:
          # Secret from the google service account key
          serviceAccountSecretRef:
            name: dns-service-account
            key: credentials.json
          # The project in which to update the DNS zone
          project: <your project id>
EOF

Now apply the ClusterIssuer

kubectl apply -f clusterissuer-letsencrypt.yaml

And you can check and see if it successfully registered you to let’s encrypt by describing.

kubectl describe clusterissuer letsencrypt
Status:
  Acme:
    Last Registered Email:  your_email@example.com
    Uri:
https://acme-v02.api.letsencrypt.org/acme/acct/111822117
  Conditions:
    Last Transition Time:  2021-02-06T00:44:25Z
  Message: The ACME account was registered with the ACME server
  Reason:                ACMEAccountRegistered
  Status:                True
  Type:                  Ready
  Events:                <none>

Create your deployment and service:

Now you will create your deployment and service for your application.

For this example I deploy a simple hello-app

kubectl create deployment hello-server --image=gcr.io/google-samples/hello-app:1.0

And expose a service of type NodePort

kubectl expose deployment hello-server --type NodePort --port 80 --target-port 8080

Create your GKE Ingress and get your managed wildcard cert at the same time:

The best part of cert-manager is it automatically adds a TXT record to your DNS and validates and issues a cert from Let’s Encrypt when you create an Ingress resource in GKE. It also auto-renews the certificate for you. Wildcard domain certificates (those covering *.yourdomain.com) can only be requested using DNS validation.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    # add an annotation indicating the issuer to use.
    cert-manager.io/cluster-issuer: letsencrypt
  # the name of your ingress
  name: myIngress
spec:
  rules:
  - host: "*.example.com"
    http:
      paths:
      - backend:
          serviceName: hello-server
          servicePort: 80
        path: /
  tls: # placing a host in the TLS config will create a cert
  - hosts:
    - "*.example.com"
    secretName: example-cert #created cert gets put into this secret

It can take over an hour to get your cert. To check the status of the certificate do

kubectl get cert
NAME                    READY   SECRET                  AGE
example-cert            True    example-cert          5h50m

If READY shows False you probably have to wait longer. You can check the status of the cert process for errors by checking the logs of the cert-manager container.

Get the name of the cert-manager pod

kubectl get pods -n cert-manager
NAME                              READY   STATUS    RESTARTS   AGE
cert-manager-8454bdfcbb-grgqt     1/1     Running   0          8h

Check the logs of the cert-manager pod

kubectl logs cert-manager-8454bdfcbb-grgqt -n cert-manager
I0206 00:47:01.585423       1 conditions.go:162] Found status change for Certificate "example-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-02-06 00:47:01.585413459 +0000 UTC m=+11290.701379832

You will see errors here if something went wrong. You can also check your DNS provider to see if the TXT record got inserted. If it did then the validation is still going and you need to wait a bit longer.

After a while you should get a public IP from the Load Balancer

kubectl get ingress
NAME            HOSTS             ADDRESS          PORTS     AGE
myIngress    *.example.com      <PUBLIC_IP>       80, 443   5h56m

Change your DNS to point to the new Public IP created by the Ingress

You can then change your DNS to point to this new IP address.

You might want to edit your local hosts file first to make sure things work okay before moving your DNS over.