ACME Certificates with LetsEncrypt and UltraDNS integration on Kubernetes

ACME — abbreviated for Automated Certificate Management Environment (https://tools.ietf.org/html/rfc8555) is the preferred way for managing (issue, renew, revoke) certificates for various IT infrastructure components and providers such as LetsEncrypt (https://letsencrypt.org/) make our life very easy — at least they made mine easier! ACME providers issue certificates based on a solution to either a “DNS challenge” or an “HTTP Challenge” that aims to verify the authenticity of our domains.

Aim — Our domain was managed using the UltraDNS provider and we needed an automated way of managing certificates in our Kubernetes environment.

Context —UltraDNS (https://portal.ultradns.com) was not very friendly for use in a Kubernetes environment for ACME(at the point of writing)and did not provide a first-class integration with software such as Cert-Manager (https://cert-manager.io/). Although the Math-Nao project(https://github.com/math-nao/certs) using ACME.sh project(https://github.com/acmesh-official/acme.sh) and its UltraDNS integration (https://github.com/acmesh-official/acme.sh/wiki/dnsapi#70-use-ultradns-api) looked promising to achieve our aim, the presence of a few bugs (especially around pagination of a zone) in the UltraDNS integration piece meant that we could not use Math-Nao project directly. Instead, we had to code a series of Github Actions that did the work for us.

This article is documenting what we did, in hopes of helping others who might face a similar issue!

Strategy — We used ACME.sh (https://github.com/acmesh-official/acme.sh) in the manual mode and called the REST API provided by UltraDNS to achieve our aim. We also set up the Certificate Renewal as a Github Action, instead of a Kubernetes Cron Job, and eased our burden around the management of secrets in Kubernetes! ACME.sh uses LetsEncrypt provider behind the scenes by default.

Step 1: Download ACME.sh and install it. We followed the steps in https://github.com/acmesh-official/acme.sh#1-how-to-install

In our case, the installation installed the acme.sh shell script in ~/.acme.sh/acme.sh path

Step 2: Issued a certificate request using ACME.sh using the manual mode

~/.acme.sh/acme.sh --issue --dns --yes-I-know-dns-manual-mode-enough-go-ahead-please -d *.mydomain.com > /temp/output1.txt

Here mydomain.com is the domain that is being managed by UltraDNS and we are trying to get a wildcard certificate for that domain. We are also redirecting the output of the command to a temporary file /temp/output1.txt

This results in ACME.sh giving us a token value that should be used to solve the DNS challenge. The output is very similar to:

Add the following txt record:
Domain:_acme-challenge.mydomain.com
Txt value:678DbjYfTExAYeDsABCDeuTo18KBzwvTEjUnSwd32-c

Step 3: The next step is to extract this token value so that we could use this in a REST call to the UltraDNS

cat /temp/output1.txt | grep 'TXT value' | awk -F 'TXT value:' '{print $2}' | awk '{$1=$1;print}'  | sed "s/'//g"

This would print out the token value alone. We could store this value in a shell variable.

Step 4: Authenticate with UltraDNS to get an Authentication token. Assuming that you have stored the UltraDNS username in $ULTRA_USR and an UltraDNS password in the $ULTRA_PWD, and the UltraDNS REST API server (Default = https://restapi.ultradns.com/v2 ) in $ultraDnsAddress shell variables, the following command gives us a JSON containing an Access Token

curl -s -w "%{http_code}" -X POST -d "grant_type=password&username=${ULTRA_USR}&password=${ULTRA_PWD}" $ultraDnsAddress/authorization/token -o /temp/output2.txt

The JSON response is saved to /temp/output.txt file. You can extract the “accessToken” from this JSON using the following command and store this in a shell variable.

cat /temp/output2.txt | jq -r '.accessToken'

Step 5: Call the UltraDNS API to add the TXT value containing the DNS Challenge from ACME

Assuming that the token that we obtained from ACME (678DbjYfTExAYeDsABCDeuTo18KBzwvTEjUnSwd32-c) was stored in a shell variable $token_from_acme, we can construct the JSON payload to the REST API using the following.

jsonPayload='{"ttl":300,"rdata":["'"${token_from_acme}"'"]}'

and call the REST API using

curl -s -w "%{http_code}" -X POST -H "Content-Type: application/json"  -H "Authorization: Bearer $ultraDnsAuthToken"  -d "$jsonPayload" $ultraDnsAddress/zones/mydomain.com/rrsets/TXT/_acme-challenge.mydomain.com -o /temp/output3.txt

In the above command, we are firing a REST call to add a TXT entry for “_acme-challenge.mydomain.com” with a value obtained in the token, with a TTL value of 300 seconds in the “mydomain.com” zone within UltraDNS.

The expected response code for the REST call is 201

Step 6: Trigger the DNS validation by ACME.sh and issuance of the certificate.

~/.acme.sh/acme.sh --renew --dns --yes-I-know-dns-manual-mode-enough-go-ahead-please -d mydomain.com

ACME.sh and by extension LetsEncrypt now fetches the TXT entry for _acme-challenge.mydomain.com and then verifies the token value. If the token value matches, then the certificate is issued. When the certificate is issued, the ACME.sh also saves the CA File, the Certificate File, the Key file, etc, in the following paths. The paths may slightly vary, but you get the idea!

ACME_CA_FILE="/root/.acme.sh/mydomain.com/ca.cer"
ACME_CERT_FILE="/root/.acme.sh/mydomain.com/mydomain.com.cer"
ACME_FULLCHAIN_FILE="/root/.acme.sh/mydomain.com/fullchain.cer"
ACME_KEY_FILE="/root/.acme.sh/mydomain.com/mydomain.com.key"

Step 7: Create the Kubernetes Secret

We can then use the files generated in the previous step to create the Kubernetes Secret. Prepare a YAML file with the help of the following shell script. Customize the Shell variables according to your needs!

CERTS_SECRET_NAME=mysecret
CERT_NAMESPACE=mynamespace
ACME_CA_FILE="/root/.acme.sh/mydomain.com/ca.cer"
ACME_CERT_FILE="/root/.acme.sh/mydomain.com/mydomain.com.cer"
ACME_FULLCHAIN_FILE="/root/.acme.sh/mydomain.com/fullchain.cer"
ACME_KEY_FILE="/root/.acme.sh/mydomain.com/mydomain.com.key"
prepare_file() {
  local FILE="$1"
  cat "${FILE}" | base64 -w 0
}
SECRET_JSON=$(echo '{}')
SECRET_JSON=$(echo ${SECRET_JSON} | jq --arg apiVersion "v1" '. + {apiVersion: $apiVersion}')
SECRET_JSON=$(echo ${SECRET_JSON} | jq --arg kind "Secret" '. + {kind: $kind}')
SECRET_JSON=$(echo ${SECRET_JSON} | jq --arg name "${CERTS_SECRET_NAME}" '. + {metadata: { name: $name }}')
SECRET_JSON=$(echo ${SECRET_JSON} | jq --arg ns "${CERT_NAMESPACE}" '. * {metadata: { namespace: $ns }}')
SECRET_JSON=$(echo ${SECRET_JSON} | jq --arg type "kubernetes.io/tls" '. + {type: $type}')
SECRET_JSON=$(echo ${SECRET_JSON} | jq '. + {data: {}}')
SECRET_JSON=$(echo ${SECRET_JSON} | jq --arg cacert "$(prepare_file "${ACME_CA_FILE}")" '. * {data: {"ca.crt": $cacert}}')
SECRET_JSON=$(echo ${SECRET_JSON} | jq --arg tlscert "$(prepare_file "${ACME_FULLCHAIN_FILE}")" '. * {data: {"tls.crt": $tlscert}}')
SECRET_JSON=$(echo ${SECRET_JSON} | jq --arg tlskey "$(prepare_file "${ACME_KEY_FILE}")" '. * {data: {"tls.key": $tlskey}}')
echo -e "${SECRET_JSON}" > "secrets.yaml"

This will produce a secrets.yaml file like below

Of course, the values of “ca.crt”, “tls.crt” and “tls.key” in the above secrets.yaml will be base64 encoded values for your files that you received in the issue step.

You can then use kubectl to apply this secret

kubectl apply -f secrets.yaml

This will create the Kubernetes Secret “mysecret” in the “mynamespace” namespace. This secret can be used by Ingress in your cluster to secure your service.

Step 8: Clean up UltraDNS

Once the challenge is done, it is also important to clean up the TXT record that we just inserted.

curl -s -w "%{http_code}" -X DELETE -H "Content-Type: application/json"  -H "Authorization: Bearer $ultraDnsAuthToken"  -d "$jsonPayload" $ultraDnsAddress/zones/mydomain.com/rrsets/TXT/_acme-challenge.mydomain.com -o /temp/output4.txt

The expected response code is 204.

This whole operation was automated using a shell script and was driven using a Github Action. You can also set up a Github Action for Renewal of your certificate based on the expiration date and run the exact same steps to store the renewed certificate in the same secret.

To check whether the certificate will expire within the next 15 days, use —

renewNumDaysBeforeExpiry=15 
secondsPerDay=86400
checkEndSeconds=`expr $renewNumDaysBeforeExpiry \* $secondsPerDay`
kubectl get secret mysecret -n mynamespace -o "jsonpath={.data['tls\.crt']}" | base64 -d | openssl x509 -checkend $checkEndSeconds

Happy Coding!