Updating the Rancher Certificate
Updating a Private CA Certificate
Follow these steps to rotate an SSL certificate and private CA used by Rancher installed on a Kubernetes cluster, or migrate to an SSL certificate signed by a private CA.
A summary of the steps is as follows:
- Create or update the
tls-rancher-ingress
Kubernetes secret object with the new certificate and private key. - Create or update the
tls-ca
Kubernetes secret object with the root CA certificate (only required when using a private CA). - Update the Rancher installation using the Helm CLI.
- Reconfigure the Rancher agents to trust the new CA certificate.
- Select Force Update of Fleet clusters to connect fleet-agent to Rancher.
The details of these instructions are below.
1. Create/update the certificate secret object
First, concatenate the server certificate followed by any intermediate certificate(s) to a file named tls.crt
and provide the corresponding certificate key in a file named tls.key
.
Use the following command to create the tls-rancher-ingress
secret object in the Rancher (local) management cluster:
kubectl -n cattle-system create secret tls tls-rancher-ingress \
--cert=tls.crt \
--key=tls.key
Alternatively, to update an existing tls-rancher-ingress
secret:
kubectl -n cattle-system create secret tls tls-rancher-ingress \
--cert=tls.crt \
--key=tls.key \
--dry-run --save-config -o yaml | kubectl apply -f -
2. Create/update the CA certificate secret object
If the new certificate was signed by a private CA, you will need to copy the corresponding root CA certificate into a file named cacerts.pem
and create or update the tls-ca
secret in the cattle-system
namespace. If the certificate was signed by an intermediate CA, then the cacerts.pem
must contain both the intermediate and root CA certificates (in this order).
To create the initial tls-ca
secret:
kubectl -n cattle-system create secret generic tls-ca \
--from-file=cacerts.pem
To update an existing tls-ca
secret:
kubectl -n cattle-system create secret generic tls-ca \
--from-file=cacerts.pem \
--dry-run --save-config -o yaml | kubectl apply -f -
3. Reconfigure the Rancher deployment
If the certificate source remains the same (for example, secret
), please follow the steps in Step 3a.
However, if the certificate source is changing (for example, letsEncrypt
to secret
), follow the steps in 3b.
3a. Redeploy the Rancher pods
This step is required when the certificate source remains the same, but the CA certificate is being updated.
In this scenario a redeploy of the Rancher pods is needed, this is because the tls-ca
secret is read by the Rancher pods when starting.
The command below can be used to redeploy the Rancher pods:
kubectl rollout restart deploy/rancher -n cattle-system
When the change is completed, navigate to https://<RANCHER_SERVER_URL>/v3/settings/cacerts
to verify that the value matches the CA certificate written in the tls-ca
secret earlier. The CA cacerts
value may not update until all of the redeployed Rancher pods start.
3b. Update the Helm values for Rancher
This step is required if the certificate source is changing. If Rancher was previously configured to use the default self-signed certificate (ingress.tls.source=rancher
) or Let's Encrypt (ingress.tls.source=letsEncrypt
), and is now using a certificate signed by a private CA (ingress.tls.source=secret
).
The below steps update the Helm values for the Rancher chart, so the Rancher pods and ingress are reconfigured to use the new private CA certificate created in Step 1 & 2.
- Adjust the values that were used during initial installation, store the current values with:
helm get values rancher -n cattle-system -o yaml > values.yaml
- Retrieve the version string of the currently deployed Rancher chart to use below:
helm ls -n cattle-system
- Update the current Helm values in the
values.yaml
file to contain:ingress:
tls:
source: secret
privateCA: trueImportant:As the certificate is signed by a private CA, it is important to ensure
privateCA: true
is set in thevalues.yaml
file. - Upgrade the Helm application instance using the
values.yaml
file and the current chart version. The version must match to prevent an upgrade of Rancher.helm upgrade rancher rancher-stable/rancher \
--namespace cattle-system \
-f values.yaml \
--version <DEPLOYED_RANCHER_VERSION>
When the change is completed, navigate to https://<RANCHER_SERVER_URL>/v3/settings/cacerts
to verify that the value matches the CA certificate written in the tls-ca
secret earlier. The CA cacerts
value may not update until all Rancher pods start.
4. Reconfigure Rancher agents to trust the private CA
This section covers three methods to reconfigure Rancher agents to trust the private CA. This step is required if either of the following is true:
- Rancher was previously configured to use the Rancher self-signed certificate (
ingress.tls.source=rancher
) or with a Let's Encrypt issued certificate (ingress.tls.source=letsEncrypt
) - The certificate was signed by a different private CA
Why is this step required?
When Rancher is configured with a certificate signed by a private CA, the CA certificate chain is trusted by Rancher agent containers. Agents compare the checksum of the downloaded certificate against the CATTLE_CA_CHECKSUM
environment variable. This means that, when the private CA certificate used by Rancher has changed, the environment variable CATTLE_CA_CHECKSUM
must be updated accordingly.
Which method should I choose?
Method 1 is the easiest, but requires all clusters to be connected to Rancher after the certificates have been rotated. This is usually the case if the process is performed right after updating or redeploying the Rancher deployment (Step 3).
If the clusters have lost connection to Rancher but Authorized Cluster Endpoint (ACE) is enabled on all clusters, then go with method 2.
Method 3 can be used as a fallback if method 1 and 2 are not possible.
Method 1: Force a redeploy of the Rancher agents
For each downstream cluster run the following command using the Kubeconfig file of the Rancher (local) management cluster.
kubectl annotate clusters.management.cattle.io <CLUSTER_ID> io.cattle.agent.force.deploy=true
Locate the cluster ID (c-xxxxx) for the downstream cluster, this can be seen in the browser URL bar when viewing the cluster in the Rancher UI, under Cluster Management.
This command will cause the agent manifest to be reapplied with the checksum of the new certificate.
Method 2: Manually update the checksum environment variable
Manually patch the agent Kubernetes objects by updating the CATTLE_CA_CHECKSUM
environment variable to the value matching the checksum of the new CA certificate. Generate the new checksum value like so:
curl -k -s -fL <RANCHER_SERVER_URL>/v3/settings/cacerts | jq -r .value | sha256sum | awk '{print $1}'
Using a Kubeconfig for each downstream cluster update the environment variable for the two agent deployments. If the ACE is enabled for the cluster, the kubectl context can be adjusted to connect directly to the downstream cluster.
kubectl edit -n cattle-system ds/cattle-node-agent
kubectl edit -n cattle-system deployment/cattle-cluster-agent
Method 3: Manually redeploy the Rancher agents
With this method the Rancher agents are reapplied by running a set of commands on a control plane node of each downstream cluster.
Repeat the below steps for each downstream cluster:
Retrieve the agent registration kubectl command:
- Locate the cluster ID (c-xxxxx) for the downstream cluster, this can be seen in the URL when viewing the cluster in the Rancher UI under Cluster Management
- Add the Rancher server URL and cluster ID to the following URL:
https://<RANCHER_SERVER_URL>/v3/clusterregistrationtokens?clusterId=<CLUSTER_ID>
- Copy the command from the
insecureCommand
field, this command is used because a private CA is un use
Run the kubectl command from the previous step using a kubeconfig for the downstream cluster with one of the following methods:
- If the ACE is enabled for the cluster, the context can be adjusted to connect directly to the downstream cluster
- Alternatively, SSH into the control plane node:
- RKE: Use the steps in the document here to generate a kubeconfig
- RKE2/K3s: Use the kubeconfig populated during installation
5. Force Update Fleet clusters to reconnect the fleet-agent to Rancher
Select 'Force Update' for the clusters within the Continuous Delivery view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher.
Why is this step required?
Fleet agents in Rancher managed clusters store a kubeconfig that is used to connect to Rancher. The kubeconfig contains a certificate-authority-data
field containing the CA for the certificate used by Rancher. When changing the CA, this block needs to be updated to allow the fleet-agent to trust the certificate used by Rancher.
Updating from a Private CA Certificate to a Public CA Certificate
Follow these steps to perform the opposite procedure as shown above, to change from a certificate issued by a private CA, to a public or self-signed CA.
1. Create/update the certificate secret object
First, concatenate the server certificate followed by any intermediate certificate(s) to a file named tls.crt
and provide the corresponding certificate key in a file named tls.key
.
Use the following command to create the tls-rancher-ingress
secret object in the Rancher (local) management cluster:
kubectl -n cattle-system create secret tls tls-rancher-ingress \
--cert=tls.crt \
--key=tls.key
Alternatively, to update an existing tls-rancher-ingress
secret:
kubectl -n cattle-system create secret tls tls-rancher-ingress \
--cert=tls.crt \
--key=tls.key \
--dry-run --save-config -o yaml | kubectl apply -f -
2. Delete the CA certificate secret object
You will delete the tls-ca
secret in the cattle-system
namespace as it is no longer needed. You may also optionally save a copy of the tls-ca
secret if desired.
To save the existing tls-ca
secret:
kubectl -n cattle-system get secret tls-ca -o yaml > tls-ca.yaml
To delete the existing tls-ca
secret:
kubectl -n cattle-system delete secret tls-ca
3. Reconfigure the Rancher deployment
This step is required if the certificate source is changing. In this scenario it's likely only changing because Rancher was previously configured to use the default self-signed certificate (ingress.tls.source=rancher
).
The below steps update the Helm values for the Rancher chart, so the Rancher pods and ingress are reconfigured to use the new certificate created in Step 1.
- Adjust the values that were used during initial installation, store the current values with:
helm get values rancher -n cattle-system -o yaml > values.yaml
- Also get the version string of the currently deployed Rancher chart:
helm ls -n cattle-system
- Update the current Helm values in the
values.yaml
file:- As a private CA is no longer being used, remove the
privateCA: true
field, or set this tofalse
- Adjust the
ingress.tls.source
field as necessary. Please refer to the chart options for more details. Here are some examples:- If using a public CA continue with a value of:
secret
- If using Let's Encrypt update the value to:
letsEncrypt
- If using a public CA continue with a value of:
- As a private CA is no longer being used, remove the
- Update the Helm values for the Rancher chart using the
values.yaml
file, and the current chart version to prevent an upgrade:helm upgrade rancher rancher-stable/rancher \
--namespace cattle-system \
-f values.yaml \
--version <DEPLOYED_RANCHER_VERSION>
4. Reconfigure Rancher agents for the non-private/common certificate
As a private CA is no longer being used, the CATTLE_CA_CHECKSUM
environment variable on the downstream cluster agents should be removed or set to "" (an empty string).
5. Force Update Fleet clusters to reconnect the fleet-agent to Rancher
Select 'Force Update' for the clusters within the Continuous Delivery view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher.
Why is this step required?
Fleet agents in Rancher managed clusters store a kubeconfig that is used to connect to Rancher. The kubeconfig contains a certificate-authority-data
field containing the CA for the certificate used by Rancher. When changing the CA, this block needs to be updated to allow the fleet-agent to trust the certificate used by Rancher.