Upgrading Cert-Manager
Rancher is compatible with the API version cert-manager.io/v1 and was last tested with cert-manager version v1.13.1.
Rancher uses cert-manager to automatically generate and renew TLS certificates for HA deployments of Rancher. As of Fall 2019, three important changes to cert-manager are set to occur that you need to take action on if you have an HA deployment of Rancher:
- Let's Encrypt will be blocking cert-manager instances older than 0.8.0 starting November 1st 2019.
- Cert-manager is deprecating and replacing the certificate.spec.acme.solvers field. This change has no exact deadline.
- Cert-manager is deprecating
v1alpha1
API and replacing its API group
To address these changes, this guide will do two things:
- Document the procedure for upgrading cert-manager
- Explain the cert-manager API changes and link to cert-manager's official documentation for migrating your data
If you are upgrading cert-manager to the latest version from a version older than 1.5, follow the steps in Option C below to do so. Note that you do not need to reinstall Rancher to perform this upgrade.
Upgrade Cert-Manager
The namespace used in these instructions depends on the namespace cert-manager is currently installed in. If it is in kube-system use that in the instructions below. You can verify by running kubectl get pods --all-namespaces
and checking which namespace the cert-manager-* pods are listed in. Do not change the namespace cert-manager is running in or this can cause issues.
In order to upgrade cert-manager, follow these instructions:
Option A: Upgrade cert-manager with Internet Access
Click to expand
Back up existing resources as a precaution
kubectl get -o yaml --all-namespaces \
issuer,clusterissuer,certificates,certificaterequests > cert-manager-backup.yamlImportant:If you are upgrading from a version older than 0.11.0, Update the apiVersion on all your backed up resources from
certmanager.k8s.io/v1alpha1
tocert-manager.io/v1alpha2
. If you use any cert-manager annotations on any of your other resources, you will need to update them to reflect the new API group. For details, refer to the documentation on additional annotation changes.helm uninstall cert-manager
Delete the CustomResourceDefinition using the link to the version vX.Y.Z you installed
kubectl delete -f https://github.com/cert-manager/cert-manager/releases/download/vX.Y.Z/cert-manager.crds.yaml
Install the CustomResourceDefinition resources separately
kubectl apply --validate=false -f https://github.com/cert-manager/cert-manager/releases/download/vX.Y.Z/cert-manager.crds.yaml
noteIf you are running Kubernetes v1.15 or below, you will need to add the
--validate=false
flag to yourkubectl apply
command above. Otherwise, you will receive a validation error relating to thex-kubernetes-preserve-unknown-fields
field in cert-manager’s CustomResourceDefinition resources. This is a benign error and occurs due to the way kubectl performs resource validation.Create the namespace for cert-manager if needed
kubectl create namespace cert-manager
Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io
Update your local Helm chart repository cache
helm repo update
Install the new version of cert-manager
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-managerkubectl apply -f cert-manager-backup.yaml
Option B: Upgrade cert-manager in an Air-Gapped Environment
Click to expand
Prerequisites
Before you can perform the upgrade, you must prepare your air gapped environment by adding the necessary container images to your private registry and downloading or rendering the required Kubernetes manifest files.
Follow the guide to Prepare your Private Registry with the images needed for the upgrade.
From a system connected to the internet, add the cert-manager repo to Helm
helm repo add jetstack https://charts.jetstack.io
helm repo updateFetch the latest cert-manager chart available from the Helm chart repository.
helm fetch jetstack/cert-manager
Render the cert manager template with the options you would like to use to install the chart. Remember to set the
image.repository
option to pull the image from your private registry. This will create acert-manager
directory with the Kubernetes manifest files.The Helm 3 command is as follows:
helm template cert-manager ./cert-manager-v0.12.0.tgz --output-dir . \
--namespace cert-manager \
--set image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-controller
--set webhook.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-webhook
--set cainjector.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-cainjectordangerHelm v2 support is deprecated as of the Rancher v2.7 line and will be removed in Rancher v2.9.
The Helm 2 command is as follows:
helm template ./cert-manager-v0.12.0.tgz --output-dir . \
--name cert-manager --namespace cert-manager \
--set image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-controller
--set webhook.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-webhook
--set cainjector.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-cainjectorDownload the required CRD file for cert-manager (old and new)
curl -L -o cert-manager-crd.yaml https://raw.githubusercontent.com/cert-manager/cert-manager/release-0.12/deploy/manifests/00-crds.yaml
curl -L -o cert-manager/cert-manager-crd-old.yaml https://raw.githubusercontent.com/cert-manager/cert-manager/release-X.Y/deploy/manifests/00-crds.yaml
Install cert-manager
Back up existing resources as a precaution
kubectl get -o yaml --all-namespaces \
issuer,clusterissuer,certificates,certificaterequests > cert-manager-backup.yamlImportant:If you are upgrading from a version older than 0.11.0, Update the apiVersion on all your backed up resources from
certmanager.k8s.io/v1alpha1
tocert-manager.io/v1alpha2
. If you use any cert-manager annotations on any of your other resources, you will need to update them to reflect the new API group. For details, refer to the documentation on additional annotation changes.Delete the existing cert-manager installation
kubectl -n cert-manager \
delete deployment,sa,clusterrole,clusterrolebinding \
-l 'app=cert-manager' -l 'chart=cert-manager-v0.5.2'Delete the CustomResourceDefinition using the link to the version vX.Y you installed
kubectl delete -f cert-manager/cert-manager-crd-old.yaml
Install the CustomResourceDefinition resources separately
kubectl apply -f cert-manager/cert-manager-crd.yaml
Important:If you are running Kubernetes v1.15 or below, you will need to add the
--validate=false
flag to yourkubectl apply
command above. Otherwise, you will receive a validation error relating to thex-kubernetes-preserve-unknown-fields
field in cert-manager’s CustomResourceDefinition resources. This is a benign error and occurs due to the way kubectl performs resource validation.Create the namespace for cert-manager
kubectl create namespace cert-manager
Install cert-manager
kubectl -n cert-manager apply -R -f ./cert-manager
kubectl apply -f cert-manager-backup.yaml
Option C: Upgrade cert-manager from Versions 1.5 and Below
Click to expand
Previously, in order to upgrade cert-manager from an older version, an uninstall and reinstall of Rancher was recommended. Using the method below, you may upgrade cert-manager without those additional steps in order to better preserve your production environment:
Install
cmctl
, the cert-manager CLI tool, using the installation guide.Ensure that any cert-manager custom resources that may have been stored in etcd at a deprecated API version get migrated to v1:
cmctl upgrade migrate-api-version
Refer to the API version migration docs for more information. Please also see the docs to upgrade from 1.5 to 1.6 and the docs to upgrade from 1.6. to 1.7 if needed.
Upgrade cert-manager to v1.7.1 with a normal
helm upgrade
. You may go directly from version 1.5 to 1.7 if desired.Follow the Helm tutorial to update the API version of a release manifest. The chart release name is
release_name=rancher
and the release namespace isrelease_namespace=cattle-system
.In the decoded file, search for
cert-manager.io/v1beta1
and replace it withcert-manager.io/v1
.Upgrade Rancher normally with
helm upgrade
.
Verify the Deployment
Once you’ve installed cert-manager, you can verify it is deployed correctly by checking the kube-system namespace for running pods:
kubectl get pods --namespace cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-5c6866597-zw7kh 1/1 Running 0 2m
cert-manager-cainjector-577f6d9fd7-tr77l 1/1 Running 0 2m
cert-manager-webhook-787858fcdb-nlzsq 1/1 Running 0 2m
Cert-Manager API change and data migration
Rancher now supports cert-manager versions 1.6.2 and 1.7.1. We recommend v1.7.x because v 1.6.x will reach end-of-life on March 30, 2022. To read more, see the cert-manager docs. For instructions on upgrading cert-manager from version 1.5 to 1.6, see the upstream cert-manager documentation here. For instructions on upgrading cert-manager from version 1.6 to 1.7, see the upstream cert-manager documentation here.
Cert-manager has deprecated the use of the certificate.spec.acme.solvers
field and will drop support for it completely in an upcoming release.
Per the cert-manager documentation, a new format for configuring ACME certificate resources was introduced in v0.8. Specifically, the challenge solver configuration field was moved. Both the old format and new are supported as of v0.9, but support for the old format will be dropped in an upcoming release of cert-manager. The cert-manager documentation strongly recommends that after upgrading you update your ACME Issuer and Certificate resources to the new format.
Details about the change and migration instructions can be found in the cert-manager v0.7 to v0.8 upgrade instructions.
The v0.11 release marks the removal of the v1alpha1 API that was used in previous versions of cert-manager, as well as our API group changing to be cert-manager.io instead of certmanager.k8s.io.
We have also removed support for the old configuration format that was deprecated in the v0.8 release. This means you must transition to using the new solvers style configuration format for your ACME issuers before upgrading to v0.11. For more information, see the upgrading to v0.8 guide.
Details about the change and migration instructions can be found in the cert-manager v0.10 to v0.11 upgrade instructions.
More info about cert-manager upgrade information.