设置 Amazon 云提供商
使用 Amazon
云提供商时,你可以利用以下功能:
- 负载均衡器:在 Port Mapping 中选择
Layer-4 Load Balancer
或使用type: LoadBalancer
启动Service
时,启动 AWS 弹性负载均衡器 (ELB)。 - 持久卷:允许你将 AWS 弹性块存储 (EBS) 用于持久卷。
有关 Amazon 云提供商的所有信息,请参阅 cloud-provider-aws 自述文件。
要设置 Amazon 云提供商:
从 Kubernetes 1.23 开始,你必须停用 CSIMigrationAWS
特性开关才能使用树内 AWS 云提供商。为此,你可以在高级集群配置中将 feature-gates=CSIMigrationAWS=false
设置为集群 Kubelet、Controller Manager、API Server 和 Scheduler 的附加参数。
1. 创建 IAM 角色并附加到实例
添加到集群的所有节点都必须能够与 EC2 交互,以便它们可以创建和删除资源。你可以使用附加到实例的 IAM 角色来启用交互。请参阅 Amazon 文档:创建 IAM 角色 来创建 IAM 角色。有两个示例策略:
- 第一个策略适用于具有
controlplane
角色的节点。这些节点必须能够创建/删除 EC2 资源。以下 IAM 策略是一个示例,请根据你的实际用例移除不需要的权限。 - 第二个策略适用于具有
etcd
或worker
角色的节点。这些节点只需能够从 EC2 检索信息。
在创建 Amazon EC2 集群时,你必须在创建节点模板时填写创建的 IAM 角色的 IAM Instance Profile Name(不是 ARN)。
创建自定义集群时,你必须手动将 IAM 角色附加到实例。
具有 controlplane
角色的节点的 IAM 策略:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeTags",
"ec2:DescribeInstances",
"ec2:DescribeRegions",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVolumes",
"ec2:CreateSecurityGroup",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:ModifyInstanceAttribute",
"ec2:ModifyVolume",
"ec2:AttachVolume",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateRoute",
"ec2:DeleteRoute",
"ec2:DeleteSecurityGroup",
"ec2:DeleteVolume",
"ec2:DetachVolume",
"ec2:RevokeSecurityGroupIngress",
"ec2:DescribeVpcs",
"elasticloadbalancing:AddTags",
"elasticloadbalancing:AttachLoadBalancerToSubnets",
"elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:CreateLoadBalancerPolicy",
"elasticloadbalancing:CreateLoadBalancerListeners",
"elasticloadbalancing:ConfigureHealthCheck",
"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:DeleteLoadBalancerListeners",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:DetachLoadBalancerFromSubnets",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer",
"elasticloadbalancing:AddTags",
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:CreateTargetGroup",
"elasticloadbalancing:DeleteListener",
"elasticloadbalancing:DeleteTargetGroup",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancerPolicies",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:ModifyListener",
"elasticloadbalancing:ModifyTargetGroup",
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
"iam:CreateServiceLinkedRole",
"kms:DescribeKey"
],
"Resource": [
"*"
]
}
]
}
具有 etcd
或 worker
角色的节点的 IAM 策略:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeRegions",
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:BatchGetImage"
],
"Resource": "*"
}
]
}
2. 创建 ClusterID
以下资源需要使用 ClusterID
进行标记:
- Nodes:Rancher 中添加的所有主机。
- Subnet:集群使用的子网。
- Security Group:用于你的集群的安全组。
不要标记多个安全组。创建弹性负载均衡器 (ELB) 时,标记多个组会产生错误。
创建 Amazon EC2 集群时,会自动为创建的节点配置 ClusterID
。其他资源仍然需要手动标记。
使用以下标签:
Key = kubernetes.io/cluster/CLUSTERID
Value = owned
CLUSTERID
可以是任何字符串,只要它在所有标签集中相同即可。
将标签的值设置为 owned
会通知集群带有该标签的所有资源都由该集群拥有和管理。如果你在集群之间共享资源,你可以将标签更改为:
Key = kubernetes.io/cluster/CLUSTERID
Value = shared
.
使用 Amazon Elastic Container Registry (ECR)
在将创建 IAM 角色并附加到实例中的 IAM 配置文件附加到实例时,kubelet 组件能够自动获取 ECR 凭证。使用低于 v1.15.0 的 Kubernetes 版本时,需要在集群中配置 Amazon 云提供商。从 Kubernetes 版本 v1.15.0 开始,kubelet 无需在集群中配置 Amazon 云提供商即可获取 ECR 凭证。
Using the Out-of-Tree AWS Cloud Provider
- RKE2
- RKE
Node name conventions and other prerequisites must be followed for the cloud provider to find the instance correctly.
Rancher managed RKE2/K3s clusters don't support configuring
providerID
. However, the engine will set the node name correctly if the following configuration is set on the provisioning cluster object:
spec:
rkeConfig:
machineGlobalConfig:
cloud-provider-name: aws
This option will be passed to the configuration of the various Kubernetes components that run on the node, and must be overridden per component to prevent the in-tree provider from running unintentionally:
Override on Etcd:
spec:
rkeConfig:
machineSelectorConfig:
- config:
kubelet-arg:
- cloud-provider=external
machineLabelSelector:
matchExpressions:
- key: rke.cattle.io/etcd-role
operator: In
values:
- 'true'
Override on Control Plane:
spec:
rkeConfig:
machineSelectorConfig:
- config:
disable-cloud-controller: true
kube-apiserver-arg:
- cloud-provider=external
kube-controller-manager-arg:
- cloud-provider=external
kubelet-arg:
- cloud-provider=external
machineLabelSelector:
matchExpressions:
- key: rke.cattle.io/control-plane-role
operator: In
values:
- 'true'
Override on Worker:
spec:
rkeConfig:
machineSelectorConfig:
- config:
kubelet-arg:
- cloud-provider=external
machineLabelSelector:
matchExpressions:
- key: rke.cattle.io/worker-role
operator: In
values:
- 'true'
Select
Amazon
if relying on the above mechanism to set the provider ID. Otherwise, select External (out-of-tree) cloud provider, which sets--cloud-provider=external
for Kubernetes components.Specify the
aws-cloud-controller-manager
Helm chart as an additional manifest to install:
spec:
rkeConfig:
additionalManifest: |-
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: aws-cloud-controller-manager
namespace: kube-system
spec:
chart: aws-cloud-controller-manager
repo: https://kubernetes.github.io/cloud-provider-aws
targetNamespace: kube-system
bootstrap: true
valuesContent: |-
hostNetworking: true
nodeSelector:
node-role.kubernetes.io/control-plane: "true"
args:
- --configure-cloud-routes=false
- --v=5
- --cloud-provider=aws
- Node name conventions and other prerequisites must be followed so that the cloud provider can find the instance. Rancher provisioned clusters don't support configuring
providerID
.
If you use IP-based naming, the nodes must be named after the instance followed by the regional domain name (ip-xxx-xxx-xxx-xxx.ec2.<region>.internal
). If you have a custom domain name set in the DHCP options, you must set --hostname-override
on kube-proxy
and kubelet
to match this naming convention.
To meet node naming conventions, Rancher allows setting useInstanceMetadataHostname
when the External Amazon
cloud provider is selected. Enabling useInstanceMetadataHostname
will query ec2 metadata service and set /hostname
as hostname-override
for kubelet
and kube-proxy
:
rancher_kubernetes_engine_config:
cloud_provider:
name: external-aws
useInstanceMetadataHostname: true
You must not enable useInstanceMetadataHostname
when setting custom values for hostname-override
for custom clusters. When you create a custom cluster, add --node-name
to the docker run
node registration command to set hostname-override
— for example, "$(hostname -f)"
. This can be done manually or by using Show Advanced Options in the Rancher UI to add Node Name.
- Select the cloud provider.
Selecting External Amazon (out-of-tree) sets --cloud-provider=external
and enables useInstanceMetadataHostname
. As mentioned in step 1, enabling useInstanceMetadataHostname
will query the EC2 metadata service and set http://169.254.169.254/latest/meta-data/hostname
as hostname-override
for kubelet
and kube-proxy
.
You must disable useInstanceMetadataHostname
when setting a custom node name for custom clusters via node-name
.
rancher_kubernetes_engine_config:
cloud_provider:
name: external-aws
useInstanceMetadataHostname: true/false
Existing clusters that use an External cloud provider will set --cloud-provider=external
for Kubernetes components but won't set the node name.
- Install the AWS cloud controller manager after the cluster finishes provisioning. Note that the cluster isn't successfully provisioned and nodes are still in an
uninitialized
state until you deploy the cloud controller manager. This can be done manually, or via Helm charts in UI.
Refer to the offical AWS upstream documentation for the cloud controller manager.
Helm Chart Installation from CLI
- RKE2
- RKE
Official upstream docs for Helm chart installation can be found on GitHub.
- Add the Helm repository:
helm repo add aws-cloud-controller-manager https://kubernetes.github.io/cloud-provider-aws
helm repo update
- Create a
values.yaml
file with the following contents to override the defaultvalues.yaml
:
# values.yaml
hostNetworking: true
tolerations:
- effect: NoSchedule
key: node.cloudprovider.kubernetes.io/uninitialized
value: 'true'
- effect: NoSchedule
value: 'true'
key: node-role.kubernetes.io/control-plane
nodeSelector:
node-role.kubernetes.io/control-plane: 'true'
args:
- --configure-cloud-routes=false
- --use-service-account-credentials=true
- --v=2
- --cloud-provider=aws
clusterRoleRules:
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- update
- apiGroups:
- ""
resources:
- nodes
verbs:
- '*'
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
- apiGroups:
- ""
resources:
- services
verbs:
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- services/status
verbs:
- list
- patch
- update
- watch
- apiGroups:
- ''
resources:
- serviceaccounts
verbs:
- create
- get
- apiGroups:
- ""
resources:
- persistentvolumes
verbs:
- get
- list
- update
- watch
- apiGroups:
- ""
resources:
- endpoints
verbs:
- create
- get
- list
- watch
- update
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
- get
- list
- watch
- update
- apiGroups:
- ""
resources:
- serviceaccounts/token
verbs:
- create
- Install the Helm chart:
helm upgrade --install aws-cloud-controller-manager aws-cloud-controller-manager/aws-cloud-controller-manager --values values.yaml
Verify that the Helm chart installed successfully:
helm status -n kube-system aws-cloud-controller-manager
- (Optional) Verify that the cloud controller manager update succeeded:
kubectl rollout status daemonset -n kube-system aws-cloud-controller-manager
Official upstream docs for Helm chart installation can be found on GitHub.
- Add the Helm repository:
helm repo add aws-cloud-controller-manager https://kubernetes.github.io/cloud-provider-aws
helm repo update
- Create a
values.yaml
file with the following contents, to override the defaultvalues.yaml
:
# values.yaml
hostNetworking: true
tolerations:
- effect: NoSchedule
key: node.cloudprovider.kubernetes.io/uninitialized
value: 'true'
- effect: NoSchedule
value: 'true'
key: node-role.kubernetes.io/controlplane
nodeSelector:
node-role.kubernetes.io/controlplane: 'true'
args:
- --configure-cloud-routes=false
- --use-service-account-credentials=true
- --v=2
- --cloud-provider=aws
clusterRoleRules:
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- update
- apiGroups:
- ""
resources:
- nodes
verbs:
- '*'
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
- apiGroups:
- ""
resources:
- services
verbs:
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- services/status
verbs:
- list
- patch
- update
- watch
- apiGroups:
- ''
resources:
- serviceaccounts
verbs:
- create
- get
- apiGroups:
- ""
resources:
- persistentvolumes
verbs:
- get
- list
- update
- watch
- apiGroups:
- ""
resources:
- endpoints
verbs:
- create
- get
- list
- watch
- update
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
- get
- list
- watch
- update
- apiGroups:
- ""
resources:
- serviceaccounts/token
verbs:
- create
- Install the Helm chart:
helm upgrade --install aws-cloud-controller-manager -n kube-system aws-cloud-controller-manager/aws-cloud-controller-manager --values values.yaml
Verify that the Helm chart installed successfully:
helm status -n kube-system aws-cloud-controller-manager
- If present, edit the Daemonset to remove the default node selector
node-role.kubernetes.io/control-plane: ""
:
kubectl edit daemonset aws-cloud-controller-manager -n kube-system
- (Optional) Verify that the cloud controller manager update succeeded:
kubectl rollout status daemonset -n kube-system aws-cloud-controller-manager
Helm Chart Installation from UI
- RKE2
- RKE
Click ☰, then select the name of the cluster from the left navigation.
Select Apps > Repositories.
Click the Create button.
Enter
https://kubernetes.github.io/cloud-provider-aws
in the Index URL field.Select Apps > Charts from the left navigation and install aws-cloud-controller-manager.
Select the namespace,
kube-system
, and enable Customize Helm options before install.Add the following container arguments:
- '--use-service-account-credentials=true'
- '--configure-cloud-routes=false'
- Add
get
toverbs
forserviceaccounts
resources inclusterRoleRules
. This allows the cloud controller manager to get service accounts upon startup.
- apiGroups:
- ''
resources:
- serviceaccounts
verbs:
- create
- get
- Rancher-provisioned RKE2 nodes are tainted
node-role.kubernetes.io/control-plane
. Update tolerations and the nodeSelector:
tolerations:
- effect: NoSchedule
key: node.cloudprovider.kubernetes.io/uninitialized
value: 'true'
- effect: NoSchedule
value: 'true'
key: node-role.kubernetes.io/control-plane
nodeSelector:
node-role.kubernetes.io/control-plane: 'true'
There's currently a known issue where nodeSelector can't be updated from the Rancher UI. Continue installing the chart and then edit the Daemonset manually to set the nodeSelector
:
nodeSelector:
node-role.kubernetes.io/control-plane: 'true'
- Install the chart and confirm that the Daemonset
aws-cloud-controller-manager
is running. Verifyaws-cloud-controller-manager
pods are running in target namespace (kube-system
unless modified in step 6).
Click ☰, then select the name of the cluster from the left navigation.
Select Apps > Repositories.
Click the Create button.
Enter
https://kubernetes.github.io/cloud-provider-aws
in the Index URL field.Select Apps > Charts from the left navigation and install aws-cloud-controller-manager.
Select the namespace,
kube-system
, and enable Customize Helm options before install.Add the following container arguments:
- '--use-service-account-credentials=true'
- '--configure-cloud-routes=false'
- Add
get
toverbs
forserviceaccounts
resources inclusterRoleRules
. This allows the cloud controller manager to get service accounts upon startup:
- apiGroups:
- ''
resources:
- serviceaccounts
verbs:
- create
- get
- Rancher-provisioned RKE nodes are tainted
node-role.kubernetes.io/controlplane
. Update tolerations and the nodeSelector:
tolerations:
- effect: NoSchedule
key: node.cloudprovider.kubernetes.io/uninitialized
value: 'true'
- effect: NoSchedule
value: 'true'
key: node-role.kubernetes.io/controlplane
nodeSelector:
node-role.kubernetes.io/controlplane: 'true'
There's currently a known issue where nodeSelector
can't be updated from the Rancher UI. Continue installing the chart and then Daemonset manually to set the nodeSelector
:
nodeSelector:
node-role.kubernetes.io/controlplane: 'true'
- Install the chart and confirm that the Daemonset
aws-cloud-controller-manager
deploys successfully:
kubectl rollout status deployment -n kube-system aws-cloud-controller-manager