说明:
vsphere cloud provider
;VMDK
做为其持久卷。vCenter
IP | Version | USER | PASSWORD | DataCenter | DataStore | Remark |
---|---|---|---|---|---|---|
172.20.0.16 | 6.0.0 内部版本5112533 | administrator@vsphere.local | AIUc63$#v4LZ | Mcloud | - | |
- | - | k8s@vsphere.local | k8S!@Vcp123 | Mcloud | vsanDatastore |
说明:
disk UUID
;vsphere cloud provider(vcp)
账号赋权;vcp
账号与vCenter交互;kubernetes-cluster nodes
Hostname | IP | Roles | Version | Remark |
---|---|---|---|---|
kubemaster01 | 172.30.200.101 | master&minion | v1.12.3 | |
kubemaster02 | 172.30.200.102 | master&minion | v1.12.3 | |
kubemaster03 | 172.30.200.103 | master&minion | v1.12.3 |
govc node
Hostname | IP | Roles | Version | Remark |
---|---|---|---|---|
ansible01 | 172.30.200.46 | govc | v0.19.0 |
vsphere vm folder
,在vCenter的虚拟机与模板
下创建,文件夹名自定义,这里定义为kubernetes-cluster
;name
需要与其操作系统hostname
一致;disk UUID
需要设置为enabled
(见下文详细设置);name
需要遵守以下regex:[a-z](([-0-9a-z]+)?[0-9a-z])?(\.[a-z0-9](([-0-9a-z]+)?[0-9a-z])?)*
:vSphere cloud provider user
,这里vcp
账号设置为k8s@vsphere.local
:k8s
账号 ;k8s-vcp
角色,角色按照官方说明中dynamic persistent volume provisioning with storage policy
赋权;System.*
权限,此类权限不用特别赋权,默认即带,且在角色清单中也找不到System.*
权限。通过govc
工具,可以设置disk UUID
,参考:GitHub vmware/govmomi
# 选择版本,直接安装binariescurl -L https://github.com/vmware/govmomi/releases/download/v0.19.0/govc_linux_amd64.gz | gunzip > /usr/local/bin/govcchmod +x /usr/local/bin/govc
# 设置环境变量export GOVC_URL=‘172.20.0.16‘ # vCenter ip地址或FQDNexport GOVC_USERNAME=‘administrator@vsphere.local‘ # vCenter administrator账号export GOVC_PASSWORD=‘AIUc63$#v4LZ‘ # vCenter administrator账号密码export GOVC_INSECURE=1 # 如果使用了自签名证书,设置为"1"# 设置环境变量后,可查看govc变量与版本govc envgovc about
通过govc可以确定kubernetes集群vm节点位置:
# 格式:/<datacenter>/vm/<vsphere vm folder># <datacenter>:vsphere中datacenter名,根据实际环境填写,这里取值"Mcloud";# vm:固定参数,如:vm,network,host,datastore;# <vsphere vm folder>:自定义名字的"vsphere vm folder",这里取值"kubernetes-cluster"govc ls /Mcloud/vm/kubernetes-cluster
设置kubernetes集群vm节点的disk UUID
为true
:
# "disk.enableUUID=1"即为"true";# 将已设置"disk.enableUUID=1"的vm打成模板,通过模板克隆出新的vm可继承此属性govc vm.change -e="disk.enableUUID=1" -vm=‘/Mcloud/vm/kubernetes-cluster/kubemaster01‘govc vm.change -e="disk.enableUUID=1" -vm=‘/Mcloud/vm/kubernetes-cluster/kubemaster02‘govc vm.change -e="disk.enableUUID=1" -vm=‘/Mcloud/vm/kubernetes-cluster/kubemaster03‘
补充govc
用法(与本实验无关):手工创建vmdk
# 格式:datastore.disk.create -ds <DataStore> -size xG <Folder>/<Name>.vmdk# <DataStore>:需要调用的后端存储名字,这里取值"vsanDatastore";# <Folder>:<DataStore>中存放新建存储卷的目录,这里自定义目录名"k8sData",需要注意的是目录需要提前创建,否则会报错;# vmdk文件创建之后的初始大小为0kbgovc datastore.disk.create -ds vsanDatastore -size 10G k8sData/MySQLDisk.vmdk
在所有kubernetes master节点创建文件(自定义路径&名字):/etc/kubernetes/vsphere.conf
vim /etc/kubernetes/vsphere.conf# 全局属性[Global]# 从安全角度考虑,生产环境应使用针对性的账号user = "k8s@vsphere.local"password = "k8S!@Vcp123"port = "443"# 如果使用自签名证书,应设置为"1"insecure-flag = "1" datacenters = "Mcloud"# 针对kubernetes集群节点跨多vCenter的情况,可设置多"VirtualCenter";# "user","password","datacenters"也可设置在此,可覆盖全局属性[VirtualCenter "172.20.0.16"]# vSphere Cloud Provider使用"SPBM"存储策略配置存储卷[Workspace]# mandatoryserver = "172.20.0.16"# mandatorydatacenter = "Mcloud"# mandatoryfolder = "kubernetes-cluster"# option,本实验省略;# kubernetes创建pvc资源时,如果选项中未指定数据存储或存储策略,则使用默认数据存储放置vmdk#default-datastore="vsanDatastore"# option,本实验省略;# 资源池文件夹,放置创建的vmdk#resourcepool-path="kubernetes-cluster"[Disk]# option;# lsilogic-sas or pvscsi, default: pvscsiscsicontrollertype = pvscsi [Network]# option,本实验省略;# Name of public VM Network to which the VMs in the cluster are connected. Used to determine public IP addresses of VMs.#public-network = "VM Network"
在所有kubernetes master节点,为kube-apiserver,kube-controller-manager,kubelet等3个服务添加参数:
--cloud-provider=vsphere--cloud-config=/etc/kubernetes/vsphere.conf
在所有kubernetes minion节点,为kubelet添加参数:
--cloud-provider=vsphere
--cloud-provider
参数,--hostname-override
参数将失效;--hostname-override
参数,可以通过kubectl delete node <IP>
删除状态为NotReady
的节点。kubernetes可以通过以下方式调用vSAN:
cat vsphere-storageclass-spbm.yaml apiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: vsphere-storageclass-spbm# 存储分配器,根据后端存储不同而变更provisioner: kubernetes.io/vsphere-volume# 默认即"Delete",删除pvc后,相应的pv及后端的volume等一起删除;# 设置为"Retain"时则保留数据,需要手工处理reclaimPolicy: Deleteparameters: # 指定具体的datastore,如果不指定则采用在"vsphere-cloud-config-file(vsphere.conf)"中设置的默认"default-datastore" datastore: vsanDatastore # 默认即"thin",另有"zeroedthick"与"eagerzeroedthick"可选 diskformat: thin # 磁盘格式,如:xfs,ext3,ext4等 fstype: xfs # 以下两条是调用SPBM策略,SPBM策略有:cacheReservation,diskStripes,forceProvisioning,hostFailuresToTolerate,iopsLimit,objectSpaceReservation # 以下设置是vCenter中默认的"Virtual SAN Default Storage Policy"的设置 hostFailuresToTolerate: "1" diskStripes: "1" # 创建StorageClasskubecet create -f vsphere-storageclass-spbm.yaml# 查看StorageClasskubectl describe sc vsphere-storageclass-spbm
创建pvc
cat vsphere-pvc002.yaml kind: PersistentVolumeClaimapiVersion: v1metadata: name: vsphere-pvc002 # 调用StrageClass annotations: volume.beta.kubernetes.io/storage-class: vsphere-storageclass-spbmspec: # ReadWriteOnce:简写RWO,读写权限,且只能被单个node挂载; # ReadOnlyMany:简写ROX,只读权限,允许被多个node挂载; # ReadWriteMany:简写RWX,读写权限,允许被多个node挂载 accessModes: - ReadWriteOnce resources: requests: # 注意格式,不能写“GB” storage: 10Gi# 创建PVCkubecet create -f vsphere-pvc002.yaml# 查看PVCkubecet describe pvc vsphere-pvc002
PV由PVC调用StorageClass自动创建
# 查看自动创建的PVkubectl get pvkubecet describe pv pvc-b3754406-0359-11e9-b1f2-005056bfac1e
同步查看vCenter信息
查看在vSAN DataStore上生成的vmdk
cat vsphere-pod002.yaml apiVersion: v1kind: Podmetadata: name: vsphere-pod002spec: containers: - name: test-container # 使用中科大的镜像仓库 image: gcr.mirrors.ustc.edu.cn/google_containers/test-webserver volumeMounts: - name: test-volume mountPath: /test-vmdk volumes: - name: test-volume # Pod挂载PVC persistentVolumeClaim: claimName: vsphere-pvc002 # 创建Podkubecet create -f vsphere-pod002.yaml# 查看Podkubectl describe pod vsphere-pod002