概述

Kubernetes(下文简称为k8s)具有对机器的资源进行分配和使用的能力,比如k8s可以指定容器最多使用多少内存以及使用多少CPU计算资源。那么问题来了,一般来说容器就是使用CPU和内存资源,那么对于需要使用显卡的Pod,k8s也能够支持吗?答案当然是可以啦!目前k8s不仅支持容器请求GPU资源,还支持请求几块显卡的GPU资源,这使得k8s在深度学习等场景下也有了用武之地。

1. 安装GPUdocker

bash
  • 01
  • 02
  • 03
  • 04
  • 05
  • 06
  • 07
  • 08
  • 09
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
#安装nvidia-docker distribution=$(. /etc/os-release;echo $ID$VERSION_ID) curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo yum install -y nvidia-container-toolkit # 安装nvidia-docker2 yum install -y nvidia-docker2 # 设置docker配置参数 cat <<EOF > /etc/docker/daemon.json { "default-runtime": "nvidia", "runtimes": { "nvidia": { "path": "nvidia-container-runtime", "runtimeArgs": [] } }, "registry-mirrors": [ "https://registry.docker-cn.com", "https://docker.mirrors.ustc.edu.cn", "http://hub-mirror.c.163.com" ], "bip": "192.168.50.1/24", "exec-opts": ["native.cgroupdriver=systemd"], "log-driver":"json-file", "log-opts": {"max-size":"500m", "max-file":"2"} } EOF

测试是否正常安装

bash
  • 01
docker run --security-opt=no-new-privileges --cap-drop=ALL --network=none -it -v /var/lib/kubelet/device-plugins:/var/lib/kubelet/device-plugins nvidia/k8s-device-plugin:1.11

出现以下信息代表没生效

bash
  • 01
  • 02
  • 03
  • 04
  • 05
2018/11/08 02:58:17 Loading NVML 2018/11/08 02:58:17 Failed to initialize NVML: could not load NVML library. 2018/11/08 02:58:17 If this is a GPU node, did you set the docker default runtime to nvidia? 2018/11/08 02:58:17 You can check the prerequisites at: https://github.com/NVIDIA/k8s-device-plugin#prerequisites 2018/11/08 02:58:17 You can learn how to set the runtime at: https://github.com/NVIDIA/k8s-device-plugin#quick-start

出现以下信息代表生效了

bash
  • 01
  • 02
  • 03
  • 04
  • 05
  • 06
2018/11/08 02:58:46 Loading NVML 2018/11/08 02:58:46 Fetching devices. 2018/11/08 02:58:46 Starting FS watcher. 2018/11/08 02:58:46 Starting OS watcher. 2018/11/08 02:58:46 Starting to serve on /var/lib/kubelet/device-plugins/nvidia.sock 2018/11/08 02:58:46 Registered device plugin with Kubelet

2. k8s集群安装gpu-scheduler

0. Prepare GPU Node

This guide assumes that the NVIDIA drivers and nvidia-docker2 have been installed.

Enable the Nvidia runtime as your default runtime on your node. To do this, please edit the docker daemon config file which is usually present at /etc/docker/daemon.json:

json
  • 01
  • 02
  • 03
  • 04
  • 05
  • 06
  • 07
  • 08
  • 09
{ "default-runtime": "nvidia", "runtimes": { "nvidia": { "path": "/usr/bin/nvidia-container-runtime", "runtimeArgs": [] } } }

if runtimes is not already present, head to the install page of nvidia-docker

1. Deploy GPU share scheduler extender in control plane

bash
  • 01
  • 02
  • 03
  • 04
  • 05
cd /etc/kubernetes/ curl -O https://raw.githubusercontent.com/AliyunContainerService/gpushare-scheduler-extender/master/config/scheduler-policy-config.json cd /tmp/ curl -O https://raw.githubusercontent.com/AliyunContainerService/gpushare-scheduler-extender/master/config/gpushare-schd-extender.yaml kubectl create -f gpushare-schd-extender.yaml

2. Modify scheduler configuration

The goal is to include /etc/kubernetes/scheduler-policy-config.json into the scheduler configuration (/etc/kubernetes/manifests/kube-scheduler.yaml).
Here is the sample of the final modified kube-scheduler.yaml

Notice: If your Kubernetes default scheduler is deployed as static pod, don't edit the yaml file inside /etc/kubernetes/manifest. You need to edit the yaml file outside the /etc/kubernetes/manifest directory. and copy the yaml file you edited to the '/etc/kubernetes/manifest/' directory, and then kubernetes will update the default static pod with the yaml file automatically.

2.1 Add Policy config file parameter in scheduler arguments

yaml
  • 01
- --policy-config-file=/etc/kubernetes/scheduler-policy-config.json

2.2 Add volume mount into Pod Spec

yaml
  • 01
  • 02
  • 03
- mountPath: /etc/kubernetes/scheduler-policy-config.json name: scheduler-policy-config readOnly: true
yaml
  • 01
  • 02
  • 03
  • 04
- hostPath: path: /etc/kubernetes/scheduler-policy-config.json type: FileOrCreate name: scheduler-policy-config

3. Deploy Device Plugin

bash
  • 01
  • 02
  • 03
  • 04
wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-rbac.yaml kubectl create -f device-plugin-rbac.yaml wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-ds.yaml kubectl create -f device-plugin-ds.yaml

Notice: please remove default GPU device plugin, for example, if you are using nvidia-device-plugin, you can run kubectl delete ds -n kube-system nvidia-device-plugin-daemonset to delete.

4. Add gpushare node labels to the nodes requiring GPU sharing

You need to add a label "gpushare=true" to all node where you want to install device plugin because the device plugin is deamonset.

bash
  • 01
kubectl label node <target_node> gpushare=true

For example:

bash
  • 01
kubectl label node mynode gpushare=true

5. Install Kubectl extension

5.1 Install kubectl 1.12 or above

You can download and install kubectl for linux

bash
  • 01
  • 02
  • 03
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.12.1/bin/linux/amd64/kubectl chmod +x ./kubectl sudo mv ./kubectl /usr/bin/kubectl

5.2 Download and install the kubectl extension

bash
  • 01
  • 02
  • 03
cd /usr/bin/ wget https://github.com/AliyunContainerService/gpushare-device-plugin/releases/download/v0.3.0/kubectl-inspect-gpushare chmod u+x /usr/bin/kubectl-inspect-gpushare

参考文档
https://github.com/AliyunContainerService/gpushare-scheduler-extender/blob/master/docs/install.md
https://bluesmilery.github.io/blogs/afcb1072/