环境：centos7

kubernetes版本：1.24

前言：这两天在搞大模型服务，之前也没了解过GPU相关的服务，因此整理了一个关于如何能够使用gpu节点文档。

1、现象

本来服务已经部署上去，但是一直处理pending状态，不能正常调度，describe查看返回这么一个错误：

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  45m (x23 over 151m)  default-scheduler  0/3 nodes are available: 3 Insufficient nvidia.com/gpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

重点是这个 Insufficient nvidia.com/gpu.，提示我没有可用的gpu，第一次遇到是也是感觉有点蒙圈，因为再机器上已经成功安装好驱动，可以正常查看GPU状态。

经过查询，kubernetes要调度GPU，也需要安装对应厂商的设备插件

• GPU容器创建流程

containerd --> containerd-shim--> nvidia-container-runtime --> nvidia-container-runtime-hook --> libnvidia-container --> runc -- > container-process

2.安装GPU插件

gpu节点加入k8s集群，做AI分析是一种比较常见的情景，当pod使用GPU卡时，通常有以下两种方案：

2.1 方案一：使用nvidia官方插件

此时资源分配时，是按照卡的数量进行资源分配，k8s的yml文件类似于如下:

resources:
  limits:
    nvidia.com/gpu: '1'
  requests:
    nvidia.com/gpu: '1'

其中1表示使用1张GPU卡。

步骤一: 安装nvidia Linux驱动

• 下载并安装nvidia Linux 驱动:

apt-get update
apt-get install gcc make
## cuda10.1
wget -c https://ops-software-binary-1255440668.cos.ap-chengdu.myqcloud.com/nvidia/NVIDIA-Linux-x86_64-430.50.run
bash NVIDIA-Linux-x86_64-430.50.run
## cuda10.2
wget -c https://ops-software-binary-1255440668.cos.ap-chengdu.myqcloud.com/nvidia/NVIDIA-Linux-x86_64-440.100.run
bash NVIDIA-Linux-x86_64-440.100.run
## cuda11
wget -c https://ops-software-binary-1255440668.cos.ap-chengdu.myqcloud.com/nvidia/NVIDIA-Linux-x86_64-450.66.run
bash NVIDIA-Linux-x86_64-450.66.run
## cuda11.4
wget -c https://ops-software-binary-1255440668.cos.ap-chengdu.myqcloud.com/nvidia/NVIDIA-Linux-x86_64-470.57.02.run
bash NVIDIA-Linux-x86_64-470.57.02.run

步骤二: 安装 nvidia runtime

https://nvidia.github.io/nvidia-container-runtime/

• ubuntu 安装 nvidia runtime

curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
cat > /etc/apt/sources.list.d/nvidia-docker.list <<'EOF'
deb https://nvidia.github.io/libnvidia-container/ubuntu16.04/$(ARCH) /
deb https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/$(ARCH) /
deb https://nvidia.github.io/nvidia-docker/ubuntu16.04/$(ARCH) /
EOF
apt-get update
apt-get install nvidia-container-runtime

• centos 安装 nvidia runtime

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo
DIST=$(sed -n 's/releasever=//p' /etc/yum.conf)
DIST=${DIST:-$(. /etc/os-release; echo $VERSION_ID)}
sudo rpm -e gpg-pubkey-f796ecb0
sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/$DIST/nvidia-docker/gpgdir --delete-key f796ecb0
sudo yum makecache
yum -y install nvidia-container-runtime

步骤三：配置containerd/docker

containerd

若K8s使用containerd，按照如下方式进行配置：

修改/etc/containerd/config.toml文件

cat /etc/containerd/config.toml

#其他的根据自己的需求修改，我这里只说明适配gpu的配置
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".containerd]
#-------------------修改开始-------------------------------------------
      default_runtime_name = "nvidia"
#-------------------修改结束-------------------------------------------

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
#-------------------新增开始-------------------------------------------
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia] 
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
            BinaryName = "/usr/bin/nvidia-container-runtime" 
#-------------------新增结束-------------------------------------------

重启containerd

systemctl restart containerd.service

docker

• 若K8s使用docker，按照如下方式进行配置:

配置daemon.json

cat /etc/docker/daemon.json

{
  "registry-mirrors": [
    "https://wlzfs4t4.mirror.aliyuncs.com"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "data-root": "/data/var/lib/docker",
  "bip": "169.254.31.1/24",
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

重启docker

systemctl restart docker

步骤四：在 Kubernetes 中启用 GPU 支持

在集群中的所有 GPU 节点上配置上述选项后，您可以通过部署以下 Daemonset 来启用 GPU 支持：

cat nvidia-device-plugin.yml

[root@ycloud ~]# cat nvidia-device-plugin.yaml 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      # Mark this pod as a critical add-on; when enabled, the critical add-on
      # scheduler reserves resources for critical add-on pods so that they can
      # be rescheduled after a failure.
      # See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
      priorityClassName: "system-node-critical"
      containers:
      - image: ycloudhub.com/middleware/nvidia-gpu-device-plugin:v0.12.3
        name: nvidia-device-plugin-ctr
        env:
          - name: FAIL_ON_INIT_ERROR
            value: "false"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
          - name: device-plugin
            mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins

通过apply进行部署，并查看

kubectl apply -f nvidia-device-plugin.yml
kubectl get po -n kube-system | grep nvidia

验证

[root@ycloud ~]#  kubectl describe nodes ycloud
......
Capacity:
  cpu:                32
  ephemeral-storage:  458291312Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             131661096Ki
  nvidia.com/gpu:     2
  pods:               110
Allocatable:
  cpu:                32
  ephemeral-storage:  422361272440
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             131558696Ki
  nvidia.com/gpu:     2
  pods:               110
......

2.2 方案二：使用第三方插件

这里使用阿里云gpushare插件，git地址：https://github.com/AliyunContainerService/gpushare-device-plugin/

此时资源分配时，是可以按照卡的显存大小进行资源分配，k8s的yml文件类似于如下：

resources:
  limits:
    aliyun.com/gpu-mem: '3'
  requests:
    aliyun.com/gpu-mem: '3'

其中3表示使用的显存大小,单位G

步骤一: 安装nvidia Linux驱动

• 下载并安装nvidia Linux 驱动:

apt-get update
apt-get install gcc make
## cuda10.1
wget -c https://ops-software-binary-1255440668.cos.ap-chengdu.myqcloud.com/nvidia/NVIDIA-Linux-x86_64-430.50.run
bash NVIDIA-Linux-x86_64-430.50.run
## cuda10.2
wget -c https://ops-software-binary-1255440668.cos.ap-chengdu.myqcloud.com/nvidia/NVIDIA-Linux-x86_64-440.100.run
bash NVIDIA-Linux-x86_64-440.100.run
## cuda11
wget -c https://ops-software-binary-1255440668.cos.ap-chengdu.myqcloud.com/nvidia/NVIDIA-Linux-x86_64-450.66.run
bash NVIDIA-Linux-x86_64-450.66.run

步骤二: 安装 nvidia runtime

https://nvidia.github.io/nvidia-container-runtime/

• ubuntu 安装 nvidia runtime

curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
cat > /etc/apt/sources.list.d/nvidia-docker.list <<'EOF'
deb https://nvidia.github.io/libnvidia-container/ubuntu16.04/$(ARCH) /
deb https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/$(ARCH) /
deb https://nvidia.github.io/nvidia-docker/ubuntu16.04/$(ARCH) /
EOF
apt-get update
apt-get install nvidia-container-runtime

• centos 安装 nvidia runtime

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo
DIST=$(sed -n 's/releasever=//p' /etc/yum.conf)
DIST=${DIST:-$(. /etc/os-release; echo $VERSION_ID)}
sudo rpm -e gpg-pubkey-f796ecb0
sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/$DIST/nvidia-docker/gpgdir --delete-key f796ecb0
sudo yum makecache
yum -y install nvidia-container-runtime

步骤三：配置containerd/docker

containerd

若K8s使用containerd，按照如下方式进行配置：

修改/etc/containerd/config.toml文件

#其他的根据自己的需求修改，我这里只说明适配gpu的配置
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".containerd]
#-------------------修改开始-------------------------------------------
      default_runtime_name = "nvidia"
#-------------------修改结束-------------------------------------------

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
#-------------------新增开始-------------------------------------------
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia] 
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
            BinaryName = "/usr/bin/nvidia-container-runtime" 
#-------------------新增结束-------------------------------------------

重启containerd

systemctl restart containerd.service	

docker

• 若K8s使用docker，按照如下方式进行配置：

配置daemon.json

cat /etc/docker/daemon.json

{
  "registry-mirrors": [
    "https://wlzfs4t4.mirror.aliyuncs.com"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "data-root": "/data/var/lib/docker",
  "bip": "169.254.31.1/24",
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

重启docker

systemctl restart docker

步骤四：安装gpushare-scheduler-extender插件

参考文档：https://github.com/AliyunContainerService/gpushare-scheduler-extender/blob/master/docs/install.md

1.修改kube-scheduler配置:

因为我们k8s使用二进制方式安装,所以直接修改kube-scheduler启动文件，步骤如下：

• 先创建/etc/kubernetes/scheduler-policy-config.json，内容如下：

{
  "kind": "Policy",
  "apiVersion": "v1",
  "extenders": [
    {
      "urlPrefix": "http://127.0.0.1:32766/gpushare-scheduler",
      "filterVerb": "filter",
      "bindVerb":   "bind",
      "enableHttps": false,
      "nodeCacheCapable": true,
      "managedResources": [
        {
          "name": "aliyun.com/gpu-mem",
          "ignoredByScheduler": false
        }
      ],
      "ignorable": false
    }
  ]
}

• 然后修改cat /etc/systemd/system/kube-scheduler.service文件，添加--policy-config-file相关内容

cat /etc/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-scheduler \
  --address=127.0.0.1 \
  --master=http://127.0.0.1:8080 \
  --leader-elect=true \
  --v=2 \
  --policy-config-file=/etc/kubernetes/scheduler-policy-config.json
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

重启kube-scheduler

systemctl daemon-reload
systemctl restart kube-scheduler.service

2.部署gpushare-schd-extender

curl -O https://raw.githubusercontent.com/AliyunContainerService/gpushare-scheduler-extender/master/config/gpushare-schd-extender.yaml
kubectl apply -f gpushare-schd-extender.yaml

3.部署device-plugin

首先需要给节点添加label "gpushare=true"

kubectl label node <target_node> gpushare=true
For example:
kubectl label node mynode gpushare=true

然后部署device-plugin插件

wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-rbac.yaml
kubectl apply -f device-plugin-rbac.yaml
wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-ds.yaml
kubectl apply -f device-plugin-ds.yaml

4.安装kubectl-inspect-gpushare插件，用来查看GPU使用情况

cd /usr/bin/
wget https://github.com/AliyunContainerService/gpushare-device-plugin/releases/download/v0.3.0/kubectl-inspect-gpushare
chmod u+x /usr/bin/kubectl-inspect-gpushare