Kubernetes和机器学习工作负载：从分布式训练到模型部署的全面指南

键盘鼓手苏苏

309人浏览 · 2026-04-27 22:04:56

键盘鼓手苏苏 · 2026-04-27 22:04:56 发布

Kubernetes和机器学习工作负载：从分布式训练到模型部署的全面指南

🔥 硬核开场

各位技术大佬们，今天咱们来聊聊Kubernetes和机器学习工作负载。别跟我说你的机器学习训练还在单机上跑，那都不叫现代化！在云原生时代，Kubernetes已经成为机器学习工作负载的最佳载体。从分布式训练到模型部署，从GPU管理到自动扩缩容，每一步都需要精心设计。今天susu就带你们从实战角度，全方位覆盖Kubernetes上的机器学习工作负载最佳实践，让你的模型训练既高效又可靠！

📋 核心内容

1. Kubernetes上的机器学习工作负载类型

模型训练：分布式训练、超参数调优
模型推理：在线推理、批量推理
数据处理：数据预处理、特征工程
模型管理：模型版本控制、模型注册

2. 准备Kubernetes集群

2.1 安装GPU支持

# 安装NVIDIA设备插件
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.13.0/nvidia-device-plugin.yml

# 验证GPU可用性
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{.status.allocatable.nvidia.com/gpu}{"\n"}{end}'

2.2 安装必要的工具

# 安装kubeflow
kubectl apply -f https://github.com/kubeflow/kfctl/releases/download/v1.2.0/kfctl_k8s_istio.v1.2.0.yaml

# 安装mpi-operator
helm repo add mpi-operator https://kubeflow.github.io/mpi-operator
helm install mpi-operator mpi-operator/mpi-operator

# 安装tf-operator
helm repo add kubeflow https://kubeflow.github.io/helm-charts
helm install tf-operator kubeflow/tf-operator

3. 分布式训练

3.1 TensorFlow分布式训练

apiVersion: kubeflow.org/v1
kind: TFJob
metadata:
  name: tensorflow-training
  namespace: default
spec:
  tfReplicaSpecs:
    Worker:
      replicas: 3
      template:
        spec:
          containers:
          - name: tensorflow
            image: tensorflow/tensorflow:latest-gpu
            command:
            - python
            - /app/train.py
            resources:
              limits:
                nvidia.com/gpu: 1
            volumeMounts:
            - name: training-data
              mountPath: /data
            - name: training-code
              mountPath: /app
          volumes:
          - name: training-data
            persistentVolumeClaim:
              claimName: training-data
          - name: training-code
            configMap:
              name: training-code
    PS:
      replicas: 2
      template:
        spec:
          containers:
          - name: tensorflow
            image: tensorflow/tensorflow:latest
            command:
            - python
            - /app/train.py
            resources:
              requests:
                cpu: 1
                memory: 4Gi

3.2 PyTorch分布式训练

apiVersion: kubeflow.org/v1
kind: MPIJob
metadata:
  name: pytorch-training
  namespace: default
spec:
  slotsPerWorker: 1
  runPolicy:
    cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          containers:
          - name: mpi-launcher
            image: mpioperator/pytorch:latest
            command:
            - mpirun
            - --allow-run-as-root
            - -np
            - "3"
            - --bind-to
            - none
            - -map-by
            - slot
            - -x
            - NCCL_DEBUG=INFO
            - python
            - /app/train.py
    Worker:
      replicas: 3
      template:
        spec:
          containers:
          - name: mpi-worker
            image: mpioperator/pytorch:latest
            resources:
              limits:
                nvidia.com/gpu: 1
            volumeMounts:
            - name: training-data
              mountPath: /data
            - name: training-code
              mountPath: /app
          volumes:
          - name: training-data
            persistentVolumeClaim:
              claimName: training-data
          - name: training-code
            configMap:
              name: training-code

4. 模型部署

4.1 部署模型服务

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-serving
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: model-serving
  template:
    metadata:
      labels:
        app: model-serving
    spec:
      containers:
      - name: model-server
        image: tensorflow/serving:latest
        ports:
        - containerPort: 8501
        env:
        - name: MODEL_NAME
          value: mymodel
        volumeMounts:
        - name: model-storage
          mountPath: /models/mymodel
      volumes:
      - name: model-storage
        persistentVolumeClaim:
          claimName: model-storage
---
apiVersion: v1
kind: Service
metadata:
  name: model-serving
  namespace: default
spec:
  selector:
    app: model-serving
  ports:
  - port: 8501
    targetPort: 8501
  type: ClusterIP

4.2 使用Seldon Core部署模型

# 安装Seldon Core
helm repo add seldon-charts https://seldonio.github.io/seldon-core
helm install seldon-core seldon-charts/seldon-core-operator --namespace seldon-system --create-namespace

# 部署模型
kubectl apply -f model-deployment.yaml

# model-deployment.yaml
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: my-model
  namespace: default
spec:
  predictors:
  - name: default
    replicas: 3
    graph:
      name: model
      implementation: MODEL_SERVER
      modelUri: gs://my-model-bucket/model
      env:
      - name: MODEL_NAME
        value: mymodel

5. 自动扩缩容

5.1 基于CPU/GPU使用率的扩缩容

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: model-serving-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: model-serving
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

5.2 基于自定义指标的扩缩容

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: model-serving-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: model-serving
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: requests-per-second
      target:
        type: AverageValue
        averageValue: 100

6. 数据管理

6.1 数据存储

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: training-data
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  storageClassName: standard

6.2 数据预处理

apiVersion: batch/v1
kind: Job
metadata:
  name: data-preprocessing
  namespace: default
spec:
  template:
    spec:
      containers:
      - name: preprocessing
        image: mycompany/data-preprocessing:latest
        command:
        - python
        - /app/preprocess.py
        volumeMounts:
        - name: raw-data
          mountPath: /data/raw
        - name: processed-data
          mountPath: /data/processed
      volumes:
      - name: raw-data
        persistentVolumeClaim:
          claimName: raw-data
      - name: processed-data
        persistentVolumeClaim:
          claimName: processed-data
      restartPolicy: Never
  backoffLimit: 4

7. 监控与日志

7.1 监控训练作业

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: training-jobs
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: training
  endpoints:
  - port: metrics
    interval: 15s

7.2 监控模型服务

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: model-serving
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: model-serving
  endpoints:
  - port: metrics
    interval: 15s

8. 最佳实践

8.1 训练作业最佳实践

使用StatefulSet：对于需要稳定存储的训练作业
配置资源限制：合理设置CPU、内存和GPU资源
使用节点亲和性：将训练作业调度到合适的节点
设置Pod中断预算：保证训练作业的稳定性

8.2 模型部署最佳实践

使用Deployment：便于水平扩缩容
配置健康检查：确保服务可用性
使用服务网格：管理流量和监控
实现蓝绿部署：无缝更新模型

8.3 资源管理最佳实践

GPU资源管理：合理分配GPU资源
使用节点池：为不同类型的工作负载创建专用节点池
资源配额：设置命名空间级别的资源限制
限制Pod优先级：确保关键工作负载的资源需求

9. 实战演练：完整的机器学习工作流

9.1 数据预处理

apiVersion: batch/v1
kind: Job
metadata:
  name: data-preprocessing
  namespace: ml-workloads
spec:
  template:
    spec:
      containers:
      - name: preprocessing
        image: mycompany/data-preprocessing:latest
        command:
        - python
        - /app/preprocess.py
        volumeMounts:
        - name: raw-data
          mountPath: /data/raw
        - name: processed-data
          mountPath: /data/processed
      volumes:
      - name: raw-data
        persistentVolumeClaim:
          claimName: raw-data
      - name: processed-data
        persistentVolumeClaim:
          claimName: processed-data
      restartPolicy: Never
  backoffLimit: 4

9.2 分布式训练

apiVersion: kubeflow.org/v1
kind: MPIJob
metadata:
  name: pytorch-training
  namespace: ml-workloads
spec:
  slotsPerWorker: 1
  runPolicy:
    cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          containers:
          - name: mpi-launcher
            image: mpioperator/pytorch:latest
            command:
            - mpirun
            - --allow-run-as-root
            - -np
            - "4"
            - --bind-to
            - none
            - -map-by
            - slot
            - -x
            - NCCL_DEBUG=INFO
            - python
            - /app/train.py
    Worker:
      replicas: 4
      template:
        spec:
          containers:
          - name: mpi-worker
            image: mpioperator/pytorch:latest
            resources:
              limits:
                nvidia.com/gpu: 1
            volumeMounts:
            - name: processed-data
              mountPath: /data
            - name: training-code
              mountPath: /app
          volumes:
          - name: processed-data
            persistentVolumeClaim:
              claimName: processed-data
          - name: training-code
            configMap:
              name: training-code

9.3 模型部署

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-serving
  namespace: ml-workloads
spec:
  replicas: 3
  selector:
    matchLabels:
      app: model-serving
  template:
    metadata:
      labels:
        app: model-serving
    spec:
      containers:
      - name: model-server
        image: tensorflow/serving:latest
        ports:
        - containerPort: 8501
        env:
        - name: MODEL_NAME
          value: mymodel
        volumeMounts:
        - name: model-storage
          mountPath: /models/mymodel
      volumes:
      - name: model-storage
        persistentVolumeClaim:
          claimName: model-storage
---
apiVersion: v1
kind: Service
metadata:
  name: model-serving
  namespace: ml-workloads
spec:
  selector:
    app: model-serving
  ports:
  - port: 8501
    targetPort: 8501
  type: LoadBalancer

9.4 自动扩缩容

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: model-serving-hpa
  namespace: ml-workloads
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: model-serving
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

🛠️ 最佳实践

集群配置：
- 为机器学习工作负载创建专用节点池
- 安装GPU驱动和设备插件
- 配置足够的存储容量
训练作业：
- 使用分布式训练框架
- 合理配置资源限制
- 使用StatefulSet管理有状态训练作业
- 实现训练数据的持久化
模型部署：
- 使用Deployment进行模型服务部署
- 配置健康检查和就绪探针
- 实现自动扩缩容
- 使用服务网格管理流量
数据管理：
- 使用PersistentVolumeClaim管理数据
- 实现数据预处理的自动化
- 考虑使用对象存储服务
监控与日志：
- 监控训练作业的进度和资源使用
- 监控模型服务的性能和可用性
- 集中管理日志
资源管理：
- 合理分配GPU资源
- 使用节点亲和性和反亲和性
- 设置资源配额和限制
安全配置：
- 限制容器权限
- 使用Secret管理敏感信息
- 配置网络策略

📊 总结

Kubernetes已经成为机器学习工作负载的理想平台，通过本文的实践，你应该已经掌握了：

分布式训练的配置和管理
模型部署的最佳实践
自动扩缩容的实现
数据管理和处理
监控与日志
资源管理和安全配置

记住，机器学习工作负载在Kubernetes上的运行需要根据实际需求进行调整。在实际生产环境中，要结合模型特点和业务需求，制定合适的部署策略，确保机器学习工作负载的高效和可靠运行。

susu碎碎念：

GPU资源是宝贵的，要合理分配和使用
分布式训练可以显著加速模型训练过程
模型部署要考虑性能和可用性
数据管理是机器学习工作流的关键环节
监控和日志对于问题排查至关重要
安全配置不能忽视，特别是处理敏感数据时

觉得有用？点个赞再走！咱们下期见～ 🔥

AtomGit开源社区

AtomGit 是由开放原子开源基金会联合 CSDN 等生态伙伴共同推出的新一代开源与人工智能协作平台。平台坚持“开放、中立、公益”的理念，把代码托管、模型共享、数据集托管、智能体开发体验和算力服务整合在一起，为开发者提供从开发、训练到部署的一站式体验。

更多推荐

CLI-Anything代码静态扫描和AI Code Review

静态分析是。

AtomGit开源社区

Claude code +Deepseek v4模型安装部署配置

本文详细记录了在Windows电脑上安装Claude Code并接入Deepseek V4模型的完整流程。首先确保Node.js 18+环境，通过npm安装Claude Code后修改配置文件解决地区限制问题。接着获取Deepseek API key，使用cc-switch工具配置模型参数，最终成功实现Claude Code与Deepseek V4的对接。整个过程包含环境准备、软件安装、配置修改和