Run standalone fine-tuning on Kubernetes

Run standalone fine-tuning on Kubernetes#

This guide deploys a supervised fine-tuning (SFT) job with Kubernetes manifests

It has the following setup:

  • A model PVC and a dataset PVC are mounted read-only at /model and /data.

  • An ephemeral PVC is mounted at /output for checkpoints and rendered runtime files.

  • An in-memory emptyDir is mounted at /dev/shm.

  • An optional sidecar copies the final adapter to a persistent export PVC.

This workflow assumes that the model and dataset have already been downloaded to PVCs.

Prerequisites#

You need access to a Kubernetes cluster with AMD GPU resources and a storage class that supports the requested volumes. The model and dataset must already be available in persistent volume claims.

Verify the input claims before creating the job:

kubectl get pvc

Record the input claim names and paths. The example below assumes:

  • Model claim: aimft-dev-model-google-gemma-3-27b-it

  • Dataset claim: aimft-dev-dataset-sft-demo-data

  • Dataset path: /data/sft-demo-data.jsonl

Deploy the training job#

Save the following manifest as standalone-sft.yaml. Replace the image, claim names, model ID, dataset path, storage class, and resource values for your environment before applying it.

apiVersion: batch/v1
kind: Job
metadata:
  name: standalone-sft
spec:
  ttlSecondsAfterFinished: 3600
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: sft
          image: docker.io/silogenai/aimft-aimfttk:YOUR_TAG
          imagePullPolicy: Always
          resources:
            requests:
              amd.com/gpu: "1"
              cpu: "8"
              memory: "64Gi"
            limits:
              amd.com/gpu: "1"
              cpu: "8"
              memory: "64Gi"
          env:
            - name: AIMFT_MODEL_PATH
              value: /model/google/gemma-3-27b-it
            - name: AIMFT_OUTPUT_PATH
              value: /output
            - name: AIMFT_TRAIN_DATA_PATH
              value: /data/sft-demo-data.jsonl
            - name: AIM_MODEL_ID
              value: google/gemma-3-27b-it
            - name: AIMFT_USE_ADAPTER
              value: "true"
          volumeMounts:
            - name: model
              mountPath: /model
              readOnly: true
            - name: data
              mountPath: /data
              readOnly: true
            - name: output
              mountPath: /output
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: model
          persistentVolumeClaim:
            claimName: aimft-dev-model-google-gemma-3-27b-it
            readOnly: true
        - name: data
          persistentVolumeClaim:
            claimName: aimft-dev-dataset-sft-demo-data
            readOnly: true
        - name: output
          ephemeral:
            volumeClaimTemplate:
              spec:
                accessModes: [ReadWriteOnce]
                storageClassName: mlstorage
                resources:
                  requests:
                    storage: 100Gi
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 12Gi

Apply the manifest and follow the job:

kubectl apply -f standalone-sft.yaml
kubectl get pods -l job-name=standalone-sft -w
kubectl logs job/standalone-sft -f

Size the output volume for the selected model and training method. Full-parameter checkpoints can require substantially more space. Omit storageClassName to use the cluster’s default storage class.

The ephemeral output claim is deleted with the Job’s pod. Export the adapter if you need to retain it after the job finishes.

Export the final adapter#

Add the following resources before the Job in standalone-sft.yaml to persist /output/checkpoint-final-adapter in a shared PVC. The export container needs permission to inspect and delete its own pod if an export error occurs. The example uses ReadWriteMany so the export and inference pods can mount the claim at the same time; choose an access mode supported by your storage class and workload topology.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: standalone-sft
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: standalone-sft-pod-reader
rules:
  - apiGroups: [""]
    resources: [pods]
    verbs: [get, delete]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: standalone-sft-pod-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: standalone-sft-pod-reader
subjects:
  - kind: ServiceAccount
    name: standalone-sft
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: standalone-sft-adapters
spec:
  accessModes: [ReadWriteMany]
  storageClassName: mlstorage
  resources:
    requests:
      storage: 128Gi
---

Then add serviceAccountName beside restartPolicy under spec.template.spec in the Job:

spec:
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: standalone-sft

Add the following sidecar under containers, and add the model-export volume shown below. Replace my-adapter with a unique destination name. Atomic directory creation prevents concurrent jobs from overwriting an existing export.

- name: model-export
  image: bitnami/kubectl:latest
  command:
    - /bin/sh
    - -c
    - |
      set -e
      DEST=/model-export/my-adapter
      die() {
        echo "ERROR: $1" >&2
        kubectl delete pod "$POD_NAME"
        exit 1
      }
      if ! mkdir "$DEST" 2>/dev/null; then
        die "destination $DEST already exists, refusing to overwrite."
      fi
      trap 'rmdir "$DEST" 2>/dev/null' EXIT
      echo "Waiting for sft container to finish..."
      while true; do
        STATE=$(kubectl get pod "$POD_NAME" -o jsonpath='{.status.containerStatuses[?(@.name=="sft")].state.terminated.reason}')
        if [ "$STATE" = Completed ]; then
          echo "sft completed successfully, copying adapter..."
          cp -r /output/checkpoint-final-adapter/. "$DEST"
          trap - EXIT
          exit 0
        elif [ -n "$STATE" ]; then
          echo "sft terminated with reason: $STATE, skipping export." >&2
          exit 1
        fi
        sleep 10
      done
  env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
  volumeMounts:
    - name: output
      mountPath: /output
      readOnly: true
    - name: model-export
      mountPath: /model-export

      volumes:
        - name: model-export
          persistentVolumeClaim:
            claimName: standalone-sft-adapters

After the Job reports Complete, the adapter is available at my-adapter/ in the standalone-sft-adapters claim. The directory must use the Hugging Face PEFT layout expected by the inference image:

my-adapter/
├── adapter_config.json
└── adapter_model.safetensors

Deploy the adapter for inference#

Deploy the trained LoRA adapter as an AIM. Replace the image, model ID, and PVC names for your environment. Use an AIM image built for the same base model as the adapter.

Verify the adapter PVC#

The PVC must contain the adapter directory created by the export step. Verify that it exists before creating the deployment:

export ADAPTER_PVC=standalone-sft-adapters

kubectl get pvc "$ADAPTER_PVC"

Create the Deployment#

Save this manifest as gemma-aim-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gemma-aim
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gemma-aim
  template:
    metadata:
      labels:
        app: gemma-aim
    spec:
      containers:
        - name: aim
          image: amdenterpriseai/aim-instinct-google-gemma-3-27b-it:0.13.0
          imagePullPolicy: Always
          ports:
            - containerPort: 8000
          resources:
            requests:
              amd.com/gpu: "1"
              memory: "64Gi"
              cpu: "8"
            limits:
              amd.com/gpu: "1"
              memory: "64Gi"
              cpu: "8"
          env:
            - name: AIM_MODEL_ID
              value: google/gemma-3-27b-it
            - name: AIM_ADAPTER_SOURCE
              value: /adapters
            - name: AIM_ADAPTER_MODE
              value: dynamic
            - name: AIM_ADAPTER_MAX_COUNT
              value: "8"
            - name: AIM_ADAPTER_MAX_CPU_COUNT
              value: "16"
            - name: AIM_ADAPTER_MAX_RANK
              value: "64"
          volumeMounts:
            - name: adapter
              mountPath: /adapters
              readOnly: true
      volumes:
        - name: adapter
          persistentVolumeClaim:
            claimName: standalone-sft-adapters
            readOnly: true

Apply the Deployment and wait for it to become ready:

kubectl apply -f gemma-aim-deployment.yaml
kubectl rollout status deployment/gemma-aim
kubectl logs deployment/gemma-aim -f

Set AIM_ADAPTER_MAX_RANK to at least the LoRA rank used during training. The Gemma recipe and the corresponding deploy-aim override use rank 64.

Create the Service#

Save this manifest as gemma-aim-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: gemma-aim
spec:
  selector:
    app: gemma-aim
  ports:
    - port: 8000
      targetPort: 8000

Apply the Service and inspect the endpoint:

kubectl apply -f gemma-aim-service.yaml
kubectl get deployment gemma-aim
kubectl get pods -l app=gemma-aim
kubectl get service gemma-aim

For an alternative automated deployment approach, use AIM Engine. For more information, see the AIM Engine repository.

Contents