Upgrading

Upgrading#

This guide covers upgrading AIM Engine to a new version and operating in the v1alpha1 → v1alpha2 bridge window. The Bridge window section is the canonical operator reference for installation order, served and storage versions, AIMService dispatch, retained model outputs, and manual recovery.

Upgrade Procedure#

1. Update CRDs First#

CRDs must be updated before the operator, as new operator versions may depend on new CRD fields:

helm upgrade aim-engine-crds oci://docker.io/amdenterpriseai/aim-engine-crds-chart \
  --version <new-version> \
  --namespace aim-system

2. Upgrade the Operator#

helm upgrade aim-engine oci://docker.io/amdenterpriseai/aim-engine-chart \
  --version <new-version> \
  --namespace aim-system \
  --reuse-values

Or with updated values:

helm upgrade aim-engine oci://docker.io/amdenterpriseai/aim-engine-chart \
  --version <new-version> \
  --namespace aim-system \
  --values my-values.yaml

Scale-from-zero Helm value migration#

The gateway provider now selects both the controller metric contract and the matching collector. Replace legacy values before upgrading:

  • The new default is scaleFromZero.gatewayProvider=none. Set a provider explicitly before upgrading any installation with minReplicas: 0 AIMServices.

  • scaleFromZero.gatewayActivationScope=httproute becomes scaleFromZero.gatewayProvider=envoyGateway. Selecting this provider manages the collector only; install the Gateway-targeted EnvoyExtensionPolicy and configure the EnvoyProxy source-side delta OTLP sink independently for every Gateway used by AIMServices. Follow the optional Envoy scale-from-zero guide.

  • scaleFromZero.gatewayActivationScope=deployment becomes scaleFromZero.gatewayProvider=kgateway.

  • scaleFromZero.gatewayMetricsCollector.enable=true|false becomes scaleFromZero.gatewayMetricsCollector.management=helm|external.

  • scaleFromZero.gatewayMetricsCollector.gatewayName moves to scaleFromZero.gatewayMetricsCollector.kgateway.gatewayName.

Changing an existing installation from the kgateway provider to Envoy Gateway also changes routing and metric identity. Follow Migrate from kgateway to Envoy Gateway rather than treating it as only a Helm value rename.

The chart rejects legacy keys rather than silently selecting the wrong collector. When a release has stored those keys, upgrade with a migrated values file and --reset-values instead of --reuse-values.

Existing Envoy Gateway installations must coordinate the collector and platform-owned EnvoyProxy change. To avoid losing a cold activation request during the handoff:

  1. Temporarily set affected AIMServices to minReplicas >= 1.

  2. Upgrade or replace the collector so its OTLP receiver Service is Ready.

  3. Add the reportCountersAsDeltas: true OTLP sink to each relevant EnvoyProxy.

  4. Verify that normalized activation metrics reach keda-otel-add-on.

  5. Restore the intended minReplicas: 0 settings.

Helm removes its obsolete pod-discovery ClusterRole and binding automatically. For the standalone collector, remove the legacy resources after applying the new manifest:

kubectl delete clusterrole envoy-gateway-metrics-collector \
  --ignore-not-found
kubectl delete clusterrolebinding envoy-gateway-metrics-collector \
  --ignore-not-found

3. Verify#

kubectl get pods -n aim-system
kubectl get aimservice --all-namespaces

Rollback#

Roll back to the previous Helm release:

helm rollback aim-engine -n aim-system

Note

Rolling back the operator does not roll back CRD changes. New CRD fields are additive and backward compatible. If a CRD change is not backward compatible, this will be noted in the release notes.

Version Compatibility#

AIM Engine follows semantic versioning. Within a major version:

  • CRD changes are additive (new optional fields)

  • Existing resources continue to work without modification

  • API group is aim.eai.amd.com. During the bridge release, only AIMService, AIMModel, and AIMClusterModel serve both versions (see Migration window).

The minimum supported Kubernetes version is 1.32 (see Prerequisites).

Profile YAML contract backfill#

Current operators record the source runtime’s profile-YAML codec, modeled-field presence, and opaque runtime extensions on discovered and derived profiles with the aim.eai.amd.com/profile-yaml-contract annotation. Profiles created by an older operator may not have this annotation. Generated and independently hand-authored profiles use the current strict accelerator_* v1 format. Discovered and derived profiles do not guess: runtime projection retains any existing runtime and reports the missing contract until the owning producer backfills the annotation.

After upgrading an installation that still serves legacy runtimes requiring gpu / gpu_count, allow the owning AIMModel, AIMClusterModel, AIMProfileSet, or AIMClusterProfileSet to reconcile. Existing projected runtimes remain untouched while the contract is unknown, avoiding a transient rewrite to the strict schema. Reconcile upstream image-backed models before real-profile-backed profile sets so the sets inherit the recovered source contract. If an object is not backfilled through normal reconciliation, update the owning resource in a way that its controller watches, or restart the operator to enqueue its resources again. You can audit profiles that still lack the contract with:

kubectl get aimprofile -A -o json | \
  jq -r '.items[] | select(.metadata.annotations["aim.eai.amd.com/profile-yaml-contract"] == null) | [.metadata.namespace, .metadata.name] | @tsv'

kubectl get aimclusterprofile -o json | \
  jq -r '.items[] | select(.metadata.annotations["aim.eai.amd.com/profile-yaml-contract"] == null) | .metadata.name'

Generated and independently hand-authored profiles have no source YAML to inspect and continue to use the strict canonical format by design.

After the contract is backfilled, AIM Engine reconciles the projected ServingRuntime and ConfigMap automatically. Existing predictor pods are not rolled by a ConfigMap data update and must be restarted to read the corrected startup profile. Newly created pods, including scale-from-zero pods, use the corrected projection. For an AIMService, restart its predictor pods with:

kubectl delete pod -n <namespace> \
  -l aim.eai.amd.com/service=<service-name>

Migration window#

The bridge keeps supported predecessor objects reconciling while native v1alpha2 resources use the production controllers. It is compatibility software, not a migration executor or proof that a workload serves inference.

Install CRDs before the operator#

Use the CRD and operator charts from the same bridge release:

  1. Upgrade aim-engine-crds.

  2. Wait for the AIM CRDs to report Established.

  3. Upgrade aim-engine.

  4. Check operator events and the affected AIM resources before changing any desired state.

Do not install the operator before its CRDs, remove v1alpha1 endpoints, or rewrite stored objects as part of bridge installation. The operator has no permission to mutate CRD definitions or status.storedVersions.

Served and storage behavior#

Exactly three CRDs overlap:

  • AIMService, AIMModel, and AIMClusterModel serve both v1alpha1 and v1alpha2; only v1alpha2 is storage.

  • AIMArtifact, AIMClusterModelSource, AIMRuntimeConfig, AIMClusterRuntimeConfig, AIMServiceTemplate, AIMClusterServiceTemplate, and AIMTemplateCache remain served and stored only as v1alpha1.

  • AIMProfile, AIMClusterProfile, AIMProfileSet, AIMClusterProfileSet, and AIMProfileCache remain v1alpha2-only.

The overlapping CRDs use conversion strategy None: there is no conversion webhook or semantic field translation. The API server persists writes through the storage version and serves the shared wire fields through the requested served endpoint. Applying a v1alpha1 manifest does not create a second object.

AIMService pipeline selection#

One v1alpha2-rooted controller dispatches each AIMService:

  • spec.profile selects the profile pipeline.

  • spec.template, spec.model.name, spec.model.image, and spec.model.custom select the template pipeline by default.

  • aim.eai.amd.com/reconciler-pipeline: profile explicitly selects the profile pipeline. Use it only with a profile-supported shape such as spec.profile, spec.model.name, or spec.model.image.

  • aim.eai.amd.com/reconciler-pipeline: template explicitly selects the template pipeline only when spec.model is present.

An empty or unknown marker, or template without spec.model, is malformed. The controller does not fail or repair the object solely for that marker. It falls back to the rules above and emits an InvalidReconcilerPipeline Warning event describing the ignored value. status.reconcilerPipeline records the active path.

The supported grandfathered service shapes are template-backed services using model name, image, or inline custom input, including existing template selection and legacy overrides. Retained legacy model inputs include image metadata/recommended deployments, model sources, custom templates, custom hardware, and fine-tuned template matching. Existing legacy-only AIMModel fields may continue through the shared storage object; v1alpha2 AIMModel admission prevents new objects from introducing those fields. Custom or ambiguous shapes are not automatically converted to profiles.

AIMModel outputs and readiness#

The v1alpha2 AIMModel and AIMClusterModel controllers are the only production roots for both served versions. They retain two bounded output paths:

  • legacy inputs can continue producing AIMServiceTemplate or AIMClusterServiceTemplate;

  • native image discovery, model-ID generation, or profile derivation can produce AIMProfile, AIMClusterProfile, or a profile set; and

  • a grandfathered image-backed object can retain both template and profile outputs when both paths apply.

Output children keep their normal scope and controller owner. Cross-version owner references converge to the v1alpha2 owner identity without changing an existing controller or blockOwnerDeletion value.

Readiness remains path-aware: when one applicable serving path is healthy, a failure reported only by the inactive compatibility path is non-blocking. A failure in the active path remains visible and blocking. This bridge rule does not qualify model serving or replace cluster observation.

Inline auto-custom compatibility#

A supported v1alpha1 AIMService.spec.model.custom can create or reuse a v1alpha2-valid compatibility AIMModel, but reuse is limited to reconciliation of the same live source AIMService UID. Distinct AIMService objects with identical inline custom inputs intentionally receive distinct compatibility models; cross-service sharing is unsupported because each model has one UID-bound controller owner. The persisted model stores the exact predecessor custom input in the internal aim.eai.amd.com/legacy-auto-custom-spec annotation and anchors it to the live source service with labels and a UID-bound controller owner reference.

Before generating a legacy template, the model controller strictly decodes the annotation, verifies the stored v1alpha2 fields and source provenance, and hydrates predecessor-only fields on an in-memory copy. It does not write those legacy fields back to the stored model.

Malformed, contradictory, ownerless, or orphaned compatibility state produces LegacyAutoCustomSpecReady=False with reason InvalidLegacyAutoCustomSpec, emits a Warning event, and plans no template or profile child mutation. The ordinary controllers do not adopt or reconstruct such a model from labels.

For manual recovery:

  1. Export the AIMService, compatibility AIMModel, conditions, and events before changing anything.

  2. Compare the model annotation, labels, owner UID, and v1alpha2-safe fields with the authoritative source service. Do not synthesize the opaque annotation or owner reference from labels alone.

  3. If the original service is still live and its exact inline custom input is authoritative, an administrator may remove the invalid auto-generated compatibility model and let that service reconcile a replacement. This can recreate children or cached data; it is not an artifact-preserving repair.

  4. If the source service is gone, the UID differs, or intent is ambiguous, leave automatic repair disabled. Remove an unneeded orphan only after confirming no workload depends on it; otherwise record it for manual migration handling.

Migration boundary#

The bridge performs no storage rewrite, Plan/Apply operation, legacy cleanup, artifact handoff, rollback, API-removal decision, GPU qualification, or inference request. Migration execution is governed by ADR-0002 and begins only after the bridge is installed and legacy serving has been checked by an administrator. The controlled, non-production operating steps and unresolved safety limitations are documented in Bounded v1alpha1 template migration.

The local bridge contract is a static contract over API markers, generated CRDs, controller roots, ownership semantics, RBAC, and release ordering. A passing contract does not prove migration, artifact preservation, GPU serving, or inference.

Next Steps#

Contents