Routing and Ingress

Routing and Ingress#

AIM Engine uses the Kubernetes Gateway API to expose inference services via HTTP. When routing is enabled, AIM Engine creates HTTPRoute resources that route traffic through a configured Gateway to the KServe predictor service.

v1alpha2

Routing fields (spec.routing.*) are identical across versions. Examples on this page use aim.eai.amd.com/v1alpha2. For the legacy template-shaped service, see Legacy AIMService.

Enabling Routing#

Per-Service Routing#

Enable routing on an individual service:

apiVersion: aim.eai.amd.com/v1alpha2
kind: AIMService
metadata:
  name: qwen-chat
  namespace: ml-team
spec:
  model:
    name: qwen-qwen3-32b
  routing:
    enabled: true
    gatewayRef:
      name: inference-gateway
      namespace: kgateway-system
    pathTemplate: "/{.metadata.namespace}/{.metadata.name}"

This creates an HTTPRoute matching the path /ml-team/qwen-chat and forwarding to the predictor service.

Cluster-Wide Routing Defaults#

Set routing defaults for all services via runtime configuration:

apiVersion: aim.eai.amd.com/v1alpha1
kind: AIMClusterRuntimeConfig
metadata:
  name: default
spec:
  routing:
    enabled: true
    gatewayRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: inference-gateway
      namespace: kgateway-system
    pathTemplate: "/{.metadata.namespace}/{.metadata.name}"

With cluster defaults in place, services inherit routing configuration automatically. A service can override any field by specifying its own spec.routing.

Path Templates#

Path templates use JSONPath expressions in {...} to build route paths from service metadata:

Template

Example Result

/{.metadata.namespace}/{.metadata.name}

/ml-team/qwen-chat

/models/{.metadata.name}

/models/qwen-chat

/{.metadata.namespace}/{.metadata.labels['team']}/inference

/ml-team/nlp/inference

Path templates have a maximum length of 200 characters. If no template is specified, the default path is /{namespace}/{uid}.

Hostnames#

By default, a generated HTTPRoute is not pinned to a hostname, so it attaches to every listener exposed by the parent Gateway. On a Gateway with multiple listeners (for example separate workloads.*, api.*, and ui.* hostnames) this means a service can be reached on listeners that may not enforce the intended authentication.

Pin the route to one or more hostnames so it only attaches to the matching listener:

spec:
  routing:
    enabled: true
    gatewayRef:
      name: inference-gateway
      namespace: kgateway-system
    hostnames:
      - workloads.example.com

hostnames can be set on the service (spec.routing.hostnames) or as a default on the runtime config (AIMRuntimeConfig / AIMClusterRuntimeConfig), following the same precedence as other routing fields. The service-level list overrides the runtime config list as a whole.

!!! warning “Required for multi-listener gateways” When the parent Gateway exposes more than one listener, a hostname is required. A routing-enabled service with no hostnames in that case does not get an HTTPRoute and reports ConfigValid=False with reason RouteHostnameRequired (condition RouteConfigReady=False). Set hostnames to clear it. Single-listener gateways are unaffected: leaving hostnames empty keeps the route attached to that one listener.

Request Timeout#

Set an HTTP request timeout for the route:

spec:
  routing:
    requestTimeout: 120s

Annotations#

Add annotations to the generated HTTPRoute:

spec:
  routing:
    annotations:
      example.com/rate-limit: "100"

Annotations from the runtime config are merged with service-level annotations, with service-level values taking precedence.

Routing Precedence#

Configuration is resolved in this order (highest to lowest priority):

  1. AIMService.spec.routing — service-level overrides

  2. AIMRuntimeConfig.spec.routing — namespace defaults

  3. AIMClusterRuntimeConfig.spec.routing — cluster defaults

How It Works#

AIM Engine creates an HTTPRoute with:

  • Parent: The gateway from gatewayRef

  • Hostnames: The configured hostnames, if any (required on multi-listener gateways; see Hostnames)

  • Path match: PathPrefix with the resolved path template

  • Backend: KServe predictor service ({isvc-name}-predictor, where isvc-name is the derived InferenceService name) on port 80

  • URL rewrite: The matched prefix is rewritten to / so backends receive clean paths like /v1/chat/completions

Next Steps#

Contents