Custom Model Profile Settings#
When you bring your own model into AMD AI Workbench from Hugging Face or your own model registry, the model carries a runtime profile — a set of baseline settings that describe how the model should run when it is deployed for inference. The runtime profile defines the container image and version, the accelerator type, product, and count, the model precision, and any custom engine arguments or environment variables.
These settings ship with sensible defaults supplied by the AMD Inference Microservices engine, so most users never need to change them. This page explains what each setting does, when you might want to adjust it, and what to keep in mind so your changes don’t destabilize a deployment.
Note
The runtime profile sets the baseline defaults for the model itself. You can still override individual values for a single deployment without changing the baseline — see Advanced Deployment Options for per-deployment overrides.
When to change the defaults#
Start with the defaults. They reflect a configuration that has been validated for the model and the available hardware. Consider changing the runtime profile only when you have a concrete reason, such as:
Running on a specific accelerator type or a partitioned cluster where you need to pin the accelerator and count.
Serving a larger model that needs more than one accelerator.
Matching a precision (for example,
fp16) required by your model or your performance goals.Enabling a specific inference engine capability through engine arguments or environment variables.
If you are unsure, leave a field at its default value. The engine selects an appropriate configuration automatically.
Where to set the runtime profile#
You can set the runtime profile in two places, both presented as Step 3 — Runtime profile of a three-step flow.
During import#
When you import a model, the Add new model wizard walks you through the model source, model information, and finally the runtime profile. The values you enter here become the model’s baseline defaults.
Click Save and start onboarding to finish the import with your chosen profile. If you leave the profile fields at their defaults, the engine’s own defaults are preserved.
After import#
You can edit the runtime profile at any time after a model has been imported. On the Custom Models tab, open a model’s action menu (the three dots) and select Model settings.
This opens the Edit model flow. Step through to Runtime profile to update the baseline settings, then click Save. The import source cannot be changed, but the display information and runtime profile can.
Note
Editing the runtime profile updates the model’s baseline. The new values apply to subsequent deployments of the model; deployments that are already running are not changed retroactively. To roll back, edit the profile again and restore the previous values.
Profile parameters#
Container image#
The base container image family used to run the model, such as aim-base (and EPYC- or Radeon-specific families where applicable). The image determines the inference runtime and the hardware it targets. Leave this at the default unless you have a specific reason to run a different image family.
Container version#
The version (tag) of the selected container image, for example 0.11. Newer versions may include performance improvements or fixes. Pinning a specific version keeps a deployment reproducible.
Accelerator type and Accelerator#
Accelerator type selects the broad class of hardware (for example, GPU). Accelerator selects the specific product within that class (for example, AMD Instinct MI300X OAM). Choose values that match the hardware available in your cluster. Pinning the accelerator is the key setting for partitioned-cluster use cases.
Accelerator count#
The number of accelerators allocated to a single model replica. Larger models may require more than one accelerator to load and serve. Setting this higher than necessary reserves hardware that other workloads could use; setting it too low can prevent the model from starting.
Model precision#
The numerical precision used to run the model, for example fp16. Precision affects memory footprint, throughput, and output quality. Use the precision recommended for your model unless you are deliberately trading accuracy for performance or memory.
Engine arguments and environment variables#
The runtime profile also lets you pass advanced configuration directly to the inference engine. Both are entered as key-value pairs in YAML format.
Engine arguments enable specific inference engine functionality. For example:
attention-backend: TRITON_ATTN
Environment variables tune model performance through the runtime environment. For example:
VLLM_ROCM_USE_AITER: "1"
Each field links to the relevant vLLM reference documentation (engine arguments and environment variables). You can also use a published AIM profile as a starting point.
Warning
Engine arguments and environment variables are advanced options. Invalid keys, unsupported values, or combinations that don’t match your model or hardware can prevent a deployment from starting or degrade its performance. Change them deliberately, one at a time, and validate the result before relying on the deployment.
Limitations and safety notes#
Validate before production. After changing a runtime profile, deploy and verify the model behaves as expected before depending on it for production traffic.
Defaults are the safe path. Leaving a field at its default defers to the engine’s validated configuration. Only the values you change deviate from that baseline.
Profile vs. deployment overrides. The runtime profile sets the model’s baseline; a single deployment can still override values (such as precision, accelerator, or count) without altering the baseline. See Advanced Deployment Options.
Rollback. There is no automatic version history for the profile. If a change causes problems, edit the profile again to restore the previous values, then redeploy.
Unsupported combinations. Custom engine arguments, environment variables, and image/precision/accelerator combinations are not all validated. If a configuration behaves unexpectedly — especially on new hardware — revert to defaults and contact AMD support.