AMD Inference Microservice job deployment guide#
This guide explains how to run the cytokineking/freebindcraft AIM as a long-running job container (ADR-0008) on an MI300X system. A job AIM has no HTTP API: it reads its inputs from a mounted directory, writes its results to a mounted directory, streams its log to stdout, and exits with a status code.
Prerequisites#
AMD Instinct GPU with ROCm support, and Docker configured for it
A host directory with the job input and a writable host directory for the results (a local or POSIX filesystem; see section 4 for the filesystem requirements)
Running the job#
docker run --rm \
--device=/dev/kfd --device=/dev/dri \
--group-add video --group-add render \
--security-opt seccomp=unconfined \
-e AIM_INPUT_PATH=/input \
-e AIM_OUTPUT_PATH=/output \
-v /path/to/input:/input:ro \
-v /path/to/output:/output \
amdenterpriseai/aim-instinct-model-cytokineking-freebindcraft:7.2.0-preview
The number of GPUs used, how tunables are passed, and any workload-specific container flags are described in section 4, together with the input, output, exit codes, and how to resume or stop a run.
Status and termination#
Follow progress with
docker logs -f <container>. The log file and result files written to$AIM_OUTPUT_PATHare described in section 4.Exit status
0means the run finished and its results are in the output directory. A non-zero status means failure or interruption; section 4 lists the exit statuses of this job and how a stopped run is resumed.
Other schedulers#
The image entrypoint runs the job, so neither recipe sets a command. Pass the settings and tunables of section 4.3 as environment variables, and use the volumes and the exit status as in the sections above.
Kubernetes Job#
One GPU, input and output on persistent volumes:
apiVersion: batch/v1
kind: Job
metadata:
name: aim-instinct-model-cytokineking-freebindcraft
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
# Keep this above the time the job needs to stop (section 4.8).
terminationGracePeriodSeconds: 90
containers:
- name: job
image: amdenterpriseai/aim-instinct-model-cytokineking-freebindcraft:7.2.0-preview
env:
- {name: AIM_INPUT_PATH, value: /input}
- {name: AIM_OUTPUT_PATH, value: /output}
resources:
limits:
amd.com/gpu: 1
volumeMounts:
- {name: input, mountPath: /input, readOnly: true}
- {name: output, mountPath: /output}
volumes:
- name: input
persistentVolumeClaim: {claimName: job-input}
- name: output
persistentVolumeClaim: {claimName: job-output}
The pod has no command:: the image starts the run. Kubernetes sends SIGTERM when the pod is deleted and waits
terminationGracePeriodSeconds (30 seconds by default) before SIGKILL; section 4.8 states how long the job needs to
stop. To resume after an interruption, start the job again against the same output volume (section 4.7).
Apptainer and Slurm#
apptainer run --rocm --bind /path/to/input:/input:ro --bind /path/to/output:/output \
--env AIM_INPUT_PATH=/input --env AIM_OUTPUT_PATH=/output \
docker://amdenterpriseai/aim-instinct-model-cytokineking-freebindcraft:7.2.0-preview
Wrap the command in an sbatch script that requests the GPU with --gres=gpu:1 and bounds the run with --time.
Slurm sends SIGTERM when the allocation ends; section 4.8 states what the job does then.
4. Job reference#
4.1 Overview#
The container is a long-running job (EnterpriseAI-specs ADR-0008). Start it with no arguments: it reads
AIM_INPUT_PATH, designs binders, writes AIM_OUTPUT_PATH and exits. The only accepted arguments are -h and --help
(print usage, exit 0); any other argument is rejected with exit status 2. A run takes minutes to many hours, so
always bound it with AIM_FBC_TIME_LIMIT and/or AIM_FBC_MAX_TRAJECTORIES.
4.2 Input#
AIM_INPUT_PATH is required and is only read, never written.
File |
Required |
Content |
|---|---|---|
|
yes |
The target structure. It must be non-empty and have ATOM records with C-alpha atoms. Every chain in |
|
no |
A JSON object. Its keys are the setting names in section 4.3 without the |
Weights are not input and are not in the image. Mount a directory that contains params/params_model_1_multimer_v3.npz
(the whole AlphaFold2 alphafold_params_2022-12-06.tar extracted into params/, CC BY 4.0: all the files in the
archive, including the five multimer parameter files params_model_1_multimer_v3.npz to params_model_5_multimer_v3.npz
and the monomer models; do not pick single files) and set AIM_CACHE_PATH to it. The job checks only for
params_model_1_multimer_v3.npz; if that file is missing it exits with status 2 and a message that names the expected path.
FBC_AF2_CACHE_PATH is not read by the job. It is used only by the GPU smoke test shipped with this asset
(image/tests/smoke/test_job_gpu_freebindcraft.py): the operator sets it to the host directory that holds
params/params_model_*_multimer_v3.npz, and the test mounts it read-only at /af2 with AIM_CACHE_PATH=/af2. The test
checks that the variable is set and that the directory holds params/params_model_1_multimer_v3.npz before it starts
Docker, and fails with a message that names the problem otherwise.
Preparing the target#
Always trim target.pdb to the smallest region that contains the binding site: a smaller target speeds up the design
and lowers the GPU memory needed (upstream BindCraft README, “Always try to trim the input target PDB to the smallest
size possible”). Before the run:
Remove waters, ligands and other HETATM records, and keep one clean conformation (no alternate locations, one model).
Keep only the domain or chains the binder should bind, with their original residue numbering, because
AIM_FBC_HOTSPOTSrefers to those numbers.Hotspots are optional. A small patch of several target residues narrows the search; with none, the design tool picks the binding site itself (upstream: “AF2 is very good at detecting good binding sites if no hotspot is specified”). The job logs how many of the hotspot residue numbers it finds in
target.pdband lists any it does not find; a number that is missing does not stop the run, so check that line in the log.
4.3 Settings#
Each setting is an AIM_FBC_* environment variable or a key in settings.json. The environment wins over the file, and
the file wins over the default. An unknown settings.json key, an unknown AIM_* variable, and the deprecated
AIM_GPU_COUNT and AIM_GPU_MODEL are errors (exit status 2). A non-empty environment value overrides the file; an
empty one is ignored.
Environment variable |
|
Default |
Meaning |
|---|---|---|---|
|
|
|
Chain(s) of |
|
|
empty |
Hotspot residues, comma separated; each item is a residue number or range, optionally with a chain letter in front, or a chain letter alone: |
|
|
|
Prefix of the design names (letters, digits, |
|
|
|
Minimum binder length, 31 to 300. The default |
|
|
|
Maximum binder length, 31 to 300, not below |
|
|
|
Number of accepted designs at which the run stops and ranks them (1 or more). A default run stops after 200 accepted designs, so it runs longer than with fewer. Upstream recommends at least 100 accepted designs, from which the top 5 to 20 are taken for experiments; screen more when high-affinity binders are required. |
|
|
no cap |
Stop after this many relaxed trajectories (1 or more; |
|
|
no limit |
Wall-clock cap: seconds, or a number with |
|
|
|
Restarts of the design process after it crashes by itself (0 or more; |
|
|
|
Check-only mode for CI and diagnostics ( |
Other AIM_* variables:
Variable |
Meaning |
|---|---|
|
Required. The input directory. |
|
Required. The output directory; it is created if missing. |
|
Optional. Directory with operator-supplied AlphaFold2 parameters (section 4.2). |
|
Overrides the detected GPU count (an integer, 0 or more). |
|
Logged only. |
|
Accepted and ignored, with a warning in the log: the profiles of this AIM have no engine arguments, so nothing is applied. |
|
Accepted and ignored. |
The resolved settings are logged and written to the settings field of summary.json. Example settings.json:
{
"target_chains": "A",
"hotspots": "A56",
"min_length": 80,
"max_length": 150,
"num_designs": 200,
"time_limit": "12h"
}
The CI smoke test uses a short binder (31 to 40 residues) and one design only to keep the run short; it is not a recommended design setup.
The job needs one AMD GPU. It counts /dev/dri/renderD* nodes when /dev/kfd exists, and exits with status 3 if there
are none; extra GPUs are not used. Set AIM_ACCELERATOR_COUNT only where the nodes cannot be counted.
4.4 Output#
Everything is written under AIM_OUTPUT_PATH.
Path |
Content |
|---|---|
|
Machine-readable result, written at the start, about every 30 seconds while the run is in progress, and at the end. |
|
Full log: the job’s |
|
Accepted designs: designs that pass all the filters of upstream |
|
Accepted designs ranked by |
|
Score tables and filter-failure counters of the design tool. |
|
Resume state and intermediate files. Not a result. |
Filters computed from the structures. On the average over the five models (the first two models also individually): pLDDT above 0.8, pTM above 0.55, ipTM above 0.5, i_pAE below 0.35, shape complementarity above 0.6, dSASA above 1, at least 7 interface residues, surface hydrophobicity below 0.35, binder loop content below 90 %, binder RMSD below 3.5 and hotspot RMSD below 6 angstrom.
Filters that pass by construction. Without PyRosetta, upstream (functions/pr_alternative_utils.py, the version this image
pins) does not compute four interface properties but substitutes fixed values chosen to satisfy the default thresholds:
the number of interface hydrogen bonds (at least 3), the number of unsatisfied interface hydrogen bonds (at most 4), the
binder energy score (below 0) and the interface dG (below 0). The four filters still run and are counted in
failure_csv.csv, but they never reject a design. An accepted design is therefore not evidence that it has at least 3
interface hydrogen bonds, at most 4 unsatisfied ones, a negative binder energy or a negative interface dG, and the values
of those four columns in the score tables are the placeholders, not measurements. Check these properties with a separate
tool (for example PyRosetta) before relying on them.
Result files are published under a temporary dot-name and renamed, so a reader never sees a half-written file.
bindcraft.log is the one file appended in place.
The upstream design animations (an HTML5 video of each trajectory) are off, because the image does not ship ffmpeg. This changes nothing in the designs, the score tables or the plots; only the optional animation files are not produced.
summary.json fields:
Field |
Meaning |
|---|---|
|
|
|
|
|
|
|
The exit status of the container; |
|
UTC timestamps; |
|
Wall-clock seconds of this invocation. |
|
Seconds from the start of the design process to the end of the first trajectory’s hallucination, JAX start-up and compilation included; |
|
Rows of |
|
Relaxed trajectory structures. |
|
Trajectories rejected before relaxation, from |
|
Time of each scored trajectory and their mean. |
|
Rows of |
|
Accepted and ranked design counts, and |
|
|
|
Restarts after crashes. |
|
One entry per crash: |
|
Detected GPU count; |
|
The resolved settings. |
|
Human-readable remarks, for example the reason for a failure. |
The profile metric is throughput: accepted designs per GPU-hour over a fixed wall-clock budget (AIM_FBC_TIME_LIMIT),
that is accepted_designs divided by wall_time_s, scaled to hours. summary.json reports the time to first trajectory
separately as startup cost (time_to_first_trajectory_s) and the per-trajectory time as the mean over scored
trajectories (mean_trajectory_time_s).
ipTM (the Average_i_pTM ranking column) is a good yes/no predictor of whether a binder binds the target, not of how
strongly (upstream BindCraft README). Ranking by it does not rank affinity; test the top designs experimentally.
4.5 Filesystem needs#
AIM_OUTPUT_PATH must be a POSIX filesystem with create and rename within a directory, append, rewrite in place
(the design tool rewrites its CSVs), and many small files (.work/). An object-store mount without rename or append does
not work. The output directory must be writable by the container user (root by default). The state folder holds many small files; its size
grows with the number of trajectories and accepted designs.
4.6 Exit status#
Code |
Meaning |
|---|---|
|
Finished. Either |
|
The run failed: the design process failed, reported no GPU, scored no trajectory, or kept crashing (more crashes than |
|
Invalid input or settings, a refused resume (section 4.7), unexpected arguments, or |
|
No AMD GPU found. |
|
The design tool stopped early with fewer designs than requested and no cap reached ( |
|
|
|
Stopped by signal |
4.7 Resume#
Resume is supported at trajectory granularity. Start the container again with the same AIM_OUTPUT_PATH and the same
target.pdb, chains and hotspots. The design tool re-reads its CSVs and result folders under .work/, so earlier
trajectories and accepted designs count towards the new run’s targets, and summary.json reports resumed: true.
A different target, chain set or hotspot set against an output directory that already holds work is refused with exit status
2.wall_time_sandAIM_FBC_TIME_LIMITapply to each invocation separately.Raise
AIM_FBC_NUM_DESIGNSorAIM_FBC_MAX_TRAJECTORIESto extend a finished run.Use an empty output directory for a new run.
Within one run, when the design process crashes by itself the job restarts it and it resumes the same way, at most
AIM_FBC_MAX_RESTARTS times; the time limit applies across restarts. Three crashes in a row without a new result give
up. A signal, the time limit and invalid input are never retried.
4.8 Interruption#
On SIGTERM, SIGINT or SIGHUP the job forwards the signal to the whole process group of the design process (SIGINT as
SIGINT, the others as SIGTERM), publishes the results, writes summary.json with status interrupted, and exits
128+n. If the design process has not stopped 20 seconds after the request, it is killed with SIGKILL. The same
20-second grace applies after AIM_FBC_TIME_LIMIT is reached (SIGTERM, then SIGKILL). There is no checkpoint on signal:
the trajectory in flight is lost and the finished ones are kept. The same handling applies whichever scheduler delivers
the signal. The default docker stop timeout (10 seconds) is shorter than
this grace, so use docker stop -t 30; the Kubernetes default of 30 seconds is enough for the publish step. SIGKILL cannot be caught, so only the last periodic publish remains.
4.9 Running the job#
/host/af2 is the directory that holds params/params_model_*_multimer_v3.npz (section 4.2); the job exits with status 2
without it.
First run (one design, bounded):
docker run --rm \
--device=/dev/kfd --device=/dev/dri --group-add video --group-add render \
--security-opt seccomp=unconfined \
-e AIM_INPUT_PATH=/input -e AIM_OUTPUT_PATH=/output -e AIM_CACHE_PATH=/af2 \
-e AIM_FBC_NUM_DESIGNS=1 -e AIM_FBC_MAX_TRAJECTORIES=1 -e AIM_FBC_TIME_LIMIT=2h \
-v /host/input:/input:ro -v /host/output:/output \
-v /host/af2:/af2:ro \
<image>
With a settings.json in /host/input and a time limit from the environment:
docker run --rm \
--device=/dev/kfd --device=/dev/dri --group-add video --group-add render \
--security-opt seccomp=unconfined \
-e AIM_INPUT_PATH=/input -e AIM_OUTPUT_PATH=/output -e AIM_CACHE_PATH=/af2 -e AIM_FBC_TIME_LIMIT=12h \
-v /host/input:/input:ro -v /host/output:/output \
-v /host/af2:/af2:ro \
<image>
Resume: run the same command again against the same /host/output. Check the result with
cat /host/output/summary.json and the exit status with echo $?.
On a scheduler with a time bound (for example Slurm --time), set the bound above AIM_FBC_TIME_LIMIT.