Testing

Testing#

AIM Engine uses Go unit tests and Chainsaw for declarative e2e tests.

Unit Tests#

make test                           # All unit tests (excludes e2e)
go test ./internal/v1alpha1/aimservice -v    # Specific package
go test ./internal/... -run TestFoo # Specific test

E2E Tests (Chainsaw)#

Chainsaw tests are declarative YAML files in tests/e2e/. Each test directory contains a chainsaw-test.yaml that defines steps: apply resources, assert conditions, run scripts.

Running Tests#

# Ensure the operator is running and ready
make wait-ready

# Run all tests for the current environment
make test-chainsaw

# Run a specific test directory
make test-chainsaw CHAINSAW_TEST_DIR=tests/e2e/aimservice/frozen

Environment Selectors#

Tests are filtered by environment. When ENV=kind (default), tests tagged with requires=longhorn or other infrastructure requirements are excluded automatically.

Test Reports#

JSON reports are written to .tmp/chainsaw-reports/chainsaw-report.json. Analyze failures:

# List failed tests
jq -r '.tests[] | select(.steps[].operations[].failure) | .name' \
  .tmp/chainsaw-reports/chainsaw-report.json | sort -u

# Get failure details
jq -r '.tests[] | select(.steps[].operations[].failure) |
  {name, failures: [.steps[].operations[] | select(.failure) | .failure.error]}' \
  .tmp/chainsaw-reports/chainsaw-report.json

Correlating with Operator Logs#

Chainsaw creates unique namespaces like chainsaw-<adjective>-<noun> for each test. Extract the namespace from test failures and search operator logs:

LOG=$(ls -t .tmp/logs/air-*.log | head -1)
grep "chainsaw-<namespace>" "$LOG"

Writing Tests#

Test Structure#

tests/e2e/my-feature/
  chainsaw-test.yaml    # Test definition
  resource.yaml         # Resources to apply
  assert.yaml           # Expected state assertions

Example Test#

apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
  name: basic-service
spec:
  steps:
    - try:
        - apply:
            file: service.yaml
        - assert:
            file: assert.yaml
            timeout: 120s

Debug Simulation#

For tests that involve model downloads, use simulation mode to avoid real network calls:

env:
  - name: AIM_DEBUG_SIMULATE_HF_DOWNLOAD
    value: "true"
  - name: AIM_DEBUG_SIMULATE_HF_DURATION
    value: "2"

Test Directories#

Key e2e test areas:

Directory

What it tests

tests/e2e/aimmodel/fine-tuned/

aimId-based template matching for fine-tuned models

tests/e2e/aimmodel/custom-models/

Custom models with explicit hardware and modelSources

tests/e2e/aimservicetemplate/

Template discovery, inline sources, GPU availability

tests/e2e/aimservice/

Full service lifecycle including frozen models and GPU tests

tests/e2e/aimartifact/

Model artifact downloads, quotas, and protocols

Run a specific test area:

make test-chainsaw CHAINSAW_ARGS="--test-dir tests/e2e/aimmodel/fine-tuned"

Next Steps#