Version

Z.ai/GLM-4.5-Air

A 106B MoE served from a GGUF checkpoint via llama.cpp on a single A100.

On this page

A 106B MoE served from an Unsloth GGUF checkpoint via llama.cpp instead of vLLM, on a single A100 40 GB. Modelplane treats the engine as any OpenAI-compatible container, so the only changes from a vLLM deployment are the image and args: the container is still named engine and listens on :8000. vLLM can’t load this Unsloth quantization format. llama.cpp can, and -hf pulls the checkpoint straight from Hugging Face at startup, so a one-time deployment needs no ModelCache.

The model is bigger than one A100’s VRAM, so --n-cpu-moe offloads the MoE expert tensors to host RAM and the GPU runs the active path and KV cache. That’s how a 106B model fits one A100 instead of a multi-GPU node. Apply the platform side first, then the ML side. The GKE InferenceCluster carries a GCP project placeholder to edit before applying.

Validated deployments

MoE 106B A12B 8,192 ctx llama.cpp

Cloud

AWS Nebius

GPU

L4 24G A100 40/80G H100 80G H200 141G 1× per node

Serving mode

Standalone LeaderWorker PrefillDecode

Precision

Engine

vLLM SGLang llama.cpp

Image

ghcr.io/ggml-org/llama.cpp:server-cuda

Manifests

docs/manifests/recipes/glm-4.5-air

Platform

# A single A100 40GB on GKE. GLM-4.5-Air is a ~106B MoE; a 4-bit GGUF doesn't
# fit one A100's VRAM, but llama.cpp offloads the expert tensors to host RAM
# (see the ModelDeployment's --n-cpu-moe), so the GPU only holds the active path
# + KV cache. One A100, not a multi-GPU node.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: gke-a100-40-1x
spec:
  description: "GKE a2-highgpu-1g, 1x NVIDIA A100 40GB"
  provisioning:
    provider: GKE
    gke:
      machineType: a2-highgpu-1g
      diskSizeGb: 200
      accelerator:
        type: nvidia-tesla-a100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
    capacity:
      memory: { value: "40960Mi" }

# A GKE cluster with a single A100 node offering the class above.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gke-a100
  labels:
    modelplane.ai/region: us-west
spec:
  cluster:
    source: GKE
    gke:
      project: my-gcp-project   # set to your project
      region: us-west1
  nodePools:
  - name: gpu-a100
    className: gke-a100-40-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
    zones:
    - us-west1-b

curl -fsSL https://docs.modelplane.ai/examples/recipes/glm-4.5-air/inference-cluster.yaml \
  | sed 's/my-gcp-project//' \
  | kubectl apply -f -

Deployment

# GLM-4.5-Air (~106B MoE) served from an Unsloth GGUF via llama.cpp instead of
# vLLM, on a SINGLE A100. Modelplane treats the engine as any OpenAI-compatible
# container, so the only changes from a vLLM deployment are the image and args:
# the container is still named `engine` and listens on :8000. vLLM can't load
# Unsloth's UD- dynamic quants; llama.cpp can, and `-hf` pulls the quant from
# HuggingFace at startup (no ModelCache needed for a one-off).
#
# The model is bigger than one A100's VRAM, so --n-cpu-moe offloads the MoE
# expert tensors to host RAM; the GPU runs the active path. That's how a 106B
# model fits one A100 instead of a multi-GPU node. --port 8000 because llama.cpp
# defaults to 8080 and Modelplane scrapes 8000.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: glm-air
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: glm
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("35Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: ghcr.io/ggml-org/llama.cpp:server-cuda
                args:
                - "-hf"
                - "unsloth/GLM-4.5-Air-GGUF:IQ4_XS"
                - "--host"
                - "0.0.0.0"
                - "--port"
                - "8000"
                - "-ngl"
                - "999"
                - "--n-cpu-moe"
                - "99"
                - "--jinja"
                - "-c"
                - "8192"

# One OpenAI-compatible endpoint for the deployment. Read its public address:
#   kubectl get ms glm-air -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: glm-air
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: glm-air