AirLLM: Run Kimi K3 Models with Just 4GB of VRAM

Project Overview

AirLLM is a Python tool for large language model inference. Its core idea isn’t to cram the entire model into GPU memory at once, but to break the model into finer-grained layers and load only what’s needed during inference. With this approach, you can try running 70B, 235B, 671B, and even more extravagant MoE models like Kimi K3 in very low-VRAM environments.

Kimi K3 has recently drawn a lot of attention for a pretty straightforward reason: it’s an ultra-large MoE model with a huge parameter count, but each token is actually routed to only a subset of experts. AirLLM leverages this directly—by doing expert-level streaming loading, it avoids loading the full model into VRAM all at once.

In this article, we’ll build a minimal project together: first get Kimi K3 inference working, then wrap it into a simple HTTP API. Finally, we’ll also discuss when local low-VRAM inference is a good fit, and when it’s easier to just use a compatible interface like Defapi.

Difficulty: Intermediate | Duration: 30-60 minutes | What You’ll Learn: Understand AirLLM’s low-VRAM inference flow, and run a minimal Kimi K3 inference service

Target Audience Profile

• Developers who want to try ultra-large models on a single card with limited VRAM
• Engineers evaluating inference costs for models like Kimi K3, DeepSeek-V3, and Qwen3-235B
• People building AI app prototypes who want to validate model quality locally first
• Developers who need to wrap a local model as an API, then integrate it with Agents, RAG, or business systems

Core Dependencies and Environment

Kimi K3 is fairly particular about the environment. Let’s clarify the dependency boundaries up front:

Complete Project Structure Tree

Let’s set up a minimal project and keep the structure simple:

airllm-kimi-k3-demo/
├── .env.example
├── requirements.txt
├── run_kimi_k3.py
├── server.py
└── README.md

1. Create a Python Virtual Environment

First, create the directory and the virtual environment:

mkdir airllm-kimi-k3-demo
cd airllm-kimi-k3-demo

python -m venv .venv
source .venv/bin/activate

For Windows PowerShell, activate it like this:

mkdir airllm-kimi-k3-demo
cd airllm-kimi-k3-demo

python -m venv .venv
.venv\Scripts\Activate.ps1

Confirm your Python version:

python --version

You should aim for at least Python 3.10.x. If you have multiple Python versions on your machine, you can explicitly specify it with py -3.10 -m venv .venv.

2. Install AirLLM and Kimi K3 Dependencies

Create requirements.txt:

airllm
accelerate
sentencepiece
safetensors
compressed-tensors
python-dotenv
fastapi
uvicorn[standard]

Install PyTorch first. For CUDA 12.1, use:

pip install --upgrade pip
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Then install the core dependencies:

pip install -r requirements.txt
pip install "transformers>=4.56,<4.57"

Finally, install flash-attn:

pip install flash-attn --no-build-isolation

3. Configure Model Access and Cache Directories

Some Hugging Face models require authentication or authorization. Prepare a .env.example:

HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
HF_HOME=/data/huggingface
AIRLLM_MODEL_ID=moonshotai/Kimi-K3

When using it for real, copy it:

cp .env.example .env

For Windows PowerShell:

Copy-Item .env.example .env

Then fill in your own Hugging Face token. The model ID should match the official repository name as it was actually released. If you’re using a mirror repository or an internally synced repository, you can also change it to the corresponding repo id.

4. Write a Minimal Kimi K3 Inference Script

Create run_kimi_k3.py:

import os

from dotenv import load_dotenv
from airllm import AutoModel


def main() -> None:
    """Load Kimi K3 and run a minimal text generation."""
    load_dotenv()

    model_id = os.getenv("AIRLLM_MODEL_ID", "moonshotai/Kimi-K3")
    hf_token = os.getenv("HF_TOKEN")

    # Control the input length: start with a short prompt to validate the pipeline quickly,
    # so you don’t wait too long on the first verification.
    max_length = 256

    # AutoModel will automatically detect the model type; hf_token is used to access
    # model repositories that require authorization.
    model = AutoModel.from_pretrained(
        model_id,
        hf_token=hf_token,
        profiling_mode=True,
    )

    prompt = "Explain in three sentences why AirLLM can reduce VRAM usage for LLM inference."

    # The tokenizer still follows Hugging Face conventions, so the migration cost is low.
    input_tokens = model.tokenizer(
        [prompt],
        return_tensors="pt",
        return_attention_mask=False,
        truncation=True,
        max_length=max_length,
        padding=False,
    )

    # During AirLLM inference, weights are loaded layer-by-layer. Keep max_new_tokens small
    # for faster validation.
    generation_output = model.generate(
        input_tokens["input_ids"].cuda(),
        max_new_tokens=128,
        use_cache=True,
        return_dict_in_generate=True,
    )

    output = model.tokenizer.decode(
        generation_output.sequences[0],
        skip_special_tokens=True,
    )

    print(output)


if __name__ == "__main__":
    main()

Run:

python run_kimi_k3.py

The first run is usually slower because AirLLM needs to download the model, analyze the weight files, then convert/split it into a format that can be loaded layer-by-layer later. You’ll find the second run is much smoother because the model cache and split results are already prepared.

5. Monitor VRAM and Disk Usage

Open another terminal and watch VRAM:

nvidia-smi -l 1

Focus on these two metrics:

Memory-Usage
GPU-Util

If everything is working, VRAM won’t immediately “explode” like with the traditional transformers.from_pretrained() approach. AirLLM is more like “fetch while walking”—it loads only the layers and the parts of experts you need, then releases them after use.

Then check disk usage:

du -sh "$HF_HOME"

For Windows PowerShell, you can use:

Get-ChildItem $env:HF_HOME -Recurse | Measure-Object Length -Sum

6. Enable Compressed Weights

AirLLM supports using the compression parameter to reduce the weight-loading footprint. We can add 4bit when initializing the model:

model = AutoModel.from_pretrained(
    model_id,
    hf_token=hf_token,
    compression="4bit",  # Use 4-bit weight compression to reduce load size and improve performance in some scenarios
    profiling_mode=True,
)

If you want a more conservative option, try 8bit first:

model = AutoModel.from_pretrained(
    model_id,
    hf_token=hf_token,
    compression="8bit",  # 8bit is more conservative, balancing speed and accuracy
    profiling_mode=True,
)

In most cases, we choose like this:

7. Wrap It as a Simple HTTP API

Once the local script is working, we usually want to integrate it into an application. Below is a minimal service built with FastAPI.

Create server.py:

import os
from typing import List, Literal

from dotenv import load_dotenv
from fastapi import FastAPI
from pydantic import BaseModel
from airllm import AutoModel


class ChatMessage(BaseModel):
    """An OpenAI-style message schema, convenient for later switching to a compatible API."""
    role: Literal["system", "user", "assistant"]
    content: str


class ChatRequest(BaseModel):
    """A minimal chat request that keeps only the fields needed by the tutorial."""
    messages: List[ChatMessage]
    max_tokens: int = 256


class ChatResponse(BaseModel):
    """Return text results and keep it simple for easier debugging."""
    text: str


load_dotenv()

app = FastAPI(title="AirLLM Kimi K3 Demo")

model_id = os.getenv("AIRLLM_MODEL_ID", "moonshotai/Kimi-K3")
hf_token = os.getenv("HF_TOKEN")

# Load the model when the service starts. In real production, you can add health checks and lazy loading.
model = AutoModel.from_pretrained(
    model_id,
    hf_token=hf_token,
    profiling_mode=False,
)


def build_prompt(messages: List[ChatMessage]) -> str:
    """Combine multi-turn messages into a simple prompt; later you can replace it with the model’s official chat template."""
    lines: list[str] = []
    for message in messages:
        lines.append(f"{message.role}: {message.content}")
    lines.append("assistant:")
    return "\n".join(lines)


@app.post("/chat", response_model=ChatResponse)
def chat(request: ChatRequest) -> ChatResponse:
    """Run one Kimi K3 text generation."""
    prompt = build_prompt(request.messages)

    input_tokens = model.tokenizer(
        [prompt],
        return_tensors="pt",
        return_attention_mask=False,
        truncation=True,
        max_length=1024,
        padding=False,
    )

    generation_output = model.generate(
        input_tokens["input_ids"].cuda(),
        max_new_tokens=request.max_tokens,
        use_cache=True,
        return_dict_in_generate=True,
    )

    text = model.tokenizer.decode(
        generation_output.sequences[0],
        skip_special_tokens=True,
    )

    return ChatResponse(text=text)

Start the service:

uvicorn server:app --host 0.0.0.0 --port 8000

Test with a call:

curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Explain in Chinese why MoE models like Kimi K3 are a good fit for expert-level streaming loading."
      }
    ],
    "max_tokens": 200
  }'

Now we have a minimal API. It’s not production-grade yet, but it’s perfect for local experiments, Agent prototypes, prompt debugging, and comparing model behavior.

8. Choosing Between Local Inference and the Defapi Interface

AirLLM’s value is clear: it lets us use ultra-large models with low-VRAM machines, which is especially useful for learning, validation, evaluation, and offline experiments. But for production, you still need to consider throughput, latency, concurrency, model updates, disaster recovery, and operational costs.

If you just want to quickly plug model capabilities into your application—or you need a unified API in OpenAI/Anthropic/Gemini styles—Defapi is often easier. Its advantages are that the pricing is usually about half of the official service, and most common models are compatible with these protocols:

• v1/chat/completions
• v1/messages
• v1beta/models/

For example, we can abstract the local /chat call above into a unified client. During development, use local AirLLM; in production, switch to Defapi:

import os
import requests


def ask_model(prompt: str) -> str:
    """Choose local inference or a Defapi-compatible interface based on environment variables."""
    provider = os.getenv("MODEL_PROVIDER", "local")

    if provider == "defapi":
        response = requests.post(
            "https://api.defapi.org/api/v1/chat/completions",
            headers={
                "Authorization": f"Bearer {os.environ['DEFAPI_API_KEY']}",
                "Content-Type": "application/json",
            },
            json={
                "model": os.getenv("DEFAPI_MODEL", "anthropic/claude-sonnet-4.5"),
                "messages": [{"role": "user", "content": prompt}],
                "temperature": 0.7,
            },
            timeout=60,
        )
        response.raise_for_status()
        return response.json()["choices"][0]["message"]["content"]

    response = requests.post(
        "http://localhost:8000/chat",
        json={
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": 256,
        },
        timeout=300,
    )
    response.raise_for_status()
    return response.json()["text"]

The benefit of this approach is that the boundary is very clear: the local model is responsible for exploration and validation, while Defapi handles stable, unified, low-cost access to online models. For OpenClaw, RAG services, and automated Agents, this dual-mode setup is quite practical.

Troubleshooting FAQ

1. What should I do if flash-attn fails to install?

First confirm that PyTorch can recognize CUDA properly:

python -c "import torch; print(torch.__version__); print(torch.version.cuda); print(torch.cuda.is_available())"

If torch.cuda.is_available() is False, don’t rush to install flash-attn. You should fix PyTorch, the driver, and the CUDA version first. For Kimi K3–class models, it’s recommended to use a CUDA 12 build to avoid CUDA 13 lacking suitable precompiled packages that may cause compilation to fail.

2. What should I do if I get an error about the transformers version?

Pin the version to 4.56.x:

pip uninstall -y transformers
pip install "transformers>=4.56,<4.57"

Then run again:

python run_kimi_k3.py

If you see the model’s remote code failing to load, it’s usually because the transformers version doesn’t match the model code.

3. What should I do if the Hugging Face download fails?

First confirm whether the token works:

huggingface-cli whoami

If you’re not logged in:

huggingface-cli login

You can also rely only on HF_TOKEN from your .env. If the model requires access permission, you need to agree to the terms on the model’s page first.

4. What should I do if I run out of disk space?

On the first run, AirLLM downloads the original weights and also generates split model files. You can move the cache directory to a larger disk:

export HF_HOME=/mnt/models/huggingface

Or configure it in .env:

HF_HOME=/mnt/models/huggingface

If disk space is still tight, you can try AirLLM’s delete_original=True to keep only the converted files:

model = AutoModel.from_pretrained(
    model_id,
    hf_token=hf_token,
    delete_original=True,  # Delete the original weights after conversion to save disk space
)

5. Why is the first run so slow?

This is normal. The first run includes steps like downloading weights, reading indexes, splitting the model, and writing caches. The second run reuses the cache and becomes much faster.

You can enable profiling:

model = AutoModel.from_pretrained(
    model_id,
    hf_token=hf_token,
    profiling_mode=True,  # Output loading and inference time to help locate bottlenecks
)

If most time is spent on disk reads, using a faster NVMe SSD often helps more than upgrading to a larger GPU just to reduce VRAM pressure.

6. What should I do if the VRAM usage doesn’t drop as expected?

First make sure your code isn’t mixing traditional loading methods:

from airllm import AutoModel

model = AutoModel.from_pretrained(model_id)

Don’t mix in the traditional loading methods:

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(model_id)

The latter follows the traditional loading path and can easily fill VRAM directly.

7. What should I do if the API service hangs when concurrency increases?

Local low-VRAM inference is best for sequential experiments. It’s not ideal for directly handling high concurrency. The simplest approach is to limit concurrency:

uvicorn server:app --host 0.0.0.0 --port 8000 --workers 1

If you need more reliable production calls, route requests online to Defapi and keep local AirLLM for evaluation and debugging.

Further Reading / Advanced Directions

• Compare load time, VRAM usage, and output quality for Kimi K3, DeepSeek-V3, and Qwen3-235B under AirLLM.
• Modify the FastAPI service into an OpenAI-compatible /v1/chat/completions endpoint.
• Integrate Defapi and use v1/chat/completions or v1/messages to unify management of online model calls.
• Add a queue to the AirLLM service to prevent multiple requests from competing for disk I/O and the GPU at the same time.
• In OpenClaw or other Agent frameworks, configure both local and cloud models, then dynamically choose based on task complexity.