Website  ·  Blog  ·  Slack  ·  Apache-2.0

macOS / Metal  ·  Linux / CUDA  ·  Windows (in progress)

Run models larger than your RAM or VRAM. InferSSD builds on llama.cpp to run large mixture-of-experts (MoE) models from SSD, using standard GGUF files, with browser chat and an OpenAI-compatible API. It supports DeepSeek-V4.1-Flash-1017, Muse-Spark-Open, Qwen3.6-35B-A3B and Qwen3.8 Flash Next, plus ordinary RAM inference for Gemma4 E4B.

These are the tested GGUF files, community conversions of the named model releases. DeepSeek's Q8_0 keeps the official bit widths: routed experts stay native 4-bit MXFP4, and FP8 tensors become 8-bit Q8_0. See weight-format details.

Gemma uses the ordinary native loader without SSD expert offload. Before loading, the shared launcher checks that weights, KV cache and working memory fit the memory budget, the machine's physical memory unless a smaller one is configured; the system pages other memory out to make room. On Apple silicon the per-layer embedding table, 2.15 GiB of the 4.80 GiB file, stays in the file and its rows are read from the SSD as tokens use them; the other weights are held in memory. A model that does not fit is rejected. macOS and Windows use the same model selection and launch entry points.

4K prompt tier · 256 output tokens · decode measured over tokens 129–256. All rates are in token/s; higher is faster. The multiple next to each InferSSD rate is its speedup over llama.cpp on the same hardware. On the MacBook Pro, Colibri runs the official Safetensors on CPU; llama.cpp and InferSSD run the GGUF with Metal. See measurement details.

Unified memory is shared by the CPU and GPU. SSD capacities are nominal. See hardware details for SSD models, CPU core layouts, and configuration notes.

N/S: Muse is not currently supported by Colibri.

Install on Apple Silicon macOS or on Linux with an NVIDIA GPU:

curl -fsSL https://github.com/wici-ai/inferssd/releases/latest/download/install.sh | sh

Each numbered release includes checked prebuilt packages for Apple Silicon macOS 15 or later and Windows 11 x64, built from the same tagged commit. Its installers download that exact release without developer tools; see the macOS package and Windows package. Linux builds that release from source for the machine's CPU and GPU, which takes about 2 to 12 minutes. The installer on main downloads the latest published package:

curl -fsSL https://raw.githubusercontent.com/wici-ai/inferssd/main/scripts/install.sh | sh

The installer works in a private environment and uses no sudo. A source build needs Git, Python 3.9 or later, and a C/C++ compiler. On macOS it also needs the Xcode Command Line Tools; Linux also needs systemd with cgroup v2, and the CUDA toolkit on a machine with an NVIDIA GPU. Follow the printed PATH instruction if it shows one.

Then start a model:

# Chat in your terminal
inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

# Or start the web UI and an OpenAI-compatible API at http://127.0.0.1:8080
inferssd serve -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

For Muse, use unsloth/Muse-Spark-Open-816B-A42B-GGUF; for Qwen3.6, unsloth/Qwen3.6-35B-A3B-GGUF; for Flash Next, bartowski/Qwen3.8-Flash-Next-GGUF; for Gemma, google/gemma-4-E4B-it-qat-q4_0-gguf. With serve, open http://127.0.0.1:8080 in your browser, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1. Add --host 0.0.0.0 to reach it from other devices on your network; the API has no authentication, so anyone who can reach the port can use it.

The first run downloads the model files (about 470 GiB for DeepSeek or Muse, 21 GiB for Qwen3.6) to ~/.local/share/inferssd/models and measures your machine. Use --model-dir /path/on/ssd to choose the drive, or -m /path/to/first-shard.gguf for files you already have. There is no memory setting to choose: InferSSD uses the memory available. Stop with Ctrl+C, or /exit in chat.

See the quick start for upgrades and uninstalling.

InferSSD has a release number, 0.1.4 in framework/pyproject.toml, and every build has a label of its own, <yymmdd>.<build>.<commit>: the day (UTC) of the commit it was built from, the branch's commit count up to that commit, and the commit. The installer records the three in engine-resources.json beside the build, from the clone's Git metadata, and so does a local build that runs scripts/engine-resources.py; a source archive has no Git metadata and records none. The macOS and Windows packages carry those of the same tagged commit in each numbered release and are named by that label. inferssd --version prints the release and the label, inferssd --version --json every field:

$ inferssd --version
inferssd 0.1.3 (261106.202.03af979)
$ inferssd --version --json
{"version": "0.1.3", "build": 202, "revision": "03af97…", "date": "2026-11-06", "branch": "main", "label": "261106.202.03af979", "capabilities": {"model_profiles": […], "gpu_backends": ["Metal"]}}

capabilities lists the model profiles the engine launches and, when the installer recorded them, the GPU backends it was built with (an empty list for a CPU-only build). The Windows packages are one per backend, so this is how something running the engine knows which one is installed, and so which package list names its updates.

The engine only says what it is. Whether a build is one it can work with is for whatever runs the engine to decide; the WiCi AI desktop app, for one, installs main, offers its newer builds as updates, and keeps a floor of its own on build. Nothing here refers to it.

• How it works
• Benchmark methodology

Contributions are welcome, including hardware results and improvements to expert I/O, caching, and batching. See the contributing guide.

Built on llama.cpp and ggml, with thanks to their contributors and the model-adaptation community.

InferSSD is licensed under Apache-2.0; bundled llama.cpp/ggml and other third-party components keep their original licenses and notices; see NOTICE. Model weights remain subject to their publishers' licenses.