veloresearch/kineticДокументация

The open engine for running AI on your own computer.

Written in Rust. It reads GGUF files, the format llama.cpp uses, and runs the models it supports on the CPU and on NVIDIA, AMD and Intel GPUs, from one binary. Fast, and no speed-up ever changes the answer.

Documentation   Technical report   Releases   veloresearch.com

1. Install Kinetic. On Windows, in PowerShell (no administrator rights needed):

irm https://raw.githubusercontent.com/Veloresearch/kinetic/main/scripts/install.ps1 | iex

On Linux:

curl -fsSL \
  https://raw.githubusercontent.com/Veloresearch/kinetic/main/scripts/install.sh | sh

2. Download a model. LFM3 1.7B (1.1 GB) is quick even on a laptop CPU:

kinetic pull LiquidAI/LFM3-1.7B-GGUF:Q4_K_M

3. Chat with it.

kinetic chat LiquidAI/LFM3-1.7B-GGUF

Not sure which model to take? Run kinetic alone: it checks your computer, suggests two or three models that run well on it (the ones you already have first), and one key press downloads it and opens a chat. A GGUF file you already have works too if its model family is supported: kinetic chat path/to/model.gguf.

_{Real time, same laptop, same GGUF file, same prompt. Every recording on this page is the output of the command it shows.}

Same GGUF files, same settings, one laptop (Ryzen 5 6600H, RTX 3060 Laptop 6 GB), llama.cpp b12001:

Every answer is token for token what plain decoding gives: the speed comes from doing less work, never from changing the output. All benchmarks.

sha256sum -c SHA256SUMS --ignore-missing      # Windows: Get-FileHash <archive>
tar xzf kinetic-0.1.0-linux-x86_64-cuda.tar.gz

cargo install --locked --git https://github.com/Veloresearch/kinetic kinetic-cli

A first model. Qwen3-0.6B (0.6 GB) is a quick try on any laptop; Qwen3.5 4B in Q4_K_M (2.8 GB) is a capable everyday model on a GPU with 4 GB or more. kinetic pull Qwen/Qwen3-0.6B-GGUF downloads one, and kinetic models lists what Ollama, LM Studio and the Hugging Face cache already hold, runnable by name: kinetic run qwen3:0.6b "Hi".

_{Back to top}

What makes it fast, Switching from llama.cpp, Chat in the terminal, Use it from your app, Supported models, Devices, Benchmarks, Privacy, FAQ, Roadmap, Contributing.

On a personal computer, generating a token is limited by memory bandwidth: a dense model reads all of its weights once per token. Kinetic works around that limit without touching the answer. Every technique below is on by default, needs no tuning, and gives the output plain decoding gives. The docs cite the research behind each.

To measure one, turn it off: --off rad1,warm-start,zero-wait,shortlist,torque,convoy works on run, chat, serve and bench.

Kinetic guesses the next few tokens from the conversation, from earlier requests and from the model's own prediction block, then checks every guess in one pass over the weights, so each correct guess is a token for free. CPU, CUDA and Vulkan.

Chat clients resend the whole conversation with every request; Warm Start resumes from the exact state an earlier prompt left behind, in memory or on disk, so only the new tokens run, also on recurrent models such as Qwen3.5 and LFM3. CPU, CUDA and Vulkan.

States are kept in memory (up to 1 GiB) and in your cache directory (up to 2 GiB, oldest dropped first), so a restarted server picks up where it left off. --cache-dir DIR moves them, --no-prompt-cache turns Warm Start off.

A pasted document is processed while you are still typing the question, so when you press Enter only the last few words are left. CPU and CUDA.

A 22,631-token document pasted, then an 80-character question typed, on Qwen3.5 4B on the RTX 3060 Laptop:

It gets that close to zero when the typing takes at least as long as the document does to process; a question typed in three seconds still saves those three seconds. Editing text that already ran goes back only as far as the edit. kinetic chat does it on its own; apps send the draft ahead with POST /v1/prefill.

A decode step is hundreds of small GPU kernels; Torque records the step once as a CUDA graph (a replayed command buffer on Vulkan) and launches it with one call per token.

With kinetic serve model.gguf --parallel 4 (llama.cpp's -np 4), up to four requests share every pass over the weights, and each answer is exactly what that request would get alone. CPU, and CUDA when the whole model is on the GPU; elsewhere requests take turns.

Choosing the next token reads an output layer of up to a quarter of a million rows; Shortlist bounds every row's score from a small copy and computes exact scores only for the rows that can still win. CPU.

Long prompts run as tiled int8 matrix products, so each weight block is unpacked once and used for many tokens. CPU (AVX2) and CUDA.

A mixture-of-experts model keeps the experts in use on the GPU and the rest in system memory, and runs each missing expert on the CPU or copies it over, whichever is cheaper at that moment. CUDA; see models bigger than your GPU.

All of the above rests on one rule: every row of a batch is summed in the same order as that token would be on its own, so checking eight guesses at once or sharing a pass with three other users gives the same bits as plain decoding. A conversation resumes only from a state a fresh run of the same prompt would also reach. The test suite checks this on every supported architecture, and kinetic bench checks it on your machine.

_{Back to top}

Your GGUF files work unchanged when their model family is supported, and one binary stands in for llama-server, llama-cli, llama-bench, llama-quantize and llama-perplexity, with their flags. Pick your case; each is one step.

For example kinetic serve -m model.gguf -c 8192 -ngl 99 -fa on --jinja works as written. One port answers the OpenAI API under /v1, llama-server's own API (/completion, /tokenize, /detokenize, /props, /health, /embedding, and /slots and /metrics in its format) and Ollama's under /api, so clients keep their settings. quantize writes the same bytes as llama.cpp for Q4_K_M, Q5_K_M, Q6_K, Q8_0 and Q4_0 (no --imatrix yet); perplexity runs on the CPU, with --kl-divergence. --lora, --lora-scaled, --api-key, --alias, --mlock, --no-mmap and --chat-template-file work as in llama.cpp. Flags for what Kinetic does by itself, like -fa, --jinja or -b, are accepted; the few with no equivalent stop with a line saying what to use instead.

pip install llama-cpp-kinetic makes import llama_cpp run on Kinetic. libllama itself is not replaced; Kinetic has its own C API. Details: Switch from llama.cpp and every llama.cpp flag.

_{Back to top}

kinetic chat LiquidAI/LFM3-1.7B-GGUF
kinetic run model.gguf "Explain recursion in two sentences."

chat keeps a conversation and processes only your new message each turn; run answers one prompt (or what you pipe in) and ends with a line giving the speed and the device:

Markdown. Answers are drawn as they stream: headings, lists, tables and code blocks with highlighting. /raw shows the plain text for copying.

Images. Qwen3.5 and Gemma 4 see images through their vision projector, stored with the model or given with --mmproj. /image PATH attaches a file; /image alone takes the image on the clipboard.

Commands. /think turns a model's reasoning on or off, /system sets the system prompt, /clear starts over, Ctrl+C stops an answer. Sampling defaults come from the model file; --temp 0 always takes the likeliest token and --seed makes a sampled answer repeatable. All options: command line.

Every command

kinetic run model.gguf "Name three rivers." --temp 0.8 --receipt answer.json
kinetic verify-receipt answer.json

Because nothing outside the request changes the output, an answer can come with a receipt: the model file's hash, the build, the device class, the exact prompt tokens, every sampling setting with the seed and the output tokens, optionally signed with an Ed25519 key. verify-receipt reruns it and reports identical or the first token that differs. Ask for one with --receipt, /receipt in chat or "receipt": true on the API.

Ten runs of one prompt at temperature 0.8 with a fixed seed, under load from three other clients, give ten identical answers. What a receipt does and does not prove: Receipts.

_{Back to top}

kinetic serve model.gguf

The server speaks the OpenAI API on http://127.0.0.1:8080/v1. Any OpenAI client, editor plugin or agent framework that takes a base URL works.

Python

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1",
                api_key="local")
stream = client.chat.completions.create(
    model="model",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")

JavaScript

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://127.0.0.1:8080/v1",
  apiKey: "local",
});
const reply = await client.chat.completions.create({
  model: "model",
  messages: [{ role: "user", content: "Hello!" }],
});
console.log(reply.choices[0].message.content);

Tool calls and structured output. Models call your functions in the OpenAI shape. response_format with a JSON Schema, or a GBNF grammar, holds the answer to that shape while it is generated, so it always parses; tool_choice set to "required" or to one function is enforced the same way.

Tool calls: Qwen3 0.6B asks for the weather in Paris.

JSON Schema: the reply always matches the schema.

Streaming. Answers arrive as server-sent events, token by token. Embeddings and reranking. An embedding model (BGE, nomic-embed, EmbeddingGemma, Qwen3-Embedding) answers /v1/embeddings; Qwen3-Reranker answers /v1/rerank. Images. With --mmproj, chat requests carry images for Qwen3.5 and Gemma 4, as inline data only: the server never fetches a URL.

Every endpoint and field: Server API.

Without a server. Kinetic is also a library for Python (pip install kinetic-engine), Rust (the kinetic crate) and C (include/kinetic.h: load a model, chat with a streaming callback, embed):

import kinetic

model = kinetic.Model("model.gguf")
print(model.chat([{"role": "user", "content": "Hello!"}], max_tokens=64))

Examples: Python, C and Rust.

_{Back to top}

Images need the model's vision projector (mmproj): qwen3vl_merger for Qwen3.5, gemma4v for Gemma 4.

Weights: F32, F16, BF16, Q8_0, Q4_0, Q4_1, Q5_0, Q5_1, the K-quants (Q2_K to Q6_K), the IQ formats (IQ1_S to IQ4_XS), TQ1_0, TQ2_0, MXFP4 and PrismML's PQ2_0. Which run on which device, and what to do with a file that does not load: Models and formats.

kinetic info shows what a file holds before you load it:

Kinetic reads the rotary scaling a model file asks for (linear or YaRN) and lets you override it, as in llama.cpp; -c may then reach the stretched context:

kinetic serve model.gguf -c 131072 --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768

--rope-freq-base sets the rotary base for NTK-aware scaling.

A .ktc file holds the weights, the tokenizer and the chat template in one file, with a manifest of BLAKE3 hashes for every section, an optional Ed25519 signature and an optional execution profile for a kind of hardware. Every command that takes a .gguf takes a .ktc, and kinetic verify checks every hash and the signature. Format: FORMAT.md.

_{Back to top}

Vulkan runs Llama, Qwen2.5, Qwen3, Qwen3.5 and LFM3 today. --device auto (the default) tries CUDA, then Vulkan, then the CPU, and prints one line when it skips a device and why; --device cuda, vulkan or cpu picks one.

CPU: LFM3 1.7B with --device cpu.

Vulkan: Qwen3 0.6B with --device vulkan.

Dense models. Kinetic puts as many of the last layers on the GPU as fit and runs the first ones on the CPU; only one hidden state per token crosses between them. --gpu-layers N (llama.cpp's -ngl N) caps the layers on the GPU, and the status line says how the model was split:

Mixture-of-experts models. With Flywheel, Qwen3.5-35B-A3B (12 GB in Q2_K_XL) runs on a 6 GB laptop GPU: attention and the shared weights on the GPU, a cache of experts in the memory left over, every other expert in system RAM. For speed, the whole file should fit in RAM next to your other programs.

More on memory, threads and drivers: Devices and offload.

_{Back to top}

Tokens per second, higher is better, against llama.cpp b12001 on the same files and settings, on a Ryzen 5 6600H with an RTX 3060 Laptop GPU (6 GB). "Edit" is a greedy request that rewrites a file, with llama.cpp's own speculative decoding on (draft-mtp for Qwen3.5, ngram-simple for LFM3); Kinetic's output is checked to be identical to plain decoding.

GPU

Bigger than the GPU, Qwen3.5 4B Q4_K_M, generate:

CPU only, no graphics card used:

Several users at once, serve --parallel 4 against llama-server --parallel 4 on the GPU, total tokens per second, every request with its own prompt. Each of Kinetic's answers is byte-identical to the same request run alone; llama-server's are not.

The six GPU rows were measured on an idle machine; the layer split, CPU and several-users rows with other programs running, so they are on the low side for both programs. GPU prompts still trail llama.cpp on LFM3 and on very short prompts, and are being tuned.

Measure your own machine. kinetic bench model.gguf --compare runs llama.cpp's llama-bench on the same file with the same settings and prints the comparison as a Markdown table, ready to paste into an issue; the LFM3 GPU rows and the CPU table above are its output.

Method and scripts/bench_vs_llamacpp.py: Benchmarks.

_{Back to top}

Kinetic never connects to the network, except when you run kinetic pull. No telemetry, no update checks, no accounts.

• The server listens on 127.0.0.1 unless you pass --host, and warns when you do; --api-key requires a key on every route but /health.
• Requests from web pages are refused, so a website you visit cannot use your local model.
• Warm Start keeps prompt states in your cache directory, up to 2 GiB; delete kinetic/warm there to clear them, or run with --no-prompt-cache.

_{Back to top}

More: FAQ and troubleshooting.

_{Back to top}

The full list: roadmap.

Bug reports, benchmarks on your hardware, fixes and new model support are welcome. Each model family is one file on top of blocks every family shares, and every device implements those blocks:

Start with CONTRIBUTING.md and the contributor guide, which walks through adding an architecture and a weight format step by step; the good first issues are a good start. Security problems go privately to [email protected]; see SECURITY.md.

Kinetic is licensed under the Apache License 2.0. Third-party material is listed in NOTICE. "Kinetic" and its logo are trademarks of Velocity; see TRADEMARKS.md.