TOKENFOLD

More context. Fewer tokens. Exact by default.

Deterministic context compression for LLMs. Cut JSON tokens by up to 68% with exact recovery, or exceed 90% with opt-in, recoverable pruning.

Tokenfold Core delivers model-free structural compression.

Tokenfold Select ranks the context that matters most when token budgets get tight.

Install | Quick start | How it works | MCP & agents | Core | Select | Benchmarks | Safety

pip install tokenfold        # Python library
npm install tokenfold        # Node.js / TypeScript library
cargo install tokenfold-cli  # Standalone CLI
cargo add tokenfold-core     # Rust library

These installs provide Core. For optional query-aware model scoring, see Select setup; its models and runtime setup are separate.

• Core (default): folds repeated structure without removing data. The compact payload decodes to the original JSON value.
• Recoverable pruning (opt-in): model-free rules rank eligible JSON rows; omissions are stored locally before replacement with $tf_ref handles. This is the path behind the 96.3% showcase, not Select.
• Select (separate): a model scores groups against your query; your allocator keeps selected original text under a token budget. Select does not automatically store omissions or create recovery handles.

// Input (conceptual illustration - 2 rows shown; small inputs like this
// pass through unchanged under the never-larger guard)
[{"id":1,"role":"admin"},{"id":2,"role":"member"}]

// Tokenfold output on larger uniform arrays -> still JSON
{"__tf_cols__":["id","role"],"__tf_rows__":[[1,"admin"],[2,"member"]]}

Send the folded JSON directly to the model: column names label each value's position in every row. Decode only when your application needs the original object shape again.

• Targets structured JSON, not prose. Folds repeated keys and columns mechanically rather than using semantic guesswork.
• Model-free and deterministic. Zero models running in Core and zero GPU overhead.
• Verified by exact decode. Every lossless transform is verified by an in-memory round trip before emitting; lossy pruning is strictly opt-in and recoverable.

%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#f8fafc", "primaryTextColor": "#0f172a", "primaryBorderColor": "#94a3b8", "lineColor": "#64748b", "edgeLabelBackground": "#f8fafc", "fontFamily": "Arial", "fontSize": "15px"}, "flowchart": {"curve": "linear", "padding": 16, "nodeSpacing": 40, "rankSpacing": 32}}}%%
flowchart LR
    accTitle: Lossless first, pruning only when opted in
    accDescr: Build a verified lossless baseline. Consider pruning only when opted in, eligible, and still needed. Store omissions before removal and use pruning only if it beats the lossless baseline; otherwise keep that baseline.
    Lossless["Build lossless JSON baseline<br/>Verify folds; retain savings"] --> OptIn{"Pruning enabled,<br/>eligible and needed?"}
    OptIn -->|No| Keep("Lossless payload<br/>+ receipt")
    OptIn -->|Yes| Prune["Rank eligible rows<br/>Store before removal"]
    Prune -->|No further savings| Keep
    Prune -->|Fewer tokens than baseline| Emit("Pruned payload<br/>+ markers and receipt")

    classDef core fill:#eff6ff,stroke:#2563eb,stroke-width:2px,color:#1e3a8a
    classDef optional fill:#f8fafc,stroke:#64748b,stroke-dasharray:5 4,color:#334155
    class Lossless,Keep core
    class Prune optional

These figures use exact o200k_base counts in the balanced preset. Versioned inputs and provenance live in tests/fixtures/readme_metrics.json. Recoverable pruning is separate, opt-in extended tooling.

After pip install tokenfold, try this self-contained example. Thirty uniform records provide enough repeated structure to demonstrate lossless folding:

import json
from tokenfold import InputFormat, compress

records = [{"id": i, "role": "member", "status": "active"} for i in range(30)]
result = compress(json.dumps(records), format=InputFormat.JSON)
print(f"{result.report.original_tokens} -> {result.report.compressed_tokens} tokens "
      f"({result.saved_pct():.1f}% saved)")

Send result.payload (bytes) to the model. Use tokenfold.decode when your application needs the original JSON shape again. Small or irregular inputs may pass through unchanged under the never-larger guard.

Install the CLI with cargo install tokenfold-cli or use a checksummed GitHub Release, then:

tokenfold inspect payload.json --format json
tokenfold compress payload.json --format json --output payload.compact.json

Start the local proxy once, then point an existing client at it. The proxy forwards provider-shaped requests to your fixed upstream and applies Core's lossless transforms before forwarding.

cargo install tokenfold-proxy
tokenfold-proxy --upstream https://api.openai.com

from openai import OpenAI

# Your client's Authorization header is forwarded to upstream unchanged -
# no credential reconfiguration needed.
client = OpenAI(base_url="http://127.0.0.1:8787/v1")
# Existing calls work unchanged: client.chat.completions.create(...)

The proxy listens on loopback (127.0.0.1:8787) by default. Provider envelopes are minified, not columnar-folded.

Works out-of-the-box with any client or framework that accepts a custom base URL (LangChain, LlamaIndex, LiteLLM, Instructor, or direct HTTP clients).

%%{init: {"theme": "base", "themeVariables": {"fontFamily": "Arial", "fontSize": "15px", "actorBkg": "#f8fafc", "actorTextColor": "#0f172a", "actorBorder": "#94a3b8", "actorLineColor": "#94a3b8", "signalColor": "#64748b", "signalTextColor": "#334155", "activationBkgColor": "#eff6ff", "activationBorderColor": "#2563eb", "sequenceNumberColor": "#ffffff"}, "sequence": {"mirrorActors": false, "actorMargin": 56, "messageMargin": 36}}}%%
sequenceDiagram
    accTitle: Proxy request and response lifecycle
    accDescr: The client sends a provider-shaped request to Tokenfold Proxy, which applies lossless transforms and forwards the compact request with Authorization unchanged. The upstream response returns through the proxy.
    autonumber
    participant C as Client / Agent
    participant P as Tokenfold Proxy
    participant U as Upstream LLM
    C->>+P: Provider-shaped request
    P->>P: Apply lossless transforms
    P->>+U: Compact request + unchanged Authorization
    U-->>-P: Completion / action
    P-->>-C: Response

Provider message formats (openai_json, anthropic_json) use minification only; columnar folding applies to generic JSON (--format json). The default proxy path follows this rule. Explicit --observations enables lossless compression of eligible JSON tool-result strings on /v1/chat/completions without folding the provider envelope; see observation compression.

Runnable examples, one per surface, all under examples/:

python examples/quickstart.py      # Python: compress messages, request bodies, JSON data
node examples/quickstart.mjs       # TypeScript/Node: compress, inspect, read the receipt
cargo run -p tokenfold-core --example quickstart   # Rust: the embedded core API
python examples/lossy_pruning.py   # CLI: opt-in recoverable pruning, end to end

examples/quickstart.ipynb is the guided tour of the whole Python surface: everything quickstart.py shows, plus previews, budgets and presets, provider payloads, recoverable pruning with retrieval, and where Tokenfold Select fits.

Coding agents can exhaust context limits on large shell outputs. Tokenfold runs as a stdio MCP server exposing tokenfold_compress, tokenfold_inspect, tokenfold_retrieve, and tokenfold_stats, with pre-configured filters for Git status, diffs, build output, and test logs before they consume agent context. tokenfold init currently supports Claude Code; Codex uses manual MCP registration:

tokenfold init --agent claude-code
tokenfold doctor --agent claude-code

# Or connect manually in Codex:
codex mcp add tokenfold -- tokenfold mcp serve

tokenfold init --agent claude-code merges a project-scoped .mcp.json entry without replacing other servers; tokenfold doctor --agent claude-code verifies it. tokenfold_inspect is side-effect-free (never returns a modified payload); tokenfold_retrieve retrieves by hash or marker (report references are reserved but not yet resolvable). See docs/configuration.md for tested MCP JSON/TOML and every environment override. Trusted filters for Git, build, and test output are available through tokenfold filters list.

Semantic prompt compressors such as LLMLingua use a smaller language model to identify and remove less-important prompt tokens. Tokenfold solves a different problem first: repeated keys, values, schemas, rows, and other structural redundancy. Core needs no model, is deterministic, and verifies an exact decode before accepting a lossless transform. Core does not inject prompt guidance or fine-tune models.

Tokenfold Select is Tokenfold's complementary query-aware offering: when structure ends, it ranks what matters under a token budget.

When structure ends, rank what matters.

Tokenfold Select v3 is an Apache-2.0, 149.6M-parameter joint-group scorer built on the full 22-layer Ettin encoder. It scores groups against the query and their neighbors within each window; your allocator keeps selected source text unchanged, protects required content, and enforces the exact token budget. The model returns scores, not a compressed string or calibrated probabilities. Core remains model-free and neither loads nor invokes Select.

Use the v3 model card for the included custom loader and explicit CPU/FP32 or CUDA/BF16 setup. The older Granite/LoRA model is Select v1, not the v3 architecture or loader.

from pathlib import Path
from huggingface_hub import snapshot_download
from tokenfold_select import OnnxScorer

root = Path(snapshot_download(
    "snchimata/tokenfold-select-v2",
    revision="78736d9afea819322f212d6d53eedcf68ac7a39b",
    allow_patterns=["model.onnx", "model.onnx.data", "onnx-manifest.json",
                    "tokenizer/*", "runtime/*", "LICENSE", "NOTICE.md"],
)).resolve()
scorer = OnnxScorer(root, development=True)
response = scorer.score({
    "schema_version": 1, "model_revision": scorer.revision,
    "query": "Which delivery city?",
    "groups": [{"id": "delivery", "text": "Delivery city: Paris"}],
})
print(response)  # scores only; your controller owns protection and allocation

from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer

base_id = "ibm-granite/granite-embedding-reranker-english-r2"
repo_dir = Path(snapshot_download("snchimata/tokenfold-select"))
adapter_dir = repo_dir / "adapter"
tokenizer = AutoTokenizer.from_pretrained(adapter_dir)
base = AutoModelForSequenceClassification.from_pretrained(
    base_id,
    dtype=torch.float32,
)
model = PeftModel.from_pretrained(base, adapter_dir).eval()

def score(query: str, spans: list[str]) -> list[float]:
    if not spans:
        return []
    encoded = tokenizer(
        [query] * len(spans),
        spans,
        padding=True,
        truncation=True,
        max_length=8192,
        return_tensors="pt",
    )
    with torch.no_grad():
        output = model(
            input_ids=encoded["input_ids"],
            attention_mask=encoded["attention_mask"],
        )
    return output.logits.view(-1).float().tolist()

Saved results from the published v3 benchmark snapshot, pinned to model-card revision 317d3424d3de78c1a708c42a47a7812ab877f1a8. These measurements include the scorer and whole-group allocator, not the scoring model alone. Core's lossless benchmarks remain separate below.

Select v3 removed 50.0-60.0% of the original context tokens in these five saved development examples. Both models had the same exact o200k_base output cap for each workload; retained V3 groups were copied without rewriting.

One exposed source per workload; JSON and logs share a source population. Token reduction does not establish answer accuracy or semantic preservation. Six of ten historical Kompress cells refused the budget across the 25%/50% screen; refusals and raw fallbacks are inconclusive, not V3 wins. The full ten-cell history retains every result.

Resident-warm RTX 5080 research pipelines, FP32 parameters/BF16 compute; startup excluded. Separate time blocks, not controlled-idle or paired trials. These are descriptive development observations, not representative quality, population p50/p95, or superiority qualification.

Exact o200k_base token counts, original input versus Tokenfold's lossless output across the six-fixture Headroom corpus:

See the Headroom methodology for environment setup, pinned comparator revision, and reproduction.

In aggregate across the seven fixtures:

On irregular structures, TOON can use more tokens than compact JSON. Tokenfold's never-larger guard retains compact JSON when an eligible lossless transform would not reduce the exact token count.

All counts use exact o200k_base recounts. Tokenfold passed exact recovery on all six Headroom fixtures and all seven TOON fixtures; Headroom's emitted JSON matched the input value on 4/6 fixtures (the semi-uniform incident feed and the deeply nested OpenAI payload differed), so its aggregate reduction is not a lossless result. See the Headroom methodology and TOON methodology for the versioned corpora, pinned comparator revisions, and full fixture-level results.

These additional capabilities extend Core with recoverable pruning and an explicit TOON output codec.

While Tokenfold Core defaults to lossless structural folding, heterogeneous array feeds (such as long logs or search results) benefit from recoverable pruning when strict context budgets apply:

Lossless folding has a ceiling on heterogeneous arrays. --prune ranks rows, keeps the strongest signals, and replaces selected rows with compact {"$tf_ref": {...}} handles. A row leaves the payload only after the local store accepts it.

tokenfold compress examples/incident_feed.json --format json \
  --prune --keep-ratio 0.35 --output feed.compact.json

On the bundled 40-event feed:

The planted 503 with success: false and retries: 7 survives every shown setting because typed failure signals outrank position and length - at --keep-ratio 0.05 it is the only row kept. Long rows amortize marker overhead further: the 100-result showcase reaches 96.3% fewer tokens (--keep-ratio 0.02, examples/max_showcase.json; see tests/fixtures/readme_metrics.json).

Fetch a dropped row using the same --retrieval-store and --retrieval-namespace as the compression run:

tokenfold compress examples/incident_feed.json --format json \
  --prune --keep-ratio 0.35 --output feed.compact.json \
  --retrieval-store ./store --retrieval-namespace incident-demo
tokenfold retrieve cb13cc59cca0c218c579cd1d4b3cbab58d6dea265eb995cc9c00faf0cd0a6856 \
  --retrieval-store ./store --retrieval-namespace incident-demo
# {"seq":1,"ts":"2026-08-15T00:01:11Z","subsystem":"index-writer",...}

What the flags mean:

• --keep-ratio is an aggression hint over eligible array items. Lower keeps fewer rows; it is not a whole-document guarantee.
• --target-tokens is the whole-document goal. Tokenfold stops when it reaches the target losslessly and reports best_effort when the safe transform set cannot reach it (unreachable when protected content alone exceeds the target).
• --preserve <path> protects a named array; nested paths conservatively protect their nearest eligible ancestor.
• Generic JSON only: lossy pruning does not run on OpenAI or Anthropic message payloads. Pass the same --retrieval-store/--retrieval-namespace to tokenfold retrieve that the compress run used.
• Storage is fail-closed: refused rows stay inline, and detected secret-shaped bytes are never persisted. Lossy pruning requires the filesystem retrieval backend with TTL >= 24h (memory backends and near-immediate expiry are rejected); $tf_ref is reserved.

Preview the projected savings with no store writes:

tokenfold inspect examples/incident_feed.json --format json \
  --prune --keep-ratio 0.35

The same flags, and the same fail-closed contract, from Python and TypeScript. Use a throwaway store directory while experimenting and reuse it for retrieval:

import json
from pathlib import Path
from tokenfold import InputFormat, PruningPolicy, compress, retrieve

store = Path(".tokenfold-readme-store")
feed_bytes = Path("examples/incident_feed.json").read_bytes()
pruning = PruningPolicy(
    keep_ratio=0.35,
    retrieval_store=store,
    retrieval_namespace="readme-demo",
)
result = compress(feed_bytes, format=InputFormat.JSON, pruning=pruning)
payload = json.loads(result.payload)
marker = next(e["$tf_ref"] for e in payload["events"] if "$tf_ref" in e)
original = retrieve(
    marker, retrieval_store=store, namespace="readme-demo"
)  # any dropped row, verbatim

Compact Tokenfold JSON remains the default. For a TOON-aware consumer, request TOON explicitly and decode it through the same verified codec:

tokenfold compress examples/toon_metrics.json --format json \
  --preset conservative --encoding toon --output payload.toon
tokenfold decode payload.toon --from toon --output payload.json

TOON is available only for generic JSON and never activates implicitly. The receipt reports the exact encoding delta and warns when TOON is larger than the already-compressed JSON representation.

• Never larger: Core keeps a transform only when exact recounting shows a reduction; a lossy branch must also beat the lossless result.
• Reversible structure: JSON folds must pass an exact round trip.
• Protected content: provider system messages and latest-user content are held behind format-aware safety gates.
• Clear provenance: exact tokenizer counts, heuristics, and extrapolations are labeled separately.
• Actionable receipts: every result lists savings, transforms, warnings, retrieval state, and final status.
• Local control: detected secrets are redacted before reports or storage; policy learning changes configuration only with --apply.

Exact decode proves data recovery, not unchanged downstream task accuracy. Validate folded payloads with your representative workload.

The offline fidelity gate uses deterministic lexical-overlap, critical-token, and containment proxies. Those checks catch regressions but do not establish semantic equivalence or downstream task success. Runtime quality fields are absent unless a versioned evaluator has supplied data; applications should run their own representative task evaluation before enabling lossy pruning.

# 96.3% maximum-compression showcase, mixed-feed curve, retrieval, and preserve
python examples/lossy_pruning.py

# Lossless transform benchmarks
cargo bench -p tokenfold-core

# Head-to-head provider benchmark (setup and pinned dependency command)
# See eval/research/README.md

# Small exact-token CLI example
cargo run --release --locked -p tokenfold-cli -- \
  inspect examples/api_response.json --format json

The bundled 30-record API response reports 3,812 -> 1,376 tokens, a 63.9% lossless reduction. Benchmark sources and thresholds live in CHANGELOG.md and crates/tokenfold-core/benches/THRESHOLDS.toml.

Issues and pull requests are welcome. Run the relevant checks before opening a PR:

cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
python eval/run_fidelity.py --gate --profile smoke-first-consumer
python eval/test_baseline_fixture_contract.py
python eval/audit_quality_sample.py --check
cd packages/tokenfold && npm ci && npm test

Apache-2.0

Start folding today

Start with one representative payload, inspect the receipt, and see how many tokens your application can stop sending today.

If Tokenfold earns a place in your stack, a star on GitHub helps the next team find it.