Website: smolcluster.com

• The fastest way to see smolcluster's training algorithms in action. grove is a terminal dashboard that allows you to discover and connect to both mac-based and Linux-based nodes seamlessly (NO SSH REQUIRED!), giving you a live unified view across every node in your cluster — no config files, no IPs, no extra services.

• Workers auto-discover the coordinator via AirDrop/mDNS on Mac or TCP/mDNS on Jetson and Linux. Every training algorithm (FSDP, ClassicDP, EDP, SyncPS, EP, GRPO) reports into it automatically — no code changes needed.

For full setup instructions see the quickstart guide. For grove internals and transport options see the grove README.

• Distributed Training Algorithms: Fully Sharded Data Parallel (ZeRO-optimized), Classic Data Parallelism (All-Reduce), Elastic Distributed Parallelism, Synchronous Parameter Server (SyncPS), Expert Parallelism (EP), and Model Parallelism
• Heterogeneous Hardware: Mac minis, Raspberry Pis, MacBooks, and Windows machines
• Model Support: MNIST, GPT-2, and custom neural networks
• Distributed Inference: Model parallelism with streaming token generation
• Web Interface: React-based chat UI for GPT inference
• Experiment Tracking: W&B integration with automatic metrics logging

Refer to the quickstart guide for an quick and easy setup or MANUAL.md for a step-by-step manual setup guide and to run your first training/inference algorithm on smolcluster.

• Cluster Setup Guide - Complete setup for distributed training cluster
• Network Configuration Guide - Detailed networking setup (Thunderbolt + Ethernet)
• Training Guide - Training algorithms and usage
• Configuration Guide - Cluster and model configuration
• Inference Guide - Model parallelism inference
• Inference API Reference - HTTP + SSE endpoints for MP and DP
• Logging Setup - Distributed log monitoring via the dashboard

ZeRO-optimized data parallelism with configurable optimizer state partitioning. Best for memory-constrained setups and large models.

bash scripts/launch_fsdp_train_gpt.sh

Features:

• ZeRO Stage 0: All-Reduce (classic data parallelism)
• ZeRO Stage 1: Optimizer state partitioning (~1/N memory per worker)
• Bandwidth-optimized weight broadcasting (only owned parameters)
• Configurable bounded staleness (0 = strict sync, K > 0 = async up to K steps)
• Real-time staleness monitoring via WandB

All-Reduce based data parallelism with bounded staleness. Best for balanced clusters with moderate network latency.

bash scripts/launch_dp_train_gpt.sh

Features:

• All-to-all gradient averaging (ring all-reduce)
• Configurable bounded staleness (0 = strict sync, K > 0 = async up to K steps)
• Real-time staleness monitoring via WandB
• Automatic stale gradient cleanup

Asynchronous data parallelism with stale gradient tolerance. Best for heterogeneous clusters.

bash scripts/launch_edp_train_gpt.sh

Synchronous data parallelism with barrier coordination. Best for homogeneous clusters.

bash scripts/launch_syncps_train_gpt.sh

Mixture-of-Experts training with experts sharded across nodes. Best for scaling MoE models efficiently across heterogeneous hardware.

bash scripts/training/launch_ep_train_moe.sh

Layer-wise model distribution. Best for large models and inference serving.

bash scripts/inference/launch_inference.sh --algorithm mp
bash scripts/inference/launch_api.sh

See training.md for detailed algorithm comparison and usage.

Real-time experiment tracking at wandb.ai

• Training/validation metrics
• Per-layer gradient norms
• Hardware utilization

See logging.md for log monitoring setup.

smolcluster/
├── docs/                           # Documentation
│   ├── configuration.md            # Config guide
│   ├── training.md                 # Training guide
│   ├── logging.md                  # Logging setup
│   ├── inference.md                # Inference guide
│   ├── api.md                      # Inference API reference
│   └── setup_cluster.md            # Hardware setup
├── src/smolcluster/
│   ├── algorithms/
│   │   ├── Elastic Distributed Parallelism/                    # Elastic Distributed Parallelism
│   │   ├── DataParallelism/        # Data Parallelism implementations
│   │   │   ├── ClassicDP/          # Classic All-Reduce Data Parallelism
│   │   │   └── SynchronousPS/      # Synchronous Parameter Server
│   │   ├── ExpertParallelism/      # Expert Parallelism (MoE)
│   │   ├── Fully Sharded Data Parallel/                   # Fully Sharded Data Parallelism
│   │   ├── ModelParallelism/       # Model Parallelism
│   │   └── ModelParallelismPipeline/  # Pipeline Model Parallelism
│   ├── models/                     # Neural network models
│   ├── utils/                      # Utilities and helpers
│   ├── data/                       # Datasets
│   ├── configs/                    # YAML configurations
│   └── chat/                       # Web inference interface
├── scripts/                        # Launch scripts
├── logging/                        # Cluster log files
└── pyproject.toml                  # Dependencies

Evaluate a trained checkpoint with G-Eval (GPT-4o-mini as judge):

python src/smolcluster/applications/reasoning/grpo/evaluation/evaluate_summarization.py \
  --checkpoint-dir checkpoints/<model>/<run>/latest \
  --model-name mlx-community/<model>

Evaluate the base model (no checkpoint) with a custom system prompt:

python src/smolcluster/applications/reasoning/grpo/evaluation/evaluate_summarization.py \
  --base-model-only \
  --model-name mlx-community/Qwen2.5-0.5B-Instruct-bf16 \
  --system-prompt "You are an assistant who is an expert at summarization task. The user gives you a post and you are required to summarize it, keeping the key points and main ideas intact, in EXACTLY 50 words"

Replace --model-name with mlx-community/LFM-2.5-350M-bf16 to evaluate the LFM model. Results are saved under src/smolcluster/applications/reasoning/grpo/evaluation/eval-rollouts/.

Pull requests welcome! Please ensure your code follows the existing style and includes appropriate logging.

• Code (smolcluster source, scripts, configs): Apache License 2.0. Keep the copyright notice and NOTICE when you redistribute.
• Docs, diagrams and published artifacts: CC BY 4.0 - free to reuse with attribution to Yuvraj Singh (name + link; indicate changes).

If you use smolcluster in academic work, please cite it via CITATION.cff (GitHub's "Cite this repository" button). Each GitHub release is archived on Zenodo, which mints a DOI for the exact version.

@software{singh2026smolcluster,
  title     = {smolcluster: An Educational Distributed Training and Inference Library for Local Computing},
  author    = {Singh, Yuvraj},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23196836},
  url       = {https://github.com/YuvrajSingh-mist/smolcluster}
}