AutoParallel is a PyTorch library that automatically shards and parallelizes models for distributed training. Given a model and a device mesh, it uses linear programming to find an optimal sharding strategy (FSDP, tensor parallelism, or a mix) and applies it — no manual parallelism code required.

Early Development Warning — AutoParallel is experimental. Expect bugs, incomplete features, and APIs that may change. Bugfixes are welcome; please discuss significant changes in the issue tracker before starting work.

New to AutoParallel? Start with the Getting Started guide.

• Python >= 3.10
• PyTorch nightly (newer than 2.10)

# Via SSH
pip install git+ssh://[email protected]/meta-pytorch/autoparallel.git

# Via HTTPS
pip install git+https://github.com/meta-pytorch/autoparallel.git

The simplest way to try AutoParallel is with a HuggingFace model. This runs entirely on a single machine using a fake process group — no multi-GPU setup needed:

pip install transformers
python examples/example_hf.py --model gpt2 --mesh 8

You should see log output ending with Forward + backward OK.

For more examples (LLaMA-3, distributed checkpointing, local_map / MoE), see the examples/ directory.

Start here if you're new to AutoParallel:

• Getting Started
• Basic Concepts
• API Walkthrough

Troubleshooting and common questions:

• Troubleshooting
• FAQ

Deeper explanations and advanced topics:

• How AutoParallel Chooses a Strategy
• Adaptive Sharding: Sequence-Parallel vs Column-Parallel
• Using local_map for MoE and Custom Communication Patterns
• Saving and Loading Optimizer State
• Documentation Index

# Run from the repository root
pip install -e .

Modified Python files are reflected immediately.

Run linters before submitting a PR:

pip install pre-commit
pre-commit run --all-files

Run tests (requires a CUDA GPU):

pip install -r requirements-test.txt
pytest tests/

AutoParallel is BSD-3 licensed, as found in the LICENSE file.