YeonwooSung/llm-architecture-from-scratch技术文档

YeonwooSung/llm-architecture-from-scratch open-source repository on GitHub

目录大纲 (TOC) (2 章节)▾

多语言回退提示：当前仓库原作者暂未发布本地化文档，已为您无缝切换至官方英文原版（English），排版及代码实例完整保真。

极速安装

git clone https://github.com/YeonwooSung/llm-architecture-from-scratch.git

LLM architecture from scratch

Implementing LLM architectures from scratch for deep understanding of modern LLM architectures.

Model	Model Size	Modality	Normalization	Positional Encoding	Attention	Mixture of Experts
GPT-2 XL	1.5B	Text	LayerNorm	Learned Absolute	Multi-Head Self-Attention (causal)	❌
Llama3-8B	8B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
Llama 3.2-3B	3B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
OlMo 2 7B	7B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
Phi-4	14B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
DeepSeek v3	671B	Text	RMSNorm	RoPE	Multi-head Latent Attention	✅
Gemma 3	27B	Text + Image	RMSNorm	RoPE	Sliding window + GQA	❌
xLSTM	7B	Text	RMSNorm	RoPE	mLSTM	❌
Mistral 3.1 Small	24B	Text + Image	RMSNorm	RoPE	Sliding window + GQA	❌
Llama 4 Maverick	400B	Text	RMSNorm	RoPE + NoPE	GQA	✅
Qwen3	32B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
Qwen3 (235B-A22B)	235B A22B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
SmolLM3	3B	Text	RMSNorm	RoPE + NoPE	Grouped-Query Attention (causal)	❌
Kimi K2	1T	Text	RMSNorm	RoPE	Multi-head Latent Attention	✅
GPT OSS	120B	Text	RMSNorm	YaRN RoPE	Grouped-Query Attention (causal)	✅
GLM 4.5	355B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
Grok 2.5	270B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
MiniMax M2 (230B)	230B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
Qwen3 Next	80B-A3B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
Ouro Thinking	2.6B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌

文档镜像归档自 GitHub 官方源路径 · 访问原项目 GitHub

最后同步快照：2026-11-09