YeonwooSung/llm-architecture-from-scratchDocumentação

YeonwooSung/llm-architecture-from-scratch open-source repository on GitHub

Índice (2 seções)▾

Aviso de idioma:A documentação traduzida não está disponível. Exibindo o original oficial em inglês.

Instalação rápida

git clone https://github.com/YeonwooSung/llm-architecture-from-scratch.git

LLM architecture from scratch

Implementing LLM architectures from scratch for deep understanding of modern LLM architectures.

Model	Model Size	Modality	Normalization	Positional Encoding	Attention	Mixture of Experts
GPT-2 XL	1.5B	Text	LayerNorm	Learned Absolute	Multi-Head Self-Attention (causal)	❌
Llama3-8B	8B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
Llama 3.2-3B	3B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
OlMo 2 7B	7B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
Phi-4	14B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
DeepSeek v3	671B	Text	RMSNorm	RoPE	Multi-head Latent Attention	✅
Gemma 3	27B	Text + Image	RMSNorm	RoPE	Sliding window + GQA	❌
xLSTM	7B	Text	RMSNorm	RoPE	mLSTM	❌
Mistral 3.1 Small	24B	Text + Image	RMSNorm	RoPE	Sliding window + GQA	❌
Llama 4 Maverick	400B	Text	RMSNorm	RoPE + NoPE	GQA	✅
Qwen3	32B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
Qwen3 (235B-A22B)	235B A22B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
SmolLM3	3B	Text	RMSNorm	RoPE + NoPE	Grouped-Query Attention (causal)	❌
Kimi K2	1T	Text	RMSNorm	RoPE	Multi-head Latent Attention	✅
GPT OSS	120B	Text	RMSNorm	YaRN RoPE	Grouped-Query Attention (causal)	✅
GLM 4.5	355B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
Grok 2.5	270B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
MiniMax M2 (230B)	230B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
Qwen3 Next	80B-A3B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
Ouro Thinking	2.6B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌

Espelho arquivado do repositório oficial no GitHub · Ver no GitHub

Data do snapshot: 2026-11-09