YeonwooSung/llm-architecture-from-scratch기술 문서

YeonwooSung/llm-architecture-from-scratch open-source repository on GitHub

목차 (TOC) (2개 섹션)▾

언어 대체 안내:해당 저장소의 한국어 문서가 제공되지 않아 공식 영문 원본(English)으로 표시됩니다.

빠른 설치

git clone https://github.com/YeonwooSung/llm-architecture-from-scratch.git

LLM architecture from scratch

Implementing LLM architectures from scratch for deep understanding of modern LLM architectures.

Model	Model Size	Modality	Normalization	Positional Encoding	Attention	Mixture of Experts
GPT-2 XL	1.5B	Text	LayerNorm	Learned Absolute	Multi-Head Self-Attention (causal)	❌
Llama3-8B	8B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
Llama 3.2-3B	3B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
OlMo 2 7B	7B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
Phi-4	14B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
DeepSeek v3	671B	Text	RMSNorm	RoPE	Multi-head Latent Attention	✅
Gemma 3	27B	Text + Image	RMSNorm	RoPE	Sliding window + GQA	❌
xLSTM	7B	Text	RMSNorm	RoPE	mLSTM	❌
Mistral 3.1 Small	24B	Text + Image	RMSNorm	RoPE	Sliding window + GQA	❌
Llama 4 Maverick	400B	Text	RMSNorm	RoPE + NoPE	GQA	✅
Qwen3	32B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌
Qwen3 (235B-A22B)	235B A22B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
SmolLM3	3B	Text	RMSNorm	RoPE + NoPE	Grouped-Query Attention (causal)	❌
Kimi K2	1T	Text	RMSNorm	RoPE	Multi-head Latent Attention	✅
GPT OSS	120B	Text	RMSNorm	YaRN RoPE	Grouped-Query Attention (causal)	✅
GLM 4.5	355B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
Grok 2.5	270B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
MiniMax M2 (230B)	230B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
Qwen3 Next	80B-A3B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	✅
Ouro Thinking	2.6B	Text	RMSNorm	RoPE	Grouped-Query Attention (causal)	❌

GitHub 공식 원본 저장소에서 아카이브된 문서 미러 · GitHub 저장소 방문

동기화 기준일: 2026-11-09