
nanochat#5 Attentionã¯ãã¢ãã«ã®5%ãããªãã£ã
é£èŒãæ¬ç©ã®LLMããåãããªããèªã¿è§£ãã
第5åãAttentionã¯ãã¢ãã«ã®5%ãããªãã£ã
ããã 555è¡ãéããŠã7,353äžåãæã§æ°ãã ã
ã©ããã§ãïŒããããã§ãã
第1åã§ãã¢ãã«æ¬äœã®ãã¡ã€ã«(gpt.py)ã555è¡ã ãšæ°ããŸããã
åã®é£èŒã§èªåãæžããåã圹å²ã®ãã¡ã€ã«(model.py)ã¯ã178è¡ã§ããã
å·®ã¯377è¡ãä»åã¯ããã®äžèº«ãèŠãŠã¿ãŸãã
ãŸããéšåããšã«äžŠã¹ãŠã¿ã
äž¡æ¹ã®ãã¡ã€ã«ã«ã¯ãåãååã®ã¯ã©ã¹ïŒéšåã®ãããŸãïŒãå
¥ã£ãŠããŸãã
ååãšè¡æ°ãããã®ãŸãŸæšªã«äžŠã¹ãŸããã
åå model.py 178è¡ nanochat gpt.py 555è¡
Head 33è¡ â
ââ CausalSelfAttention 62è¡
MultiHeadAttention 14è¡ â
FeedForward 19è¡ â MLP 11è¡
Block 17è¡ â Block 10è¡
SmallGPT 72è¡ â GPT 400è¡èŠãç¬éã«ãæã£ãŠããã®ãšéããŸããã
çãäžã®2ã€ããã³ã³ãã¯ãã«ãªã£ãŠããŸãã
FeedForward ã¯19è¡ãã11è¡ãžãBlock ã¯17è¡ãã10è¡ãžã
7幎ã¶ãã®é²æ©ã§ååããªã£ãŠããã¯ãã®å Žæããéã«ç©ããŠããŸããã
ãããŠ400è¡ã
ãã¡ã°ãå€åŽã®ç®±ã ããã72è¡ãã400è¡ã«ãµãããã§ããŸãã
å·®ã®377è¡ã¯ãã»ãšãã©ããã«å ¥ã£ãŠããŸããã
ãã®400è¡ã®ãååã¯ã¢ãã«ã§ã¯ãªãã£ã
400è¡ã®äžèº«ãã圹å²ã§åããŠã¿ãŸãã
GPT ã¯ã©ã¹ 400è¡
éããšã¡ã¢ãªãèŠç©ãã颿° 7å 93è¡
ã©ã®éšåãã©ãåãããã®èšå® 39è¡
æåã®æ°åã®å
¥ãæ¹ 66è¡
ââââââââââââââââââââââââââââ
æ®ãïŒæ¬åœã®ã¢ãã«éšåïŒ 202è¡äžã®3ã€ã¯ãã¢ãã«ãã®ãã®ã§ã¯ãããŸããã
äžèº«ãèŠãŠãããšã
â 93è¡ã¯ãæ°ããããã®éå
·ã§ãã
1ããŒã¯ã³ãããäœåã®æãç®ãèŠããã1ããŒã¯ã³ã¶ãã®èšæ¶ã«äœãã€ãèŠããã
ãã©ã¡ãŒã¿ã¯äœåãããããè¿ãã ãã®é¢æ°ã7åã䞊ãã§å
¥ã£ãŠããŸãã
ååã®ç§ã®ã³ãŒãã«ã¯ããããã颿°ã¯1åããããŸããã§ããã
21äžåã®ã¢ãã«ãªããæ°ããªããŠãå°ããªãããã§ãã
7,000äžåã14ååãšå€§ãããªããšãåããåã«æž¬ã£ãŠãããªããšã
ããããã¡ã¢ãªã«èŒããã©ãããåãããŸããã
倧ãããªããšããããšã¯ã枬ãéå
·ãèŠãããã«ãªããšããããšã
â 39è¡ã¯ãéšåããšã«åããæ¹ãå€ããèšå®ã§ããããã¯çªå€ç·šã§æ±ããŸãã
â 66è¡ã¯ãåŠç¿ãå§ããåã«æ°åãã©ãå
¥ããŠãããã®æé ã§ãã
ååã¯8è¡ã§ããã8è¡ã66è¡ã«ãªã£ãã¶ããçœ®ãæ¹ã现ãããªã£ãŠããŸãã
åŒãç®ãããšããããªããŸãã
nanochat gpt.py 555è¡ â 198è¡ = 357è¡
åå model.py 178è¡ â 8è¡ = 170è¡
âââââââââââââââââââââââââââââââââ
ã¢ãã«ãã®ãã®ã®å·® 187è¡377è¡ã ãšæã£ãŠããå·®ã¯ãéããŠã¿ãã187è¡ã§ããã
ååã®å®¿é¡ã®ãçãåãã
åã®é£èŒã®æçµåã§ããä»åã¯è§Šããªãã£ããã®ãã5ã€äžŠã¹ãŸããã
æ¬ç©ã®ã¢ãã«ã«ã¯å
¥ã£ãŠããããèªåã®ã³ãŒãã«ã¯å
¥ããªãã£ãéšåã§ãã
nanochat ãã©ãããŠããããã1ã€ãã€èŠãŸãã
ååã®ãªã¹ã nanochat ã§ã¯
RoPE æ¡çšãäœçœ®ã®è¡šã¯ã1ã€ãç¡ã
RMSNorm æ¡çšããã ãèŠããæ°åã¯ãŒãå
SwiGLU äžæ¡çšãReLUã®çµæã2ä¹ããã ã
FlashAttention æ¡çšã䜿ããªãæ©æ¢°ã§ã¯èªåã§å¥ã®ããæ¹ã«åãæ¿ãã
MoE äžæ¡çšããã¡ã€ã«å
šäœãæ€çŽ¢ããŠ0ä»¶5ã€ã®ãã¡3ã€ãæ¡çšã2ã€ã¯äžæ¡çšã§ããã
ãšãã«é¢çœãã£ãã®ã¯ RMSNorm ã§ãã
ååã®ç§ã®ã³ãŒãã¯ãå±€ã®ãã³ã«æ°åã®å€§ãããããããåŠçãå
¥ããŠããŸããã
ããã«ã¯ãã©ã®ããããããããããèŠããæ°åãã1,152åãããŸããã
nanochat ã®åãå Žæã«ã¯ãèŠããæ°åã1åããããŸããã
ãã å²ãç®ããŠããããã ãã§ãåŠç¿ã§èª¿æŽããäœå°ãæ®ããŠããªãã
åãããŠããã®ã¯ãããã ãã§ã¯ãããŸããã§ããã
ååã«ãã£ãŠãnanochat ã«ç¡ããã®
äœçœ®ãèŠãã衚 2,048å â 0å
ãããæ¹ãèŠããæ°å 1,152å â 0å
äžé§ïŒãã€ã¢ã¹ïŒ 1,635å â 0å
ããããã¢ãŠã 3ãæ â 0ãæåèš4,835åãååã®21äžåã®ã2.3%ã«ããããŸãã
7幎ã¶ãã®å·®ã¯ãè¶³ãç®ã ãã§ã¯ãããŸããã§ããã
ãŸãããã®4çš®é¡ãæ¶ããŠããŸããã
æã§æ°ããŠãçªãåããã
ãããããæ¬é¡ã§ãã
åã®é£èŒã®æçµåã§ã76ååã®ã¢ãã«ã®äžèº«ãçŽã®äžã§æ°ããŠã
衚瀺ãããæ°ãš1åãéããªãããšã確ãããŸããã
åãããšããnanochat ã§ãããŸãã
æå ã®Macã§åãèšå®ã¯ãããã§ãã
å±€ã®æ° 6å±€
1ããŒã¯ã³ãããã®æ°å 384å
èªåœ 32,768åååã®é£èŒã§ãã£ãã®ãšåãã§ããã®3ã€ã ãã§å šéšæ±ºãŸããŸããé ã«æ°ããŸãã
èªåœã®è¡š
32,768åã®èšèã«ã384åãã€ã®æ°åãæããã衚ã§ãã
32,768 à 384 = 12,582,912ãã®è¡šãããã®ã¢ãã«ã«ã¯5åå ¥ã£ãŠããŸãããªã5åãªã®ãã¯ãããšã§èŠãŸãã
AttentionïŒ1å±€ã¶ãïŒ
384æ¬ã®å
¥åã384æ¬ã«ããåè§ã衚ãã1å±€ã«4ã€ã
384 à 384 = 147,456 ããã4〠= 589,824MLPïŒ1å±€ã¶ãïŒ
384æ¬ããã£ãã4åã®1,536æ¬ã«åºããŠããŸã384æ¬ã«æ»ããŸãã
384 à 1,536 = 589,824 è¡ããšåž°ãã§2〠= 1,179,648ããšã¯è¶³ãã ãã§ãã
èªåœã®è¡š 5å 12,582,912 à 5 = 62,914,560
Attention 6å±€å 589,824 à 6 = 3,538,944
MLP 6å±€å 1,179,648 à 6 = 7,077,888
现ããæ°ïŒäžèšïŒ 254
ââââââââââââââââââââââââââââââââ
åèš 73,531,646æåŸã®254åã¯ãããšã§åºãŠããŸããããŸã¯è¶³ããŠããã ãã«ããŸãã
ã§ã¯ãã¢ãã«èªèº«ã«èããŠã¿ãŸããåŠç¿ã1ã¹ãããã ãåããŸããã
python -m scripts.base_train --depth=6 --head-dim=64 \
--window-pattern=L --max-seq-len=512 \
--device-batch-size=32 --total-batch-size=16384 \
--num-iterations=1åºãŠãããã°ã§ãã
Parameter counts:
wte : 12,582,912
value_embeds : 37,748,736
lm_head : 12,582,912
transformer_matrices : 10,617,048
scalars : 38
total : 73,531,646 æã§æ°ããåèš 73,531,646
ã¢ãã«ã®çã 73,531,646
ââââââââââââââââââââââââ
å·® 00åã
ååãšåãã§ãããäžèº«ã¯ãå šéšæ°ããããŸãã
ã€ãã§ã«ããã°ã®æåŸã®è¡ã«ãããåºãŠããŸããã
step 00000/00001 | loss: 10.397551åã®é£èŒã§ã¯ãåŠç¿ãå§ããåã®æå€±ã 4.60 ã§ããã
èªåœ99åããå®å
šãªåœãŠãã£ãœãã§1åéžã¶ãšãçå±ã®äžã§ã¯4.5951ã«ãªããŸãã
ãŸã äœãåŠãã§ããªãã¢ãã«ã¯ããã®å€ããå§ãŸãã
ä»åã®èªåœã¯32,768åãåãèšç®ããããš 10.397 ã§ãã
åºãŠããã®ã¯ 10.397551 ã§ããã
èªåœã331åã«ãªã£ãŠããåãçå±ããã®ãŸãŸéã£ãŠããŸããã
å èš³ãèŠãŠãæãæ¢ãŸã£ã
æ°ãåã£ãã®ã¯ããã®ã§ãããå èš³ã®ã»ãã§æãæ¢ãŸããŸããã
ãã£ãã®ãã°ãã倧ããé ã«äžŠã¹æ¿ããŸãã
value_embeds 37,748,736 51.3%
wteïŒå
¥å£ã®è¡šïŒ 12,582,912 17.1%
lm_headïŒåºå£ã®è¡šïŒ 12,582,912 17.1%
transformer_matrices 10,617,048 14.4%
scalars 38 0.0%äžã®3ã€ã¯ãå
šéšèªåœã®è¡šã§ãã32,768è¡ Ã 384åã®ãåã倧ããã®è¡šã
value_embeds ã¯1ã€ã§ã¯ãªãã1å±€ããã«3ã€å
¥ã£ãŠããŸãã
ã€ãŸãããã®ã¢ãã«ã¯åã倧ããã®è¡šã5åæã£ãŠããŸãã
12,582,912 Ã 5 = 62,914,560 å
šäœã® 85.6%æ®ãããAttention ãš MLP ã§ããåãããšãããªããŸãã
èªåœã®è¡š 5å 62,914,560 85.6%
MLP 7,077,888 9.6%
Attention 3,538,944 4.8%
ãã®ä»ã®çްããæ° 254 0.0%4.8%ã
ãAttention ããããã¹ãŠããšããæåãªè«æã®é¡åãããã®ä»çµã¿ã¯å§ãŸããŸããã
ãã®Attentionããã¢ãã«ã®æã¡ç©ã®20åã®1ãããããŸããã
ååã®èªåã®ã¢ãã«ãšæ¯ã¹ããšããã£ãšã¯ã£ããããŸãã
åå 21äžå ä»å 7,353äžå
èšç®ããéšå 93.1% 14.4%
èªåœãªã©ã®è¡š 6.9% 85.6%ãŸãããšè£è¿ã£ãŠããŸããã
çç±ã®ååã¯èªåœã§ããååã¯99æåãä»åã¯32,768åã331åã«ãªããŸããã
衚ã1ã€ããã ãã§1,258äžåã§ãã
ããååã¯ããã®è¡šã5åæã£ãŠããããšã§ãã
衚ãå¢ãããšãéããªã
念ã®ãããæ¬çªãµã€ãºã§ãåãããšãèµ·ããã®ãã確ãããŸããã
H100ãåããŠåãèšå®ã¯ã24å±€ã»å€ªã1,536ã§ããåãæ°ãæ¹ãããŸãã
6å±€ïŒæå
ïŒ 24å±€ïŒæ¬çªïŒ
ç·ãã©ã¡ãŒã¿æ° 73,531,646 1,384,122,122
èªåœã®è¡šã®æ° 5å 14å
èªåœã®è¡šã®å²å 85.6% 50.9%
MLP 9.6% 32.7%
Attention 4.8% 16.4%å±€ãå¢ãããšã衚ã®å²åã¯äžãããŸããããã§ãã¡ããã©ååã衚ã§ããã
ãããŠè¡šã®æ°ã¯ã5åãã14åã«å¢ããŠããŸãã
1å±€ããã«1ã€å¢ããäœããªã®ã§ãå±€ãç©ããšè¡šãç©ã¿äžãããŸãã
第7åã§ã6ã®ã¬ã®ã°ã©ãã£ãã¯ããŒãã«äœãèŒããã詊ããŸãã
ãã®ãšãå¹ããŠããã®ã¯ããã¶ãå±€ã®æ°ã§ã¯ãªããŠããã®è¡šã®æ°ã§ãã
7幎ã§è¶³ããããã®ã¯ãã»ãšãã©1çš®é¡ã ã£ã
æåŸã«ããã²ãšã€ã
ååã®ã³ãŒãã«ç¡ããŠãnanochat ã«ããéšåãå
šéšéããŠã
ãã®ãã©ã¡ãŒã¿ã ããæ°ããŠã¿ãŸããã
value_embeds 37,748,736å
ãã®ä»ããã¶ïŒ6çš®é¡ïŒ 254åãã®ä»6çš®é¡ãšããã®ã¯ããããããã®ã§ãã
å±€ããšã«æ®å·®ã®åŒ·ããæ±ºããæ°åã6åãå
¥å£ã®æ
å ±ãã©ãã ãæ··ãæ»ããã6åã
1ã€åã®èšèãå°ãæ··ãã仿ãã25åãéäžã®å±€ã®çµæãåŒãç®ããæ°åã1åã
ãã㊠value_embeds ã®å¹ãå
·åã調æŽããæ°åã216åã
å šéšåãããŠ254åã§ãã
7幎ã§è¶³ãããéšåã¯ãæ°ã®äžã§ã¯ã»ãŒ1çš®é¡ã§ããã
ååã ãæããŠãããšãGQAãQKãã«ã ãã¹ã©ã€ãã£ã³ã°ãŠã£ã³ããŠã
value embeddingsãsmearãbackoutãå±€ããšã®ä¿æ°ã
7ã€ã®ãã¡ã倧ããªå Žæãåã£ãŠããã®ã¯ value embeddings ã ãã
æ®ãã¯ãæ°ååããæ°çŸåã®å°ããªæ°åã§ãã
ã¡ãªã¿ã«æå
ã®èšå®ã§ã¯ããã®ãã¡2ã€ã¯å®éã«ã¯åããŸããã
GQAã¯ä»åãªãïŒèšå®ã§åãæ°ã«ããããŠããïŒã
ã¹ã©ã€ãã£ã³ã°ãŠã£ã³ããŠããMacã§ã¯äœ¿ããªãã®ã§åã£ãŠãããŸãã
WARNING: SDPA has no support for sliding window attentionèµ·åæã«ãããèŠåãåºãŸãã
ãã®åã®æ°å
4.8%
7,353äžåã®ãã¡ãAttention ãæã£ãŠããã®ã¯3,538,944åã ãã§ããã
åã®é£èŒãæžããŠãããšããç§ã¯ã¢ãã«ã®äžèº«ã
ãèšç®ãã仿ãã®éãŸããã ãšæã£ãŠããŸããã
å®éããã®ãšãã®ã¢ãã«ã¯93%ãèšç®ããéšåã§ããã
æ¬ç©ãéããããéã§ããã
äžèº«ã®å€§éšåã¯ãèšèãæ°ã«çœ®ãæããããã®ããã ã®å¯Ÿå¿è¡šã§ãã
ãããåãããšãã¢ãã«ã®å€§ããã®æå³ãå°ãå€ãã£ãŠèŠããŸãã
è³¢ã仿ããè©°ãŸã£ãŠãããšããããã
ååãèŸæžãäœåãæ±ããŠããããšããã»ããè¿ãã
第13åã§ããäºå®ã® Q4_K_M ã¯ããã®èŸæžãèããã話ã§ãã
ã©ããåãã®ããã¡ã°ãå¹ãã®ããã仿¥ã®è¡šã§å
ã«èŠããŠããŸããŸããã
次å
äžèº«ãèŠãã®ã¯ããããŸã§ã§ãã
次åãããå®éã«åããŸãã
Mac mini ã§6å±€ã®ã¢ãã«ã5,000ã¹ãããããããã®ç°å¢ã§çŽ30åãããèŠèŸŒã¿ã§ãã
åã®é£èŒã§ã¯ã198æåãèŠããããã®ã«æ°åç§ã§ããã
ä»åºŠã¯2åæåãçžæã«ã30åã§ãã
100ã¹ãããããšã«ãéäžã®æç« ãåºãèšå®ã«ããŠãããŸãã
ååãè±ãæœããããšããå£ããæ¥æ¬èªãåºãŠãããšããã«çžåœããŸãã
äœã¹ãããç®ã§ãè±èªãããåèªãåºãŠãããã
ã§ã¯ã§ã¯ã
泚ïŒãã®èšäºã®å·çã«ã¯AIã䜿ã£ãŠããŸããæ§æãšæçµçãªæç« ã¯èªåã®æã«ãããã®ã§ãã