
nanochat#8 GPT-2ใฎ็นๆฐใฏ 0.256525 ใ ใฃใ
้ฃ่ผใๆฌ็ฉใฎLLMใใๅใใใชใใ่ชญใฟ่งฃใใ
็ฌฌ8ๅใGPT-2ใฎ็นๆฐใฏ 0.256525 ใ ใฃใ
ใใใ ่ชๅใฎใขใใซใ22็ง็ฎใงๆก็นใใใใ3็ง็ฎใงใพใใไปฅไธใ ใฃใ ใ
ใฉใใใงใ๏ผใใใใใงใใ
ๅๅใฎๆๅพใซใใใๆธใใพใใใ
ใGPT-2ใพใงใใใจใฉใใใใใใใ
ไปๅใใใใๆธฌใใพใใใ็ญใใๅ ใซๆธใใพใใ
GPT-2 ใฎ 14.2% ใฎใจใใใซใใพใใใ
ใใใฆๆธฌใฃใฆใใ้ไธญใงใใใใพใง5ๅใถใไฝฟใฃใฆใใๆฐๅญใใ
ๆใฃใฆใใใใๅฐใชใใใผใฟใงๅบใใใใฎใ ใจๅใใใพใใใใใกใใฎ่ฉฑใใใพใใ
nanochat ใซใฏใ็ฉๅทฎใใ2ๆฌใใ
ๅใฎ้ฃ่ผใงใฏใๆๅคฑใจใใๆฐๅญใ1ๆฌใ ใ่ฆใฆใใพใใใ
nanochat ใซใฏ2ๆฌใใใพใใ
val_bpb 1ใใคใใใใไฝใใใใง่ฟทใฃใฆใใใ
CORE 22็ง็ฎใฎใในใใฎ็นๆฐไธใฏๅฝใฆใซใใใไธใฏใงใใใใจใๆธฌใใพใใ
ไปๆฅใฏไธกๆนใใ5,000ในใใใๅใใใใจใฎ่ชๅใฎใขใใซใซๅฝใฆใพใใ
ๅฉใใใฎใฏใใฎ1ๆฌใงใใ
python -m scripts.base_eval --model-tag d6 \
--eval core,bpb --device-batch-size=16 --max-per-task=500ๆๅคฑใใใใคใใงๅฒใ
ใพใ bpb ใฎใปใใใใ
ๅใฎ้ฃ่ผใง่ฆใฆใใๆๅคฑใฏใใใผใฏใณ1ๅใใใใฉใใใใ่ฟทใฃใใใงใใใ
ใใใซใฏๅฐใฃใใจใใใใใใพใใๅใๆนใ้ใใจๆฏในใใใชใใฎใงใใ
็ฌฌ3ๅใง่ชๅใฎใใผใฏใใคใถใไฝใฃใใจใใhelloย ใฏ1ๅใฎใใผใฏใณใงใใใ
ใใๅใๆนใฎ็ฒใใใผใฏใใคใถใชใใๅใ helloย ใ h , e , l , l , o ใฎ5ๅใซใชใใพใใ
ๅใๆ็ซ ใใ็ๆนใฏ1ๅใงๅฝใฆใใใ็ๆนใฏ5ๅใซๅใใฆๅฝใฆใใ
1ๅใใใใฎๆ็ธพใๆฏในใฆใใๆๅณใใใใพใใใ
ใใใง nanochat ใฏใใใผใฏใณใงใฏใชใใใคใใงๅฒใใพใใ
bpb = ่ฟทใใฎๅ่จ รท (log2 ร ใใคใๆฐใฎๅ่จ)helloย ใฏใใฉใๅใฃใฆใ5ใใคใใงใใ
ๅใๆนใๅคใใฃใฆใใใคใๆฐใฏๅคใใใชใใฎใงใ้ใใขใใซๅๅฃซใไธฆในใใใพใใ
ๅฎ่ฃ
ใฏ nanochat/loss_eval.py ใฎ30่กใปใฉใงใใ
้ข็ฝใใฎใฏใใใผใฏใณใใจใซใใคใๆฐใฎ่กจใๆใฃใฆใใฆใใใใงๅฒใฃใฆใใใจใใใงใใใ
token_bytes[y] ใใฎใใผใฏใณใไฝใใคใใ<|bos|>๏ผๆ็ซ ใฎๅงใพใใฎๅฐ๏ผใฎใใใช็นๆฎใชใใผใฏใณใฏใใใฎ่กจใง 0ใใคใใซใใฆใใใพใใ
0ใๆใใใจๆถใใใฎใงใๆก็นใใ่ชๅ็ใซๅคใใพใใ
ใ็นๆฎใใผใฏใณใ้คๅคใใใใจใใๅฆ็ใใฉใใซใๆธใใใซใ่กจใฎๅคใ ใใงๆธใพใใฆใใใใใพใใงใใ
22็ง็ฎใฎใในใ
ใใ1ๆฌใฎ CORE ใฏใใในใใงใใไธญ่บซใ้ใใใ22็ง็ฎใใใพใใใ
้ธๆๅผ 11็ง็ฎ hellaswag, arc_easy, piqa, boolq ...
็ถใๅฝใฆ 9็ง็ฎ jeopardy, squad, coqa, lambada ...
็ฉบๆฌ่ฃๅ
2็ง็ฎ winograd, winograndeๆก็นใฎใใใใใๅฝขๅผใใจใซ้ใใพใใ
้ธๆๅผใฏใ้ธๆ่ขใๅ
จ้จใขใใซใซ่ชญใพใใฆใใใกใฐใ้ฉใใชใใฃใใใฎใ้ธใฐใใพใใ
ใใใฎ็ถใใๆฅใฆใไธๆ่ญฐใใใชใใใจไธ็ชๅผทใๆใฃใ้ธๆ่ขใใใขใใซใฎ็ญใใงใใ
็ถใๅฝใฆใฏๅณใใใฆใ็ถใใฎๅ่ชใๅ
จ้จ่จใๅฝใฆใชใใจไธๆญฃ่งฃใงใใ
1่ชใงใ้ใใฐใผใญใ้จๅ็นใฏใใใพใใใ
ใใใฆใใใใ CORE ใฎใใกใฐใ้ข็ฝใใจใใใชใฎใงใใใ
็ด ็นใใใฎใพใพ่ถณใใพใใใ
่ฃๆญฃๅพ = (ๆญฃ็ญ็ โ ใพใใ) รท (1 โ ใพใใ)4ๆใชใใใงใใใใซ้ธใใงใ25%ๅฝใใใพใใ
ใ ใใใใฎ25%ใๅผใใฆใใใๆฎใใฎ75%ใๆบ็นใจใใฆๅฒใ็ดใใ
ใพใใใงๅใใใถใใใ็นๆฐใจใใฆ่ชใใชใใจใใใใจใงใใ
ใใฎๅผใซใฏใใฏใฃใใใใๅฏไฝ็จใใใใพใใ
ใใคใในใๅบใ
็งใฎใขใใซใฎ็ตๆใงใใๆชใใปใใใไธฆในใพใใ
็ง็ฎ ๆญฃ็ญ็ ใพใใ ่ฃๆญฃๅพ
boolq 0.3800 0.62 -0.6316
copa 0.4000 0.50 -0.2000
arc_challenge 0.2440 0.25 -0.0080ใใคใในใงใใ
boolq ใฏ Yes ใ No ใง็ญใใๅ้กใงใใใใพใใใฎๅบๆบใ 0.62 ใซ่จญๅฎใใใฆใใพใใ
Yes ใจ No ใฎๆฐใๅใฃใฆใใใฎใงใๅคใใปใใๆฉๆขฐ็ใซ็ญใใใ ใใง62%ๅใใใใใงใใ
ใใใซ็งใฎใขใใซใฏ 0.38 ใใๅฑใใพใใใงใใใ
Yes/No ใฎใฉใกใใใ่จใ็ถใใใ ใใฎๆฉๆขฐใซใ่ฒ ใใฆใใพใใ
copa ใๅใใงใใ2ๆใง 0.40ใใณใคใณใๆใใใปใใใใทใงใใใ
ใพใใไปฅไธใฏใ0็นใงใฏใชใใใคใในใจใใฆ่จ้ฒใใใใ
ใใใใ็ฉๅทฎใใงใใใ
ใใใฆใใผใญใใใใพใใใ
jeopardy 0.0000
bigbench_dyck_languages 0.0000
bigbench_repeat_copy_logic 0.0000500ๅใใฃใฆใ1ๅใๅฝใใฃใฆใใพใใใ
ใฉใใ็ถใๅฝใฆใฎ็ง็ฎใงใใใๅ
จ้จ่จใๅฝใฆใชใใจไธๆญฃ่งฃใใๅนใใฆใใพใใ
ใใใงใ็นใๅ ฅใฃใ็ง็ฎ
ไธๆนใงใใกใใใจๅใใ็ง็ฎใใใใพใใ
bigbench_cs_algorithms +0.4020
arc_easy +0.2027
bigbench_language_identification +0.1925
piqa +0.1680
lambada_openai +0.1580ๆ้ซ็นใฏ bigbench_cs_algorithms ใฎ 0.4020 ใงใใใ
ใใใฏใพใใใฎๅบๆบใ0ใฎ็ง็ฎใชใฎใงใๆญฃ็ญ็ใใใฎใพใพ็นๆฐใซใชใใพใใ
piqa ใฏใ็ฉใไฝฟใใจใใฎๅธธ่ญใใ้ธใถๅ้กใงใๆญฃ็ญ็ 0.5840ใ
็ฌฌ6ๅใงใใใฉใณในใฎ้ฆ้ฝใฏใใฉใณในใฎ้ฆ้ฝใ ใฃใใใจ็ฌใฃใใขใใซใใ
ๆฅ็จๅใฎๆฑใใซใคใใฆใฏๅๅไปฅไธๅฝใฆใฆใใพใใ
ๅ่จใใฆใฟใ
22็ง็ฎใฎ่ฃๆญฃๅพใฎ็นใใๅ จ้จ่ถณใใฆ22ใงๅฒใใพใใ
็งใฎใขใใซ๏ผ7,353ไธๅใป5,000ในใใใ๏ผ 0.036373
GPT-2๏ผ16ๅๅ๏ผ 0.256525
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
ๅทฎ -0.2201520.036 ๅฏพ 0.257ใGPT-2 ใฎ 14.2% ใงใใ
ใใฎ 0.256525 ใใnanochat ๅ
จไฝใฎ็ฎๆจ็ทใงใใ
README ใซใใๆธใใฆใใใพใใ8ๅใฎGPUใง1.5ๆ้ใ$48ใ
ใใใงใใฎ็ทใ่ถ
ใใใฎใใใใฎใชใใธใใชใฎ็ฎ็ใงใใ
2019ๅนดใGPT-2 ใ่จ็ทดใใใฎใซ 168ๆ้ใป็ด$43,000 ใใใใพใใใ
7ๅนดใงใ$43,000 ใ $48 ใซใชใฃใใ900ๅใฎ1ใงใใ
็งใฏใใพใใใฎ็ทใฎ14%ใฎใจใใใซใใพใใ
Mac mini ใง2ๆ้15ๅใ0ๅใฎใจใใใใๅงใใๅฐ็นใจใใฆใฏใใใใชใใฎใใจๆใใพใใใ
ๆก็นใซใๅญฆ็ฟใฎ2ๅใใใฃใ
ไปๅใใกใฐใไบๆณๅคใ ใฃใใฎใฏใๆ้ใงใใ
ๅญฆ็ฟ๏ผ5,000ในใใใ๏ผ 2ๆ้15ๅ
ๆก็น๏ผ22็ง็ฎ๏ผ 4ๆ้57ๅๆก็นใฎใปใใ2ๅใใใใพใใใ
ใใใ็ง็ฎใใจใฎๅใใ็ฐๅธธใงใใใ
bigbench_repeat_copy_logic 1.8็ง
bigbench_language_identification 7,088็ง๏ผ1ๆ้58ๅ๏ผ3,900ๅใงใใๅใใ1็ง็ฎใใจๆฐใใฆใใใใฎใใ่ฟทใๅทฎใงใใใ
็็ฑใฏใ10-shot ใจใใ่จญๅฎใซใใใพใใ
ๅ้กใๅบใๅใซใใๆๆฌใ10ๅ่ชญใพใใใใๆนใงใใ
ใใฎใถใ1ๅใ้ทใใชใใ้ธๆ่ขใฎๆฐใ ใ็นฐใ่ฟใใฎใงใ
ใ้ทใๅ้กๆ ร ้ธๆ่ขใฎๆฐ ร 500ๅใใง่จใใฟใพใใ
็ฌฌ6ๅใง --core-metric-every=-1ย ใไปใใฆใ
ๅญฆ็ฟไธญใฏ CORE ใๆธฌใใใชใ่จญๅฎใซใใฆใใพใใใ
ใใฎใจใ็็ฑใฏๅใใฃใฆใใพใใใงใใใใๆญฃ่งฃใงใใใ
100ในใใใใใจใซใใใๆธฌใฃใฆใใใใๅญฆ็ฟใ็ตใใใพใใใ
22็ง็ฎ็ฎใงใ่ฝใกใ
ใใใฆๆๅพใฎ็ง็ฎใงใๆญขใพใใพใใใ
RuntimeError: MPS backend out of memory
(MPS allocated: 18.16 GiB, ... Tried to allocate 1.06 GiB)21็ง็ฎใ้้ใใใใจใ22็ง็ฎ็ฎใงใกใขใชใ่ถณใใชใใชใใพใใใ
็ฌฌ7ๅใงใ6GBใ่ถ
ใใฆใๆญขใพใใชใใใจๆธใใใฎใซใไปๅใฏๆญขใพใฃใฆใใพใใ
้ใใฏใ่ฒฏใพใๆนใงใใใ
่ฝใกใ็ง็ฎ่ชไฝใฏ4ๆใฎ่ปฝใใใฎใงใๅไฝใชใ้ใใพใใ
21็ง็ฎใถใใฎ็ขบไฟใ็ฉใฟไธใใฃใ็ถๆ
ใงๆๅพใซๅ
ฅใฃใใฎใงใ่ถณใใชใใชใฃใใ
1็ง็ฎใ ใๆฐใใๆธฌใ็ดใใใใใใฃใใ้ใใพใใใ
base_evalย ใฏ้ไธญ็ต้ใไฟๅญใใชใใฎใงใ
21็ง็ฎใป5ๆ้ใถใใฎ็ตๆใฏใญใฐใซใใๆฎใฃใฆใใพใใใ
ใญใฐใใๆพใ็ดใใฆใๆธฌใ็ดใใ1็ง็ฎใจๅๆตใใใฆใไธใฎๅ่จใๅบใใฆใใพใใ
้ทใใใฎใๆธฌใใจใใฏใ้ไธญใง่ฝใกใฆใๆพใใใใใซใใฆใใใ
็ฌฌ7ๅใงใใญใฐใซ PYTHONUNBUFFERED=1ย ใไปใใใใจๆธใใใฎใจใๅใ่ฉฑใงใใใ
ใคใใงใซใๅใฎๆฐๅญใๆบใใ
ใใใฒใจใคใๆธฌใฃใฆใใใใกใซๆฐใฅใใใใจใใใใพใใ
็ฌฌ6ๅใจ็ฌฌ7ๅใงใๅฝใฆใซใใใฎๆฐๅญใจใใฆ 1.165885 ใไฝฟใฃใฆใใพใใใ
ไปๅใใใๆธฌใ็ดใใใใใใใชใใพใใใ
ๅญฆ็ฟใญใฐใฎๅค 1.165885
2,097ไธใใผใฏใณใงๆธฌใ็ดใ 1.146981
4,194ไธใใผใฏใณใงๆธฌใ็ดใ 1.1464470.019 ใใใใไฝใๅบใพใใ
ๅๅ ใ่ฟฝใใพใใใbase_trainย ใๅญฆ็ฟไธญใซๆธฌใ้ใฏใๆขๅฎใง 524,288 41,943,040ย ใใผใฏใณใงใใ๏ผ9/8่จๆญฃ๏ผ
็ญใๅใใ็จใฎใใผใฟใฏใ8ไธ4,992ไปถใป็ด2ๅ4,500ไธๆๅญใใใพใใ
็ญใๅใใ็จใใผใฟๅ
จไฝ ็ด6,000ไธใใผใฏใณ
ๅญฆ็ฟไธญใซ่ฆใฆใใ้ 524,288ใใผใฏใณ = ็ด 0.8%0.8% ใงใใ
ๅญฆ็ฟใๆญขใใฆ100ๅใๆธฌใใใฎใชใฎใงใ่ปฝใใใฆใใใฎใฏๅฝ็ถใงใใใ
ใใ ใ็งใฏใใฎๆฐๅญใใใใฎใขใใซใฎๆ็ธพใใจใใฆ5ๅใถใๆธใใฆใใพใใใ
็ฌฌ7ๅใฎใMac ใจ 3060 ใฎๅทฎใฏ 0.000047ใใจใใๆฏ่ผใใฎใใฎใฏใ
ไธกๆนใจใๅใ 0.8% ใงๆธฌใฃใฆใใใฎใงๆๅนใงใใ
ใใใฉ็ตถๅฏพๅคใฎใปใใฏใใกใใใจๆธฌใใจ 1.16 ใงใฏใชใ 1.15 ใงใใใ
ๆฐๅญใๅบใฆใใใจใใคใๆธฌใๆนใ็ขบใใใใซไฝฟใฃใฆใใพใใ
ไปๅใใใใใฃใฆใใพใใใ
ใใฎๅใฎๆฐๅญ
0.256525
GPT-2 ใฎ็นๆฐใงใใ
2019ๅนดใซ168ๆ้ใจ$43,000ใใใใฆไฝใใใใขใใซใใ22็ง็ฎใฎใในใใงๅใฃใ็นใ
nanochat ใฏใใใฎ1ใคใฎๆฐๅญใ่ถ
ใใใใจใ็ฎ็ใซๆธใใใฆใใพใใ
ใ ใใ README ใฎ่กจใใbase_evalย ใฎๅฎ่ฃ
ใใๅ
จ้จใใใซๅใใฆใใพใใ
็งใฎใขใใซใฏ 0.036373 ใงใใใ14.2%ใ
้ข็ฝใใฃใใฎใฏใใใฎ14%ใใไฝใใงใใฆใใชใใใงใฏใชใใฃใใใจใงใใ
็ฉใฎๆฑใใฏๅๅๅฝใฆใ็ฐกๅใช็็งใฎๅ้กใฏ4ๅฒๅฝใฆใ
ใใฎไธๆนใง Yes/No ใฎๅ้กใงใฏใณใคใณๆใใซ่ฒ ใใฆใใใ
ใงใใใใจใจใงใใชใใใจใใ็ง็ฎใใจใซใฐใใฐใใซ็ซใกไธใใฃใฆใใใ
ๅนณๅใใใจGPT-2ใฎ14%ใงใใใไธญใ้ใใจ 0.40 ใใ -0.63 ใพใงๆฃใใฐใฃใฆใใพใใใ
่ณขใใ1ๆฌใฎ็ทใงไผธใณใฆใใใฎใงใฏใชใใฆใ
22ๆฌใฎๅฅใ
ใฎ็ทใใใฆใใงใฐใใฐใใซๅใใฆใใใ
ใใใใใใฎใใๅนณๅใใฆ1ใคใฎๆฐๅญใซใใฆ็บใใฆใใใใใงใใ
ๆฌกๅ
ใใใพใงใงใไธใใใใใ็ตใใใพใใใ
ๆ็ซ ใๅใฃใฆใ่ชญใพใใฆใใขใใซใ็ตใใงใ2ๅฐใงๅใใฆใ็นๆฐใไปใใใ
ใใใฉใใงใใใใฃใใใฎใฏใพใ ไผ่ฉฑใใงใใพใใใ
็ฌฌ6ๅใง7ใค่ณชๅใใฆใ่ฟใฃใฆใใใฎใฏใใใงใใใ
The capital of France is the capital of France่ณชๅใซ็ญใใฆใใพใใใ็ถใใๆธใใฆใใใ ใใงใใ
ๆฌกๅใฏใใใใไผ่ฉฑใงใใใใใซใใพใใ
ๅใฎ้ฃ่ผใงใฏใ6ไปถใฎไพใ0.3็งใ ใ่ฟฝๅ ๅญฆ็ฟใใใฆใใใใใใ็ญใใใใใซใใพใใใ
ใใฎใใใใ่ฆใใฆใใ198ๆๅญใๅฟใใพใใใ
nanochat ใฎใใๆนใฏใใพใฃใใ้ใใพใใใ
็นๅฅใชใใผใฏใณใ8ๅ่ฟฝๅ ใใใจใใใใๅงใพใใพใใ
ใงใฏใงใฏใ
ๆณจ๏ผใใฎ่จไบใฎๅท็ญใซใฏAIใไฝฟใฃใฆใใพใใๆงๆใจๆ็ต็ใชๆ็ซ ใฏ่ชๅใฎๆใซใใใใฎใงใใ