
nanochat#6 100ในใใใใง่ฑ่ชใฏๆธใใใๆฎใใฎ4,900ในใใใใฏใไฝใใใฆใใใฎใ
้ฃ่ผใๆฌ็ฉใฎLLMใใๅใใใชใใ่ชญใฟ่งฃใใ
็ฌฌ6ๅใ100ในใใใใง่ฑ่ชใฏๆธใใใๆฎใใฎ4,900ในใใใใฏใไฝใใใฆใใใฎใ
ใใใ Mac miniใง2ๆ้15ๅใใใฉใณในใฎ้ฆ้ฝใฏใใฉใณในใฎ้ฆ้ฝใ ใฃใ ใ
ใฉใใใงใ๏ผใใใใใงใใ
ใฏใฉในใใกใคใซใ่ชญใใฎใฏๅๅใพใงใซใใฆใไปๅใใๅฎ้ใซๅใใพใใ
ๅๅใฎๆๅพใซใไฝในใใใ็ฎใงใ่ฑ่ชใใใๅ่ชใๅบใฆใใใใใจๆธใใพใใใ
็ญใใๅ
ใซ่จใใจใ100ในใใใ็ฎใซใฏใใใๅฎ็งใช่ฑ่ชใๆธใใฆใใพใใใ
ใใ ใใไธญ่บซใฏไฝใใใใพใใใงใใใ
ๅฉใใใฎใฏใใใฎ1ๆฌ
runs/runcpu.shย ใซๆธใใฆใใใใฎใใใใฎใพใพๅฎ่กใใพใใ
python -m scripts.base_train \
--depth=6 --head-dim=64 --window-pattern=L \
--max-seq-len=512 --device-batch-size=32 \
--total-batch-size=16384 \
--eval-every=100 --sample-every=100 \
--num-iterations=5000ๅๅๆฐใใ7,353ไธๅใฎใขใใซใใ5,000ๅใพใใใพใใ
--sample-every=100ย ใไปใใฆใใใฎใงใ100ๅใใจใซๅญฆ็ฟใๆญขใใฆใ
ใใฎใจใใฎใขใใซใซ7ใคใฎ่ณชๅใใใฆใ็ญใใๅบๅใใใพใใ
ใใฎ้ไธญ็ต้ใใไปๅใฎไธปๅฝนใงใใ
่ตทๅ็ดๅพใฎใญใฐใฏใใใใงใใใ
Autodetected device type: mps
COMPUTE_DTYPE: torch.float32 (auto-detected: no CUDA (CPU/MPS))
WARNING: Flash Attention 3 not available, using PyTorch SDPA fallbackMacใชใฎใงใ็ฒพๅบฆใฏ float32ใๅๅๅๅใ ใๅบใใ FlashAttention 3 ใฏไฝฟใใชใใฎใงใ
PyTorchๆจๆบใฎใใๆนใซ่ชๅใงๅใๆฟใใใพใใใใใฏ่ญฆๅใๅบใใ ใใงใๆญขใพใใพใใใ
2ๆ้14ๅ58็งใใใฃใ
ๅ ใซใๆ้ใฎ่ฉฑใใใพใใ
้ๅง 08:29:58
็ตไบ 10:44:56
โโโโโโโโโโโโโโโโโ
ๅ่จ 2ๆ้14ๅ58็งruncpu.shย ใซใฏใnanochatไฝ่
ใฎ็ฐๅข๏ผMacBook Pro M3 Max๏ผใง
็ด30ๅใง็ตใใใใใซ่ชฟๆดใใใจๆธใใฆใใใพใใ
ใใกใฎMac mini๏ผM4็กๅฐใปใกใขใช16GB๏ผใงใฏใ4.5ๅใใใใพใใใ
ๅ ่จณใฏใใใงใใ
ๅญฆ็ฟใใฎใใฎ 119.26ๅ
100ๅใใจใฎ่ฉไพกใจ็ๆ 15.71ๅ
โโโโโโโโโโโโโโโโโโโโโโโโโ
ๅ่จ 134.97ๅ1ในใใใใใใใฎ้ใใฏใ5,000ๅใถใๆกใใจใใใชใใพใใใ
1ในใใใ ไธญๅคฎๅค 1,377 ms ๏ผๆ้ 1,244 / ๆ้
16,088๏ผ
tok/sec ไธญๅคฎๅค 11,901ๆ้
ใฎ16็งใฏใ่ฉไพกใจ็ๆใๆใพใฃใ็ดๅพใฎๅใงใใใใใ้คใใฐใ
ใปใผ1.3ใ1.4็งใงไธๅฎใงใใใ
ๅใฎ้ฃ่ผใจไธฆในใพใใ
ๅๅ๏ผ21ไธๅ๏ผ ไปๅ๏ผ7,353ไธๅ๏ผ
ในใใใๆฐ 3,000ๅ 5,000ๅ
1ในใใใ 16.2 ms 1,377 ms
ๅ่จๆ้ 48.6็ง 2ๆ้15ๅ1ในใใใใซใใใๆ้ใใ85ๅใซใชใใพใใใ
ๅๅใฏ3,000ๅใพใใใฆ49็งใใณใผใใผใๆทนใใฆใใใใใ ใซ็ตใใฃใฆใใพใใใ
ไปๅใฏใๆทนใใฆใ้ฃฒใฟ็ตใใฆใๆดใ็ฉใพใงๆธใพใใฆใใใพใ 10ๅใฎ1ใ้ฒใฟใพใใใ
ๅญฆ็ฟใๅงใใๅใซใไฝใ็ญใใใฎใ
100ในใใใ็ฎใฎ่ฉฑใใใๅใซใใใฎๆๅใ่ฆใฆใใใพใใ
--sample-every=100ย ใฏ100ๅใใจใชใฎใงใ0ๅ็ฎใฎ็ญใใฏๅบๅใใใพใใใ
ใชใฎใงใๅๆๅใใใ ใใฎใขใใซใๅฅใซไฝใฃใฆใๅใ7ใคใ่ใใฆใฟใพใใใ
The capital of France is diver Practices exceptional Length prevailing
ernesswing East Switzerland Style malt ...
The opposite of hot is diver Practices exceptional Length prevailing
ernesswing East Switzerland Style malt ...
My favorite color is diver Practices exceptional Length prevailing
ernesswing East Switzerland Style malt ...ๅ่ชใฎใตใฉใใงใใใใใพใงใฏไบๆณใฉใใใงใใใ
ไบๆณใใฆใใชใใฃใใฎใฏใ7ใคใฎ่ณชๅใฎใใก5ใคใซใใพใฃใใๅใ็ญใใ่ฟใใใใจใงใใ
้ฆ้ฝใ่ใใฆใใ่ฒใ่ใใฆใใ่ฟใฃใฆใใใฎใฏๅใไธฆใณใ
่ณชๅใฎไธญ่บซใใใพใ 1ๆๅญใ่ฆใฆใใพใใใ
ใใใๅบ็บ็นใงใใใ
100ในใใใ็ฎ
ใใใใ2ๅๅ๏ผ100ในใใใ๏ผ่ตฐใใใ็ตๆใงใใ
The capital of France is a very popular and the most common in the most
common in the most common in
The opposite of hot is the most common of the most common of the most
common of the most common of
My favorite color is a good way to the use of the same way to the
use of the same่ฑ่ชใงใใ
ๅ่ชใฎใตใฉใใงใฏใใใพใใใๅ ่ฉใใใฃใฆใๅ็ฝฎ่ฉใใใฃใฆใ่ช้ ใๆญฃใใใ
the most common in theย ใฏใ่ฑ่ชใจใใฆ1ใคใ้้ใฃใฆใใพใใใ
ใใ ใใๆๅณใฏ็ฉบใงใใๅใ่จใๅใใๅปถใ ใจ็นฐใ่ฟใใฆใใใ ใใงใใ
ใใใฆใ่ณชๅใใจใซ็ญใใๅคใใใใใซใชใใพใใใ
a very popularย ใจ a good way toย ใฏ้ใๅบใ ใใงใใ
ๅ
ฅๅใ่ชญใใใใซใฏใชใฃใใ่ชญใใ ็ตๆใ่จใใใจใใชใใฃใใ
ๅใฎ้ฃ่ผใงใ100ในใใใ็ฎใซใ่ฑใๆฝใใใใจใใๅฃใใๆฅๆฌ่ชใๅบใพใใใ
ใใใฏ่ฆใใ้ไธญใฎๆทท็ทใงใใใไปๅใฏๆทท็ทใใใใฆใใพใใใ
ๆๅใซใงใใใใใซใชใฃใใฎใฏใๆญฃใใใงใฏใชใใใใใใใใงใใใ
้ไธญ็ต้ใใ1ๆฌใฎ่ณชๅใง่ฟฝใ
ใใกใฐใ่ฆๅฟใใใใฃใใฎใฏใ็ฑใใฎๅๅฏพใฏ๏ผใใงใใใ
50ๅใถใใฎไธญใใๆใๅบใใฆไธฆในใพใใ
step 0 diver Practices exceptional Length prevailing ...
step 100 the most common of the most common of the most common of
step 400 the most common cause of the cold weather.
step 700 a hot and cold. The cold is cold and cold. The cold is cold
step 1600 true. The opposite of hot is true. The opposite of hot is true.
step 2600 the heat of the sun. The heat of the sun is the heat of the
step 3600 that the temperature of the water is too hot.
step 4600 that the heat of the sun is absorbed by the sun.400ๅ็ฎใง cold๏ผๅทใใ๏ผใๅบใพใใใใใๅใใฆใฎใๆญฃใใๆนๅใใงใใใ
ใจใใใใcold ใๅบใฆใใใฎใฏ 400ใป600ใป700ใป800 ใฎ4ๅใ ใใงใใใ
ใใฎใใจ4,200ในใใใๅใใพใใใใไบๅบฆใจๅบใฆใใพใใใงใใใ
ไปฃใใใซ1600ๅ็ฎใซใฏใ
ใ็ฑใใฎๅๅฏพใฏใๆญฃใใใ็ฑใใฎๅๅฏพใฏใๆญฃใใใใ ใจ่จใๅบใใพใใ
ใพใฃใใ่ฏใใชใใใใงใฏใชใใใจใใใฎใใใๅใใ1ๆฌใงใใใ
ใใกใฐใใใใใซไผธใณใใฎใฏใๅฅฝใใช่ฒใฏ๏ผใใงใใ
step 100 a good way to the use of the same way to the use of the same
step 1100 the white, white, and white. The white is a white, white,
step 2100 the red, orange, and orange. I have a few of my favorite color
step 3100 the black. I have a black, brown, black, black, black,
step 4600 the red. I have a red, yellow, and blue color. I have1100ๅ็ฎใงๅใใฆ่ฒใฎๅๅ๏ผwhite๏ผใ1ใคๅบใฆใ2100ๅ็ฎใง3ใคใซๅขใใพใใ
ๆๅพใซใฏใ่ตคใป้ปใป้ใไธฆในใฆใ็ถใๆใพใงๆธใใใใใซใชใใพใใใ
้ใซใๆๅพใพใงไธๅบฆใ่ฏใใชใใชใใฃใใฎใใใใงใใ
step 100 If 5*x + 3 = 13, then x is 2020, 2020, 2020, 2020,
step 1600 If 5*x + 3 = 13, then x is 3*x + 3*x + 3*x +
step 4600 If 5*x + 3 = 13, then x is the number of times the number of
times the number of times็ญใใฏ2ใงใใ5,000ๅใพใใใฆใใๆฐๅญใฎ2ใฏๅบใพใใใงใใใ
5,000ในใใใๅพใฎ็ญใ
ๆๅพใฎ็ๆใงใใ
The capital of France is the capital of France, which is the capital of
France. The capital of Franceใใใฉใณในใฎ้ฆ้ฝใฏใใใฉใณในใฎ้ฆ้ฝใงใใใใใใฏใใฉใณในใฎ้ฆ้ฝใ ใใ
2ๆ้15ๅใใใฆใใใใงใใใ
ๆใจใใฆใฏใใฉใใๅฃใใฆใใพใใใไธป่ชใใใใ่ฟฐ่ชใใใใ้ขไฟไปฃๅ่ฉใพใง
ๆญฃใใไฝฟใใฆใใพใใใใ ใใใชใจใใๅ่ชใ1ๅใๅบใฆใใชใใ
ๅใฎ้ฃ่ผใฎใ่ฑใๆฝใใใใฏใๆฅๆฌ่ชใจใใฆๅฃใใฆใใฆใใใใๆใใใใฃใใ
ไปๅใฎใฏๅฃใใฆใใพใใใๅฎ็งใชๆใงใไฝใ่จใฃใฆใใพใใใ
ใ็ฑใใฎๅๅฏพใใฎใปใใฏใๆๅพใซใใใชใใพใใใ
The opposite of hot is the hot air. The hot air is the most important
part of the body'sใ็ฑใใฎๅๅฏพใฏใ็ฑใ็ฉบๆฐใงใใใ
ใๅฝใฆใซใใใใฎใปใใฏใใกใใใจไธใใฃใฆใใ
็ๆใฎไธญ่บซใฏๆ็ซ ใจใใฆๆๅณใๆใใฆใใชใใฎใงใใใ็นๆฐใฎใปใใฏ็ด ็ดใซ่ฏใใชใฃใฆใใพใใ
100ๅใใจใซๅญฆ็ฟใๆญขใใฆใไธๅบฆใๅญฆ็ฟใซไฝฟใฃใฆใใชใๆ็ซ ใ่ชญใพใใฆใใพใใ
ใใใฆใๆฌกใซๆฅใๆๅญใๅฝใฆใใใพใใ
ๅฝใใใชใใปใฉๆฐๅญใฏๅคงใใใใใๅฝใใใปใฉๅฐใใใชใใพใใ
ใในใใฎ็นใจ้ใงใๅฐใใใปใฉ่ฏใๆฐๅญใงใใ
ใญใฐใซใฏใใๅบใพใใ
Step 00100 | Validation bpb: 1.940628bpbย ใฏใ1ใใคใ๏ผ่ฑ่ชใชใใปใผ1ๆๅญ๏ผใถใใๅฝใฆใใฎใซใ
ไฝใใใ่ฟทใฃใใใใจใใๆธฌใๆนใงใใใใฎๅไฝใใฎใใฎใฏ็ฌฌ8ๅใงๆฑใใพใใ
ใใพใฏๅฝใฆใซใใใฎ็นๆฐใจใ ใๆใฃใฆใใ ใใใ
100ๅปใฟใฎ้ไธญ่จๆธฌ็นใใใๆใๅบใใฆไธฆในใพใใ
ในใใใ ๅฝใฆใซใใ๏ผๅฐใใใปใฉ่ฏใ๏ผ
step 0 3.195800
step 100 1.940628
step 500 1.530011
step 1000 1.371891
step 2000 1.279500
step 3000 1.224675
step 4000 1.187887
step 5000 1.1658853.196 ใใ 1.166 ใพใงใ100ๅปใฟใฎ้ไธญ่จๆธฌ็นใฏใไธๅบฆใไธใใใใซไธใใ็ถใใพใใใ
ใใ ใไธใใๆนใซๅใใใใใพใใ
ๅ
จไฝใฎๆนๅๅน
ใ100ใจใใฆใใฉใใพใง้ฒใใ ใใไธฆในใพใใ
step 100 ๆนๅใฎ 61.8%ใๅฎไบ ๏ผๅ
จ่ก็จใฎ 2%๏ผ
step 500 ๆนๅใฎ 82.1%ใๅฎไบ ๏ผๅ
จ่ก็จใฎ 10%๏ผ
step 1000 ๆนๅใฎ 89.9%ใๅฎไบ ๏ผๅ
จ่ก็จใฎ 20%๏ผ
step 2000 ๆนๅใฎ 94.4%ใๅฎไบ ๏ผๅ
จ่ก็จใฎ 40%๏ผ
step 5000 ๆนๅใฎ 100% ๏ผๅ
จ่ก็จใฎ100%๏ผๆๅใฎ2%ใงใ6ๅฒใ็ตใใฃใฆใใพใใ
2ๆ้15ๅใฎใใกใๆๅใฎ2ๅๅใง6ๅฒใๆฎใใฎ2ๆ้ใง4ๅฒใงใใ
็ๆใไธฆในใใจใใฎๆใใจใใดใฃใใๅใใพใใ
100ในใใใ็ฎใฎใ่ฑ่ชใซใชใฃใใใใใกใฐใๅคงใใช้ฃ่บใงใ
ใใฎใใจใฏ็ดฐใใ่ฉฐใใใใฃใจใใฃใฆใใใ
ใชใใใชใๅบใฆใใชใใฎใ
็ญใใฏ็ฌฌ4ๅใงใใใๅบใฆใใพใใใ
ไปๅ่ชญใพใใ้ใฏใใใใงใใ
5,000ในใใใ ร 16,384ใใผใฏใณ = 81,920,000ใใผใฏใณ็ฌฌ4ๅใงใใใผใฟใฎใใกใคใซ1ๅใ็ด5,226ไธใใผใฏใณใ ใจๆฐใใพใใใ
ใคใพใไปๅ่ชญใพใใใฎใฏใใใกใคใซ1.57ๅใถใใงใใ
ใใใฆๅใ็ฌฌ4ๅใงใGPT-2ใจๅใใใใใซใใใซใฏ170ๅ่ฆใใใจๆธใใพใใใ
ไปๅ่ชญใพใใ้ ใใกใคใซ 1.57ๅใถใ
GPT-2็ดใซๅฟ
่ฆ ใใกใคใซ 170ๅใถใ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
้ๆ็ 0.92%1%ใ่ชญใใงใใพใใใ
่ฑ่ชใฎๆธใๆนใฏใ1%ใงใ่บซใซใคใใพใใใ
ใใฉใณในใฎ้ฆ้ฝใใใชใ ใจใใ1ใคใฎไบๅฎใฏใ่บซใซใคใใชใใฃใใ
่จใๆนใๅคใใใจใๆธใๆนใฏ่ใๅบใๆธใใฆใใใใไบๅฎใฏ1ใๆใซใใๆธใใฆใใชใ
ใจใใใใจใ ใจๆใใพใใๅ็ฝฎ่ฉใฎไฝฟใๆนใฏ2ๅๆๅญใฎใฉใใซใงใๅบใฆใใพใใใ
ใParis is the capital of Franceใใจใใๆใฏใใใไฝๅบฆใๅบใฆใใพใใใ
ใใพใ๏ผMacใฏใใใฃใใๆฎดใใใฆใใ
ๅใใฆใใ้ใๆๅ
ใฎใกใขใชใฏ95%๏ผ15GB / 16GB๏ผใงๅผตใไปใใฆใใพใใใ
ๆฐใซใชใฃใใฎใงๆธฌใฃใฆใใใพใใใ
ใกใขใชๅง็ธฎ ๆฏ็ง 448 MB
ๅง็ธฎใฎๅฑ้ ๆฏ็ง 443 MB
ใใผใธใคใณ ๆฏ็ง 146 MBๆฏ็ง450MBๅใใ็ณใใงใฏๅบใ็ดใใฆใใพใใ
ๆๅคใ ใฃใใฎใฏใใใฎ็ฝฎใๅ ดใ้ฃใฃใฆใใใฎใใขใใซใงใฏใชใใฃใใใจใงใใ
ใใฉใกใผใฟๆฌไฝใจๆ้ฉๅใฎ็ถๆ
็ด 0.8 GB
1ๅใถใใฎ็นๆฐ่กจ ใกใใใฉ 2.0 GB็นๆฐ่กจใจใใใฎใฏใใใใ32ๆฌ ร 512ใใผใฏใณใฎใใใใใซใคใใฆใ
่ชๅฝ32,768ๅใถใใฎ็นๆฐใไธฆในใใใฎใงใใ
32 ร 512 ร 32,768 ร 4ใใคใ = 2.00 GBใใใ่จ็ฎใฎ้ไธญใงไฝๅใ่ฆใใพใใ7,353ไธๅใฎใขใใซๆฌไฝใใใ
ใใฎๅ ดใใใใฎ่จ็ฎ็จ็ดใฎใปใใใใฏใใใซๅคงใใใ
--device-batch-sizeย ใๅๅใซใใใจใใใใ็ด ็ดใซๅๅใซใชใใพใใ
ๆฌกๅใ6GBใฎใฐใฉใใฃใใฏใใผใใง่ฉฐใพใฃใใใใพใใใใๅใใใจใซใชใใพใใ
ใกใชใฟใซๆธฉๅบฆใฏไธใใใพใใใใ้ ใใฏใชใใพใใใงใใใ
step 0- 499 1ในใใใ 1,426 ms
step 2500-2999 1ในใใใ 1,333 ms2ๆ้ๅใใฆใๅพๅใฎใปใใ้ใใใใใงใใ
pmset -g thermย ใซใใ็ฑใฎ่ญฆๅใฏ1ไปถใ่จ้ฒใใใฆใใพใใใงใใใ
ใงใใใใฃใใใฎ
ไฟๅญใใใใใกใคใซใงใใ
model_005000.pt 281 MB
optim_005000.pt 521 MBๅใฎ้ฃ่ผใงไฝใฃใใขใใซใฏ939KB๏ผ1ใกใฌใใคใๅผฑ๏ผใงใใใ
306ๅใซใชใใพใใใ
ใใฎๅใฎๆฐๅญ
100
100ในใใใใๅ
จ่ก็จใฎ2%ใๆ้ใซใใฆ2ๅๅใ
ใใใงใๆนๅใฎ6ๅฒใ็ตใใฃใฆใใพใใใ
็งใฏใใฃใจใๅญฆ็ฟใจใใใฎใฏใใ ใใ ใ่ณขใใชใฃใฆใใใใใฎใ ใจๆใฃใฆใใพใใใ
ๅฎ้ใซ100ๅใใจใฎ็ญใใ50ๅใถใไธฆในใฆใฟใใจใใใใงใฏใใใพใใใงใใใ
ใใกใฐใๆๅใซใใใใใใๅใใใจใ่ฆใใพใใ
ใใใๆฉใใ2ๅๅใง็ตใใใพใใ
ๆฎใใฎ2ๆ้ใฏใไธญ่บซใ1ใคใใคๆพใ้ใใๆ้ใงใใใ
่ฒใฎๅๅใ1ใค่ฆใใๅคช้ฝใฎ็ฑใฎ่ฉฑใ่ฆใใใใใงใใใชใซใฏๅฑใใชใใ
ChatGPTใใจใใฉใใ็ซๆดพใชๆ็ซ ใงๅ ใ
ใจ้้ใฃใใใจใ่จใใฎใ่ฆใใใพใใ
ใใใฏๅฃใใฆใใใฎใงใฏใชใใฆใใใฎ้ ็ชใฎๅๆฎใชใใใใชใใใจๆใใพใใใ
ๅฝขใๅ
ใซใงใใฆใไธญ่บซใๅพใใๅใพใใๅใพใใใๅใงใใๅฝขใฎใปใใฏๅฎๆใใฆใใพใใ
ๆฌกๅ
ๅใใณใใณใใใ2ๅฐ็ฎใงๅฉใใพใใ
ๆๅ
ใซใใใฎใฏใRTX 3060 ใฎ6GBใงใใใใผใฟใปใณใฟใผใฎGPUใฏ80GBใ
13ๅใฎ1ใใใใใพใใใ
ๆฉๆขฐใๆฟใใใ ใใงใๅๆใซๅคใใใใจใ1ใคใใใพใใ
Macใงใฏ float32 ใ ใฃใ่จ็ฎใฎ็ฒพๅบฆใใbfloat16 ใซๅใๆฟใใใพใใ
ใณใผใใฏ1ๆๅญใๅคใใฆใใชใใฎใซใๆฐใฎๆใกๆนใๅคใใใ
ใใใฆใไปๆฅใใพใใงๆธฌใฃใ2GBใฎ่จ็ฎ็จ็ดใใไปๅบฆใฏ้ใๅ ดใฎใชใๅ ดๆใซ็ฝฎใใใพใใ
Macใฏๅง็ธฎใใฆใงใ็ฒใฃใฆใใใพใใใใใฐใฉใใฃใใฏใใผใใฏ็ฒใใพใใใ
6GBใซใไฝใ่ผใใฎใใ
ใงใฏใงใฏใ
ๆณจ๏ผใใฎ่จไบใฎๅท็ญใซใฏAIใไฝฟใฃใฆใใพใใๆงๆใจๆ็ต็ใชๆ็ซ ใฏ่ชๅใฎๆใซใใใใฎใงใใ