
AIã®å¿çé床ã47åã«ïŒNVIDIAãéçºããæ¬¡äžä»£AIæè¡ã®æ£äœ
ãAIã®å¿çãåŸ ã€æéã¯ãå©çšè ã«ãšã£ãŠåå®ãªåé¡ã§ããç¹ã«é·ãæç« ãæ±ãéãåŠçæéã¯ææ°é¢æ°çã«å¢å€§ããå®çšæ§ãæãªãèŠå ãšãªã£ãŠããŸããåŸæ¥ãAIã¢ãã«ã®é«éåã¯ç²ŸåºŠäœäžãšåŒãæãã§ããã
ãèšç®å¹çãäžããã°æ§èœãèœã¡ãæ§èœã远æ±ããã°åŠçãéããªãããã®äºåŸèåã¯ãå€§èŠæš¡èšèªã¢ãã«éçºã«ãããæ ¹æ¬çãªå¶çŽãšèããããŠããŸããã
ãNVIDIAãçºè¡šããJet-Nemotronã¯ããã®åžžèã«ææŠããŸããåã¢ãã«ã¯æå 端ã®èšèªçè§£èœåãç¶æããªãããçæé床ãæå€§47åãŸã§åäžãããããšã«æåããŸããã
ãéµãšãªãã®ã¯ãPostNASããšåŒã°ããæ°ææ³ã§ããæ¢åã®é«æ§èœã¢ãã«ã®ç¥èãä¿æãããŸãŸãæ§é ãå¹ççã«åèšèšãããšããçºæ³ã®è»¢æã«ããããããŸã§äžå¯èœãšãããäž¡ç«ãå®çŸããŠããŸãã
Spotifyã§ããããããé³å£°é ä¿¡ïŒããã¿ã®AIããã¯ã©ãžãªã
ãäžççµæžãã©ãŒã©ã ã®èª¿æ»ã§ã¯ã2030幎ã«å¿
èŠãªã¹ãã«ãã®ç¬¬5äœã«ã奜å¥å¿ãšç涯åŠç¿ããã©ã³ã¯ã€ã³ã奜å¥å¿ã¯è¶£å³ã§ã¯ãªãçµæžçå¿
é ã¹ãã«ã
æèãAIæä»£ã®æåŒ·ã¹ãã«ã奜å¥å¿åããã§è©³èª¬ããŠããŸãã
ãŸããã
ãLLMã«ã¯æ§é çãªéçããããŸãããæç« ãé·ããªãã»ã©ãåŠçæéãäºä¹ã«æ¯äŸããŠå¢å€§ãããšããèšç®äžã®å¶çŽã§ãã1äžæåã®ææžãªãæ°ç§ã§æžãåŠçãã10äžæåã«ãªããšæ°åãããããã®åé¡ã¯ãèªå·±æ³šææ©æ§ãšããAIã®äžæ žæè¡ã«èµ·å ããŠããŸãã
ãç ç©¶è ãã¡ã¯æ§ã ãªé«éåææ³ã詊ã¿ãŸãããããããã粟床ã®ç ç²ã䌎ããŸãããæ°åŠçãªåé¡è§£æ±ºãè€éãªæšè«ã§ã¯ãå¹çåã¢ãã«ã¯åŸæ¥åã«åã°ãªãã£ãã®ã§ãã
ããã®ç¶æ³ãæç Žããã®ããNVIDIAã®Jet-Nemotronã§ããå瀟ã¯ãåŠç¿æžã¿ã¢ãã«ãåŸããæé©åããããšããé転ã®çºæ³ã§ã粟床ãä¿ã£ããŸãŸåçãªé«éåãå®çŸããŸããã
èæ¯
解決ãã課é¡
ã仿¥ã®LLMã®é©ç°çãªæ§èœã¯ã2017幎ã«çºè¡šããããTransformerããšããã¢ãŒããã¯ãã£ã«ãã®æºæµãæã¡ãŸã ãTransformerã®äžæ žããªãã®ããèªå·±ã¢ãã³ã·ã§ã³ããšåŒã°ããã¡ã«ããºã ã§ãã
ãããã¯ãæç« äžã®ããåèªããä»ã®ãã¹ãŠã®åèªãšã©ã®ãããªé¢ä¿æ§ãæã€ããåæã«èšç®ããããšã§ãæèã®æ·±ãçè§£ãå¯èœã«ããç»æçãªä»çµã¿ã§ã ã
ãäŸãã°ãããã®éè¡ã®åæã§éé£ããããããšããæã«ãããŠããéè¡ããéèæ©é¢ã§ã¯ãªãå·ã®å²žèŸºãæå³ããããšãæ£ããçè§£ã§ããã®ã¯ãèªå·±ã¢ãã³ã·ã§ã³ããåæãããéé£ãããšãã£ãä»ã®åèªãšã®é¢é£æ§ãæããããã§ãã Â
ããããããã®åŒ·åãªã¡ã«ããºã ã«ã¯ãæ·±å»ãªèšç®äžã®æ¬ ç¹ãå åšããŠããŸããèªå·±ã¢ãã³ã·ã§ã³ã¯ãå ¥åãããæ å ±ã®é·ããé·ããªãã»ã©ãèšç®éããã®äºä¹ã§ççºçã«å¢å ããã®ã§ã ãããã¯ãäºä¹ã®èšç®éåé¡ããšããŠç¥ãããŠããŸãã
ãå ¥åæã®é·ãã2åã«ãªããšèšç®éã¯4åã10åã«ãªããš100åã«èšãäžããããšãæå³ããŸãããã®ç¹æ§ã«ãããé·ãææžã®èŠçŽãé«è§£å床ã®ç»åè§£æããããã¯é·æéã®å¯Ÿè©±å±¥æŽã䌎ããã£ããããããªã©ã倧éã®æ å ±ãäžåºŠã«æ±ãã¿ã¹ã¯ã®å®è¡ããèšç®ã³ã¹ãã®èгç¹ããæ¥µããŠå°é£ããããã¯éçŸå®çã«ãªã£ãŠããŸãã Â
ãããã«ãLLMãæç« ãçæããéã«ã¯ãããäžã€ã®å€§ããªããã«ããã¯ãååšããŸããããããKey-Valueãã£ãã·ã¥ïŒKVãã£ãã·ã¥ïŒãã§ã ãã¢ãã«ã次ã®äžèªãçæããéããããŸã§ã«å ¥åã»çæããããã¹ãŠã®åèªã®æèæ å ±ãåç §ããå¿ èŠããããŸãã
ããã®æèæ å ±ãäžæçã«ä¿åããŠããã¡ã¢ãªé åãKVãã£ãã·ã¥ã§ã ãæç« ãé·ããªãã°ãªãã»ã©ããã®KVãã£ãã·ã¥ã®ãµã€ãºãç·åœ¢ã«å¢å€§ãç¶ããŸãã
ãç¹ã«é·ãæèãæ±ãã·ããªãªã§ã¯ãã¢ãã«èªäœã®ãã©ã¡ãŒã¿ãæ ŒçŽããã¡ã¢ãªãããããã®KVãã£ãã·ã¥ãæ¶è²»ããã¡ã¢ãªã®æ¹ã倧ãããªãããšãããããŸã ã Â
ããã®å·šå€§ãªKVãã£ãã·ã¥ã¯ãæšè«ã®é床ãèããäœäžãããäž»èŠå ãšãªããŸãããªããªããæ¬¡ã®äžèªãçæãããã³ã«ããã®å·šå€§ãªãã£ãã·ã¥å šäœãGPUã®ã¡ã¢ãªäžã§èªã¿æžãããå¿ èŠãçããåŠçãã¡ã¢ãªã®è»¢éé床ã«ãã£ãŠåŸéããããã¡ã¢ãªåž¯åå¹ åŸéããšããç¶æ ã«é¥ãããã§ã ã
ãã€ãŸããLLMã®å¹çåãèããäžã§ãäºä¹ã®èšç®éåé¡ãšãããã«ä»éããKVãã£ãã·ã¥ã®å¢å€§ãšããäºã€ã®èª²é¡ã¯ãé¿ããŠéãããšã®ã§ããªãæ ¹æºçãªå¶çŽãªã®ã§ãã Â
æ¢åç ç©¶ã®æµã
ããã®ãäºä¹ã®èšç®éããšããå·šæªã«ç«ã¡åãããããäžçäžã®ç ç©¶è ãã¡ãæ§ã ãªã¢ãããŒãã詊ã¿ãŠããŸããããããã®æ¢æ±ã¯ã倧ããäžã€ã®æœ®æµã«åé¡ããããšãã§ããŸã ã Â
ã第äžã®æœ®æµã¯ããç·åœ¢ã¢ãã³ã·ã§ã³ãããç¶æ 空éã¢ãã«ããšãã£ããTransformerã®èªå·±ã¢ãã³ã·ã§ã³ãæ ¹æ¬çã«ä»£æ¿ããæ°ããã¢ãŒããã¯ãã£ã®éçºã§ãã代衚çãªç ç©¶ãšããŠãMamba ãRetNet ãRWKV ãªã©ãæããããŸãããããã®ã¢ãã«ã¯ããªã«ã¬ã³ããã¥ãŒã©ã«ãããã¯ãŒã¯ïŒRNNïŒã®ã¢ã€ãã¢ãçŸä»£çã«åè§£éããç¶æ ãå¹ççã«æŽæ°ããŠããããšã§ç·åœ¢èšç®éãå®çŸããŸãã
ããã®çµæãç¹ã«é·ãã·ãŒã±ã³ã¹ãæ±ãéã®æšè«é床ãã¡ã¢ãªå¹çã§ç®èŠãŸããææãäžããŠããŸããããããããã®äžæ¹ã§ãå€ãã®ã¿ã¹ã¯ãç¹ã«è€éãªæšè«ãèŠãããã³ãããŒã¯ã«ãããŠãäŸç¶ãšããŠæé«æ§èœã®Transformerã¢ãã«ã®ç²ŸåºŠã«åã°ãªããšãã課é¡ãæ±ããŠããŸãã ãå¹çæ§ã¯é«ããã®ã®ãæ§èœé¢ã§åŠ¥åãå¿ èŠãªã±ãŒã¹ãå€ãã£ãã®ã§ãã Â
ã第äºã®æœ®æµã¯ãããã€ããªããã¢ãã«ãã®æ§ç¯ã§ããããã¯ãå šãŠãæ°ããã¢ãŒããã¯ãã£ã«çœ®ãæããã®ã§ã¯ãªããèšç®ã³ã¹ãã®é«ããã«ã¢ãã³ã·ã§ã³ãšãå¹ççãªç·åœ¢ã¢ãã³ã·ã§ã³ãå·§ã¿ã«çµã¿åãããã¢ãããŒãã§ã ã
ãã¢ãã«ã®äžéšã®å±€ã«ã¯è¡šçŸåã®é«ããã«ã¢ãã³ã·ã§ã³ãä¿æããæ®ãã®å€ãã®å±€ãç·åœ¢ã¢ãã³ã·ã§ã³ã«çœ®ãæããããšã§ã粟床ãšå¹çã®ãã©ã³ã¹ãåãããšè©Šã¿ãŸãã
ããã®ã¢ãããŒãã¯åççã§ãããã®ã®ãã©ã®å±€ããã«ã¢ãã³ã·ã§ã³ã«ãã¹ãããšããæé©ãªé 眮åé¡ãæªè§£æ±ºã§ããããããŸã§ã®å€ãã®ãã€ããªããã¢ãã«ã¯ãäŸç¶ãšããŠæå 端ã®ãã«ã¢ãã³ã·ã§ã³ã¢ãã«ã«å¯ŸããŠæ§èœé¢ã§äžæ©è²ãçµæãšãªã£ãŠããŸããã Â
ã第äžã®æœ®æµããããã¥ãŒã©ã«ã¢ãŒããã¯ãã£æ¢çŽ¢ïŒNASïŒãã§ããããã¯ã人éãæäœæ¥ã§ã¢ãã«æ§é ãèšèšãã代ããã«ãèšç®æ©èªèº«ã«æé©ãªã¢ãŒããã¯ãã£ãèªåçã«æ¢çŽ¢ãããæè¡ã§ã ã
ãç¹ã«ããã®åéã«ãããéèŠãªå è¡ç ç©¶ãšããŠãOnce-for-All (OFA)ããæããããŸã ãOFAã®é©æ°çãªç¹ã¯ãæ¢çŽ¢ã®ãã³ã«ã¢ãã«ããŒãããåŠç¿ãããã®ã§ã¯ãªããæ§ã ãªãµããããã¯ãŒã¯ãå å ãã巚倧ãªãã¹ãŒããŒãããã¯ãŒã¯ããäžåºŠã ãåŠç¿ãããããšã§ãæ¢çŽ¢ããã»ã¹ãšåŠç¿ããã»ã¹ãåé¢ããç¹ã«ãããŸãã
ãããã«ãããç¹å®ã®ããŒããŠã§ã¢ãã¬ã€ãã³ã·ã®å¶çŽã«åãããæé©ãªã¢ãã«ãã远å åŠç¿ãªãã§è¿ éã«èŠã€ãåºãããšãå¯èœã«ãªããŸãããããããOFAã¯äž»ã«ç»åèªèã®åéã§çºå±ããLLMã®ãããªå·šå€§ã¢ãã«ã®ã¢ãŒããã¯ãã£ãã®ãã®ãæ ¹æ¬ããæ¢çŽ¢ããã«ã¯ãäŸç¶ãšããŠè«å€§ãªèšç®ã³ã¹ããéå£ãšãªã£ãŠããŸããã Â
ããããã®å è¡ç ç©¶ã¯ããããããéèŠãªé²æ©ãéããªãããããæå 端ã®ç²ŸåºŠããšãå§åçãªå¹çæ§ããšããäºã€ã®ç®æšãã劥åãªãäž¡ç«ãããæ±ºå®çãªè§£æ±ºçã«ã¯è³ã£ãŠããŸããã§ããã
ãã®ç ç©¶ã解決ãã課é¡ã»ã©ã解決ããã®ã
ããã®ãããªèæ¯ã®äžãæ¬çš¿ã§è§£èª¬ãããJet-Nemotronããšããã®éçºææ³ã§ãããPostNASããç»å ŽããŸãããã®ç ç©¶ã解決ããããšããŠããã®ã¯ãåã«ãå¹ççãªã¢ãã«ãäœãããšãã§ã¯ãããŸãããããã¯ãããæ ¹æºçãªãAIã¢ãŒããã¯ãã£ã«ãããã€ãããŒã·ã§ã³ã®ã³ã¹ãåé¡ãã§ãã
ãæå 端ã®LLMããŒãããäºååŠç¿ããã«ã¯ãæ°çŸäžãã«ããæ°åäžãã«èŠæš¡ã®èšç®è³æºãšãèšå€§ãªç¬èªããŒã¿ãå¿ èŠãšãããŸã ããã®åå ¥éå£ã®é«ãããæ°ããã¢ãŒããã¯ãã£ã®è©Šè¡é¯èª€ãå°é£ã«ããã€ãããŒã·ã§ã³ã®åæ»ãæãäžå ãšãªã£ãŠããŸãã Â
ãPostNASã¯ããã®æ§é çãªèª²é¡ã«å¯ŸããŠå®çšçãªè§£æ±ºçãæç€ºããŸãããã®æ žå¿çãªã¢ã€ãã¢ã¯ãããŒãããäœããªãããšããçºæ³ã®è»¢æã«ãããŸããPostNASã¯ãæ¢ã«å·šé¡ã®ã³ã¹ããæããŠåŠç¿ãããã髿§èœãªæ¢åã®ãã«ã¢ãã³ã·ã§ã³ã¢ãã«ããåºçºç¹ããšããŠå©çšããŸã ã
ããããŠãã¢ãã«ã®ç¥èã®å€§éšåãèç©ãããŠãããšãããMLPå±€ã®éã¿ãåçµãããŸãŸãèšç®éã®ããã«ããã¯ãšãªã£ãŠããã¢ãã³ã·ã§ã³å±€ã®ã¿ãå¹ççã«åèšèšããã®ã§ã ã Â
ããã®ã¢ãããŒãã«ãããåŠç¿ã³ã¹ããšããŒã¿èŠä»¶ã¯åçã«åæžãããŸãããã¯ããæ°ããã¢ãŒããã¯ãã£ã®æ§èœãæ€èšŒããããã«ãå€§èŠæš¡ãªäºååŠç¿ãæ¯åè¡ãå¿ èŠã¯ãããŸããã
ãæ¢åã®é«æ§èœã¢ãã«ã®èœåãç¶æ¿ãã€ã€ããã®å¹çæ§ãæå€§éã«é«ããããã®ãå€ç§æè¡çãªæ¹è¯ãã«éäžã§ããã®ã§ãã
ãããã«ãããJet-Nemotronã¯ãå è¡ç ç©¶ãçŽé¢ããŠãããå¹çåã®ããã®ç²ŸåºŠäœäžããšãããã¬ãŒããªããå æããæå 端ã®ãã«ã¢ãã³ã·ã§ã³ã¢ãã«ã«å¹æµãã粟床ãšãç·åœ¢ã¢ãã³ã·ã§ã³ã¢ãã«ã«å¹æµããå¹çæ§ããåæã«éæããããšã«æåããŸããã
æ¹æ³è«
ææ¡ææ³
ãJet-Nemotronãæ¯ããæ žå¿æè¡ããPost Neural Architecture Search (PostNAS) ãšåŒã°ããæ°ããã¢ãã«èšèšãã€ãã©ã€ã³ã§ã ããã®ææ³ã®å²åŠã¯ãæ¢åã®è³ç£ãæå€§éã«æŽ»çšããããšã«ãããŸãã
ãå šãæ°ãããšã³ãžã³ãäžããèšèšã»è£œé ããã®ã§ã¯ãªããæ¢ã«é«ãæ§èœã蚌æãããŠããæ¢åã®é«æ§èœãšã³ãžã³ïŒäºååŠç¿æžã¿LLMïŒãããŒã¹ã«ããã®æãéèŠãªéšåã§ããã¢ãã³ã·ã§ã³æ©æ§ãç¥çã«ããã€äœç³»çã«æ¹è¯ã»æè£ ãããšããã¢ãããŒããåããŸãããã®ãæ¹è¯ãã®ããã»ã¹ã¯ãç²ã段éãã现ããæ®µéãžãšé²ãã4ã€ã®ã¹ãããã§æ§æãããŠããŸã ã Â
ææ¡ææ³ã®çŽæçãªèª¬æ
ããã®4段éã®ããã»ã¹ããããçŽæçã«çè§£ããããã«ãäžçæé«å³°ã®æ§èœãæã€ã¯ã©ã·ãã¯ãªæ©æ¢°åŒè æèšããçŸä»£ã®æè¡ã§ããã«é«æ§èœåãããã¹ã¿ãŒãŠã©ããã¡ãŒã«ãŒã®ä»äºã«å©ããŠã¿ãŸãã
ãã«ã¢ãã³ã·ã§ã³å±€ã®é 眮
ããŸããæèšè·äººã¯ãªãªãžãã«ã®ã ãŒãã¡ã³ããåè§£ãããã®äžã§æãè€éãã€ç²Ÿå¯ãªèšæïŒäŸãã°ã倩æè¡šç€ºæ©èœïŒã«äžå¯æ¬ ãªãæ°åã®ç¬åµçãªæ¯è»ïŒãã«ã¢ãã³ã·ã§ã³å±€ïŒãç¹å®ããŸãããã¹ãŠã®æ¯è»ãåãéèŠæ§ãæã€ããã§ã¯ãªãããšãèŠæããæ§èœã®æ ¹å¹¹ããªãéšåã ããä¿æããã®ã§ãã
ç·åœ¢ã¢ãã³ã·ã§ã³ãããã¯ã®éžæ
ãæ¬¡ã«ããã®ä»ã®æšæºçãªæ¯è»ããçŸä»£ã®æè¡ã§äœãããããã軜éã§æ©æŠã®å°ãªãå¹ççãªä»£æ¿åïŒæ§ã ãªç·åœ¢ã¢ãã³ã·ã§ã³ãããã¯ïŒã«çœ®ãæããããšãæ€èšããŸããåœŒã¯æ¢åã®æé«çŽã®ä»£æ¿æ¯è»ãããã€ãåãå¯ããæ§èœãèä¹ æ§ãã³ã¹ãã®èгç¹ããæãåªãããã®ãéžæããŸãã
æ°ããã¢ãã³ã·ã§ã³ãããã¯ã®èšèš
ãåœŒã¯æ¢åã®ä»£æ¿åã«æºè¶³ãããããã«äžæ©é²ãã æ¹è¯ãç®æããŸããå ¥åãããåïŒã³ã³ããã¹ãïŒã«å¿ããŠåœ¢ç¶ã埮調æŽã§ãããå šãæ°ãããåçæ¯è»ïŒJetBlockïŒããèªãèšèšã»éçºããŸããããã«ãããæ¢åã®ã©ã®ä»£æ¿åãããåªããæ§èœãçºæ®ãããç¬èªã®éšåãçã¿åºãã®ã§ãã
ããŒããŠã§ã¢èªèåã¢ãŒããã¯ãã£æ¢çŽ¢
ãæåŸã«ã圌ã¯ãããã®æ°ããæ¯è»ãç¹å®ã®è æèšã®ã±ãŒã¹ïŒã¿ãŒã²ãããšãªãããŒããŠã§ã¢ïŒã«çµã¿èŸŒãã«ãããã忝è»ã®çŽåŸãåã¿ãšãã£ã埮现ãªãã©ã¡ãŒã¿ïŒãã€ããŒãã©ã¡ãŒã¿ïŒã調æŽããŸãããã®ç®çã¯ãåã«éšåã®æ°ãå¢ããããšã§ã¯ãªããã±ãŒã¹å ã§æãã¹ã ãŒãºã«ããã€æå€§ã®ãšãã«ã®ãŒå¹çã§åäœããçµã¿åãããèŠã€ãåºããæèšå šäœã®ããã©ãŒãã³ã¹ãæå€§åããããšã§ãã
ããã®ããã«ãPostNASã¯ãæ¢åã®åäœãå°éãã€ã€ãäœç³»çãã€ããŒã¿é§åçãªã¢ãããŒãã§ãã®æ§èœãšå¹çãæ¥µéãŸã§é«ãããæŽç·Žããããšã³ãžãã¢ãªã³ã°ããã»ã¹ãªã®ã§ãã
ææ¡ææ³è©³çŽ°
ã¹ããã1: ãã«ã¢ãã³ã·ã§ã³å±€ã®é 眮ãšåé€
ãPostNASã®æåã®ã¹ãããã¯ãã¢ãã«ã®æ§èœç¶æã«äžå¯æ¬ ãªãã«ã¢ãã³ã·ã§ã³å±€ãç¹å®ãããã以å€ãå¹ççãªç·åœ¢ã¢ãã³ã·ã§ã³ã«çœ®ãæããããã®æé©ãªé 眮ãèŠã€ãåºãããšã§ã ã Â
ããã®ããã«ãç ç©¶ããŒã ã¯ãOnce-for-All (OFA)ãã®ææ³ãå¿çšãããonce-for-allã¹ãŒããŒãããã¯ãŒã¯ããæ§ç¯ããŸã ãããã¯ãåã¢ãã³ã·ã§ã³å±€ã«ãããŠãå ã®ãã«ã¢ãã³ã·ã§ã³ã®çµè·¯ãšãæ°ãã远å ãããç·åœ¢ã¢ãã³ã·ã§ã³ã®çµè·¯ã®äºã€ãéžæè¢ãšããŠæã€ã巚倧ãªãããã¯ãŒã¯ã§ãã
ãåŠç¿äžããããã®çµè·¯ã¯ã©ã³ãã ã«éžæããããµããããã¯ãŒã¯ã圢æãããŸãããããŠãå ã®é«æ§èœã¢ãã«ã®åºåãæåž«ããŒã¿ãšãããç¹åŸŽèžçããšããææ³ãçšããŠãã©ã®çµè·¯ãéžæããŠãæ§èœãå£åããªãããã«ã¹ãŒããŒãããã¯ãŒã¯å šäœãåŠç¿ãããŸã ã Â

ãåŠç¿å®äºåŸããããŒã ãµãŒãããšããæ¢çŽ¢ã¢ã«ãŽãªãºã ãçšããŠãããã«ã¢ãã³ã·ã§ã³å±€ã2ã€ã ã䜿çšããããšãã£ãå¶çŽã®äžã§ãæ§èœãæå€§åãããå±€ã®çµã¿åãããèªåçã«çºèŠããŸãã
ããã®æ¢çŽ¢ãããããã€ãã®éèŠãªç¥èŠãåŸãããŸãã ã第äžã«ã Â
ããã¹ãŠã®ã¢ãã³ã·ã§ã³å±€ãçããéèŠãªã®ã§ã¯ãªããšããããšã§ããäŸãã°ãMMLUïŒå€å²ã«ãããç¥èãåããã³ãããŒã¯ïŒã§ã¯ç¹å®ã®2å±€ããæ€çŽ¢ã¿ã¹ã¯ã§ã¯ãŸãå¥ã®2ã3å±€ãæ¥µããŠéèŠãªåœ¹å²ãæãããŠããããšãæããã«ãªããŸããïŒKey Finding 1ïŒã
ã第äºã«ãã¿ã¹ã¯ã«ãã£ãŠéèŠãªå±€ãç°ãªããšããããšã§ãïŒKey Finding 2ïŒãããã¯ãLLMã®å éšã§æ©èœã®ååãèµ·ããŠããããšã瀺åããŠãããç»äžçãªã¢ãŒããã¯ãã£ããããã¿ã¹ã¯ã«å¿ããŠç¹åããç°ç𮿷·åïŒããããžãã¢ã¹ïŒãªæ§é ãæãŸããå¯èœæ§ã瀺ããŠããŸãã

ãå³5(b)ã瀺ãããã«ããã®æ¢çŽ¢ææ³ã«ãã£ãŠçºèŠãããé 眮ã¯ãåçŽã«å±€ãçééã«é 眮ãããåäžé çœ®ãæŠç¥ã«æ¯ã¹ãŠãMMLUã®ç²ŸåºŠãå€§å¹ ã«åäžãããŸãããããã¯ãPostNASãã¢ãã«ã®èœåãç¶æããäžã§ãå€ç§æè¡ã®ãããªç²Ÿå¯ãã§éèŠãªéšåãç¹å®ã§ããããšã蚌æããŠããŸãã
ã¹ããã2: ç·åœ¢ã¢ãã³ã·ã§ã³ãããã¯ã®éžæ
ããã«ã¢ãã³ã·ã§ã³å±€ã®æé©ãªé çœ®ãæ±ºå®ããåŸãæ®ãã®å±€ã«é©çšããæãåªããç·åœ¢ã¢ãã³ã·ã§ã³ãããã¯ãéžå®ããŸã ãç ç©¶ããŒã ã¯ãRWKV7ãRetNetãMamba2ãGLAãGated DeltaNetãªã©ãåœæã®æå 端ïŒSOTAïŒãšããã6ã€ã®ç·åœ¢ã¢ãã³ã·ã§ã³ãããã¯ãäœç³»çã«è©äŸ¡ããŸããã Â
ãããã§ã®éèŠãªç¹ã¯ãå°ããªä»£çã¢ãã«ïŒãããã·ã¢ãã«ïŒã§è©äŸ¡ããã®ã§ã¯ãªããå®éã®ã¢ãã«ãµã€ãºã§ã倿§ãªã¿ã¹ã¯ïŒMMLUãæ°åŠãæ€çŽ¢ãªã©ïŒã«ããã粟床ãåŠç¿å¹çãæšè«é床ãçŽæ¥æ¯èŒããããšã§ããããã«ãããæ¢çŽ¢çµæãæçµçãªã¢ãã«æ§èœã«çŽçµããããšãä¿èšŒãããŸãã

ããã®å³å¯ãªè©äŸ¡ã®çµæãGated DeltaNetãå šäœãšããŠæãé«ã粟床ãéæããããšã倿ããŸãããããã¯ãå ¥åããŒã¿ã«å¿ããŠæ å ±ã®æµããåçã«å¶åŸ¡ããã²ãŒãã£ã³ã°æ©æ§ãšãå¹ççãªç¶æ æŽæ°ãå¯èœã«ãããã«ã¿ã«ãŒã«ãšããäºã€ã®ç¹åŸŽã广çã«æ©èœãããããšèããããŸã ã Â
ã¹ããã3 & 4: æ°ããã¢ãã³ã·ã§ã³ãããã¯ã®èšèšãšããŒããŠã§ã¢èªèåã¢ãŒããã¯ãã£æ¢çŽ¢
æè¯ã®æ¢åãããã¯ãéžãã åŸãPostNASã¯ããã«ãã®å ãžãšé²ã¿ãŸãã
æ°ããã¢ãã³ã·ã§ã³ãããã¯ãJetBlockãã®èšèš
ãå€ãã®ç·åœ¢ã¢ãã³ã·ã§ã³ãããã¯ã¯ãæ§èœåäžã®ããã«ç³ã¿èŸŒã¿åŠçãçµã¿èŸŒãã§ããŸããããããã¯éåžžãåºå®çãªã«ãŒãã«ã䜿çšããŸã ãããã¯ãå ¥åã®å 容ã«é¢ããããåžžã«åããã¿ãŒã³ã§ç¹åŸŽãæœåºããããšããããšãæå³ããŸãã Â
ããã®éçãæç Žããããã«ãç ç©¶ããŒã ã¯æ°ããç·åœ¢ã¢ãã³ã·ã§ã³ãããã¯ãJetBlockããèšèšããŸãããJetBlockã®æå€§ã®ç¹åŸŽã¯ãå ¥åããŒã¿ãã®ãã®ããç³ã¿èŸŒã¿ã«ãŒãã«ãåçã«çæãããã«ãŒãã«ãžã§ãã¬ãŒã¿ãŒãã¢ãžã¥ãŒã«ãæèŒããŠããç¹ã§ã ã
ãããã«ãããã¢ãã«ã¯æèã«å¿ããŠç¹åŸŽæœåºã®ãã¿ãŒã³ãæè»ã«å€åãããããšãã§ãã衚çŸåãå€§å¹ ã«åäžããŸããããã«ããã®åçç³ã¿èŸŒã¿ãValueïŒVïŒããŒã¯ã³ã«é©çšããããšã§ãQueryïŒQïŒãšKeyïŒKïŒã«é©çšãããŠããåé·ãªéçç³ã¿èŸŒã¿ãåé€ã§ããããããªããå¹çãæ¹åãããŸãã ã Â
ããŒããŠã§ã¢èªèåã¢ãŒããã¯ãã£æ¢çŽ¢
ãæåŸã«ãPostNASã¯ã¢ãã«ã®åŸ®çŽ°ãªæ§é ããå®éã®ããŒããŠã§ã¢äžã§ã®æ§èœãæå€§åããããã«æé©åããŸã ãåŸæ¥ãã¢ãã«ã®å¹çæ§ã¯ããã©ã¡ãŒã¿æ°ããä»£çææšãšããŠèšèšãããããšãäžè¬çã§ããã
ãããããç ç©¶ããŒã ã¯ãç¹ã«æç« çæã®ãããªã¿ã¹ã¯ã«ãããŠã¯ããã©ã¡ãŒã¿æ°ãšå®éã®åŠçé床ãå¿ ãããçžé¢ããªãããšãèŠåºããŸãã Â
ã圌ãã¯ãããæ¬è³ªçãªããã«ããã¯ããKVãã£ãã·ã¥ã®ãµã€ãºãã«ããããšãçªãæ¢ããŸããïŒKey Finding 4ïŒãKVãã£ãã·ã¥ã®ãµã€ãºãåãã§ããã°ããã©ã¡ãŒã¿æ°ãå€å°ç°ãªã£ãŠããçæã¹ã«ãŒãããã¯ã»ãŒåãã«ãªãããšãå®éšçã«ç€ºãããã®ã§ãã Â

ã衚 2ã瀺ãããã«ãKVãã£ãã·ã¥ãµã€ãºã154 MBã«åºå®ãããŸãŸãããŒã®æ¬¡å ïŒdKâïŒãããªã¥ãŒã®æ¬¡å ïŒdVâïŒããããæ°ïŒ#headïŒãæ§ã ã«å€æŽããŠããã¹ã«ãŒãããã¯ã»ãŒ2,960 token/sååŸã§å®å®ããŠããŸãã
ããã®ç¥èŠã«åºã¥ããPostNASã¯KVãã£ãã·ã¥ãµã€ãºãäžå®ã«ä¿ã¡ãªããã粟床ãæãé«ããªããã€ããŒãã©ã¡ãŒã¿ã®çµã¿åããïŒéè²ã®è¡ïŒãæ¢çŽ¢ããŸãããããã«ãããå¹çãç ç²ã«ããããšãªãããã©ã¡ãŒã¿ããã广çã«æŽ»çšãã粟床ãããã«åäžãããããšã«æåããã®ã§ãã
ææ¡ææ³ã®æ§æã³ã³ããŒãã³ãããä»çµã¿ã®è©³çް
ãPostNASãã€ãã©ã€ã³ãçã¿åºããæé«ã®ææç©ããæ°ããç·åœ¢ã¢ãã³ã·ã§ã³ãããã¯ã§ãããJetBlockãã§ããJetBlockã®é©æ°æ§ã¯ãåè¿°ã®éããåçç³ã¿èŸŒã¿ã®å°å ¥ã«ãããŸãã

ããã®æ§é ã詳ããèŠãŠã¿ãŸããããJetBlockã¯ãå ¥åãããããŒã¯ã³ã·ãŒã±ã³ã¹ãããQïŒã¯ãšãªïŒãKïŒããŒïŒãVïŒããªã¥ãŒïŒãšããäžã€ã®ãã¯ãã«ãçæããŸãããããŸã§ã¯æšæºçãªã¢ãã³ã·ã§ã³æ©æ§ãšåæ§ã§ãã
ãããããJetBlockã¯ãããšäžŠè¡ããŠãåãå ¥åãããã«ãŒãã«ãžã§ãã¬ãŒã¿ãŒããçšããŠãVãã¯ãã«ã«é©çšããããã®å°çšã®ç³ã¿èŸŒã¿ã«ãŒãã«ããã®å Žã§çæããŸã ã Â
ããã®ã«ãŒãã«ãžã§ãã¬ãŒã¿ãŒã¯ãå°ããªãã¥ãŒã©ã«ãããã¯ãŒã¯ãšããŠå®è£ ãããŠãããå ¥åããŒã¯ã³ã®ç¹åŸŽãæããŠããã®æèã«æé©ãªç¹åŸŽæœåºãã£ã«ã¿ãŒïŒã«ãŒãã«ïŒã®éã¿ãåçã«åºåããŸãã
ãçæãããã«ãŒãã«ã¯ãVãã¯ãã«ã«å¯ŸããŠç³ã¿èŸŒã¿æŒç®ãæœããæèã«å¿ãã屿çãªç¹åŸŽãããè±ãã«æããV'ãã¯ãã«ãçæããŸãããã®åŸããã®V'ãã¯ãã«ãç·åœ¢ã¢ãã³ã·ã§ã³ã®èšç®ïŒãã®ç ç©¶ã§ã¯Gated DeltaNetã®æ©æ§ã䜿çšïŒã«çšããããŸãã
ããã®èšèšã«ãããJetBlockã¯éçãªç·åœ¢ã¢ãã³ã·ã§ã³ãæã€ç»äžçãªç¹åŸŽæœåºèœåãè¶ ããå ¥åã«å¿ããŠé©å¿çã«æ¯ãèãããã衚çŸåã®é«ãåŠçãå®çŸããŸãã
ã衚 1ã®çµæã瀺ãããã«ããã®æ¹è¯ã«ãããJetBlockã¯ããŒã¹ã©ã€ã³ã§ããGated DeltaNetãšæ¯èŒããŠãç¹ã«æ°åŠçæšè«ãæ€çŽ¢ãšãã£ãè€éãªã¿ã¹ã¯ã§ç²ŸåºŠãåäžããã€ã€ãåçã®åŠçå¹çãç¶æããããšã«æåããŠããŸã ã
å®é𿹿³
ãJet-Nemotronã®æå¹æ§ãæ€èšŒãããããç ç©¶ããŒã ã¯æ¥µããŠåºç¯ãã€å³å¯ãªè©äŸ¡å®éšã宿œããŸãã ã  è©äŸ¡ã«ã¯ãçŸä»£ã®LLMã®èœåãå€è§çã«æž¬å®ããããã®ãäž»èŠãªãã³ãããŒã¯ã¹ã€ãŒããç¶²çŸ çã«äœ¿çšãããŸããã
ãå ·äœçã«ã¯ãäžè¬çãªç¥èãšåé¡è§£æ±ºèœåãæž¬ããMMLU(-Pro)ããšãBBHããæ°åŠçèœåãæž¬ããMathematical Reasoningããåžžèçæšè«èœåãæž¬ããCommonsense Reasoningããæ å ±æ€çŽ¢èœåãæž¬ããRetrievalããããã°ã©ãã³ã°èœåãæž¬ããCodingãããããŠé·ãæèã®çè§£èœåãæž¬ããLong-Context Tasksããå«ãŸããŸã ã Â
ãæ¯èŒå¯Ÿè±¡ãšããŠãåœæã®æå 端ãè¡ãäž»èŠãªã¢ãã«çŸ€ãéžå®ãããŸãããããã«ã¯ãQwen3ãQwen2.5ãLlama3.2ãšãã£ã匷åãªãã«ã¢ãã³ã·ã§ã³ã¢ãã«ãMamba2ãRWKV7ã®ãããªä»£è¡šçãªç·åœ¢ã¢ãã³ã·ã§ã³ã¢ãã«ããããŠGemma3nãZamba2ãšãã£ããã€ããªããã¢ãã«ãå«ãŸããŠãããJet-Nemotronãæ¢åã®ã©ã®ã«ããŽãªãŒã®ã¢ãã«ã«å¯ŸããŠãåªäœæ§ãæã€ãã©ãããå³ããåãããŸãã ã Â
ãã¹ã«ãŒãããã®æž¬å®ã¯ãNVIDIA H100 GPUãæèŒããDGX H100ãµãŒããŒãšãããçŸä»£ã®AIç ç©¶ã«ãããæšæºçãªé«æ§èœèšç®ç°å¢ã§è¡ãããŸãããããã«ãããå®éšçµæã®ä¿¡é Œæ§ãšåçŸæ§ãæ ä¿ãããŠããŸãã
å®éšçµæ
ãå®éšçµæã¯ãJet-Nemotronãã粟床ããšãå¹çãã®ãã¬ãŒããªããåçã«æ¹åããããšãæç¢ºã«ç€ºããŸããã
ãæã象城çãªã®ã¯ãäž»èŠãªæ¯èŒå¯Ÿè±¡ã§ãããã«ã¢ãã³ã·ã§ã³ã¢ãã«ãQwen3-1.7B-Baseããšã®çŽæ¥å¯Ÿæ±ºã§ããJet-Nemotron-2Bã¯ãMMLU-Proãæ°åŠã¿ã¹ã¯ãªã©å€ãã®ãã³ãããŒã¯ã§Qwen3-1.7B-Baseãäžåã粟床ãéæããªããã64KããŒã¯ã³ãšããé·ãæèé·ã«ãããçæã¹ã«ãŒãããã§å®ã«47åãã®é«éåãå®çŸããŸãã ãããã¯ãã»ãŒåçã®è³¢ããæã¡ãªãããå¿çé床ã47åéãAIãå®çŸããããšãæå³ããŸãã Â

ããã®åŸåã¯ãç¹å®ã®ã¿ã¹ã¯ã«éã£ããã®ã§ã¯ãããŸããã§ãããæ°åŠïŒTable 4ïŒãåžžèæšè«ïŒTable 5ïŒãæ€çŽ¢ïŒTable 6ïŒãã³ãŒãã£ã³ã°ïŒTable 7ïŒãé·æèïŒTable 8ïŒã®ãã¹ãŠã®è©äŸ¡é åã«ãããŠãJet-Nemotronã¯æ¢åã®ãã«ã¢ãã³ã·ã§ã³ã¢ãã«ãšåçããããã¯ãã以äžã®æ§èœã瀺ããåæã«ä»ã®ãã¹ãŠã®ç·åœ¢ã»ãã€ããªããã¢ãã«ãåé§ããŸãã ã Â
ãç¹ã«æ³šç®ãã¹ãã¯ãé·æèã«ãããå¹çæ§ã®åäžã§ããå³ 6ã¯ãæèé·ãé·ããªãã«ã€ããŠJet-Nemotronã®åªäœæ§ãããã«æ¡å€§ããããšã瀺ããŠããŸãã
ã256KããŒã¯ã³ãšããéåžžã«é·ãæèé·ã«ãããŠãJet-Nemotron-2Bã¯Qwen3-1.7B-Baseãšæ¯èŒããŠãå ¥ååŠçã§æå€§6.14åãæç« çæã§æå€§53.6åãšããé©ç°çãªé床åäžãéæããŸãã ã
ãããã¯ãJet-Nemotronã®ã¢ãŒããã¯ãã£ããå°æ¥ãŸããŸãéèŠãšãªãé·æèåŠçæä»£ã«ãããŠã決å®çãªã¢ããã³ããŒãžãæã€ããšã瀺åããŠããŸãã Â

ããããã®çµæã¯ãPostNASãšããèšèšææ³ããåãªãçè«äžã®ã¢ã€ãã¢ã§ã¯ãªããå®äžçã§SOTAæ§èœãéæããã¢ãã«ãçã¿åºãããã®ã匷åãã€å®çšçãªããŒã«ã§ããããšã蚌æããŠããŸãã
èå¯
ãªããã®ææ³ãåªããŠããã®ã
ãJet-Nemotronãããã»ã©ãŸã§ã«é«ãå¹çãšç²ŸåºŠãäž¡ç«ã§ããæ ¹æ¬çãªçç±ãçè§£ããããã«ã¯ãåã³ãKVãã£ãã·ã¥ãã«çŠç¹ãåœãŠãå¿ èŠããããŸãã
ãæ¬ç ç©¶ãæããã«ããæãéèŠãªæŽå¯ã®äžã€ã¯ããçæAIã®æšè«ã¹ã«ãŒããããæ±ºå®ã¥ããæãã¯ãªãã£ã«ã«ãªèŠå ã¯ãã¢ãã«ã®ãã©ã¡ãŒã¿æ°ã§ã¯ãªããKVãã£ãã·ã¥ã®ãµã€ãºã§ããããšããç¹ã§ãïŒKey Finding 4ïŒã Â
ãåè¿°ã®éããLLMãæ¬¡ã®äžèªãçæãããã³ãŒãã£ã³ã°æ®µéã®åŠçã¯ãèšç®èœåãã®ãã®ããããã¡ã¢ãªãšã®ããŒã¿è»¢éé床ã«ãã£ãŠåŸéãããŸãïŒã¡ã¢ãªåž¯åå¹ åŸéïŒããããŠããã®ããŒã¿è»¢éã®å€§éšåãå ããã®ãã巚倧åããKVãã£ãã·ã¥ã®èªã¿æžãã§ãã
ãPostNASã®ã¹ããã1ïŒãã«ã¢ãã³ã·ã§ã³å±€ã®é 眮ïŒã¯ããã®ããã«ããã¯ãçŽæ¥çã«è§£æ¶ããŸããJet-Nemotron-2Bã¯ã28å±€ããã¢ãã³ã·ã§ã³å±€ã®ãã¡ãããã2å±€ã®ã¿ããã«ã¢ãã³ã·ã§ã³ãšããæ®ãããã¹ãŠå¹ççãªJetBlockïŒç·åœ¢ã¢ãã³ã·ã§ã³ïŒã«çœ®ãæããŸãã ã
ããã«ã¢ãã³ã·ã§ã³å±€ã¯ãå ¥åé· Â $n$ã«å¯ŸããŠ$O(n)$ã®ãµã€ãºã®KVãã£ãã·ã¥ãå¿ èŠãšããŸãããç·åœ¢ã¢ãã³ã·ã§ã³å±€ã¯$O(1)$ãã€ãŸãå ¥åé·ã«äŸåããªãåºå®ãµã€ãºã®ãã£ãã·ã¥ïŒç¶æ ïŒããå¿ èŠãšããŸããã
ããã®ã¢ãŒããã¯ãã£äžã®æ±ºæã«ãããJet-Nemotron-2Bã®KVãã£ãã·ã¥ãµã€ãºã¯ãå šå±€ããã«ã¢ãã³ã·ã§ã³ã§ããQwen3-1.7B-Baseãšæ¯èŒããŠãå®ã«47åã®1ã«ãŸã§åæžãããŸãã ãã¡ã¢ãªäžã§ããåãããããŒã¿éã47åã®1ã«ãªãã°ãã¹ã«ãŒããããçŽ47åã«åäžããã®ã¯ã極ããŠè«ççãªåž°çµã§ãã Â
ãã€ãŸããJet-Nemotronã®æåã¯ãéæ³ã®ãããªæ°ããã¢ã«ãŽãªãºã ã«ãããã®ã§ã¯ãªããLLMã®æšè«ããã»ã¹ã«ãããçã®ããã«ããã¯ãã©ãã«ããããæ£ç¢ºã«èŠæãããã®ããã«ããã¯ãè§£æ¶ããããã«ãã¢ãŒããã¯ãã£ã粟å¯ã«åèšèšããããšã«èµ·å ããã®ã§ãã
ãããã¯ãLLMã®å¹çåã«é¢ããè°è«ã®çŠç¹ããåŸæ¥ã®èšç®éïŒFLOPsïŒããã©ã¡ãŒã¿æ°ãããã¡ã¢ãªåž¯åå¹ ãšãã£ãã·ã¥ç®¡çãžãšç§»è¡ãããããã©ãã€ã ã·ãããæå³ããŸããKVãã£ãã·ã¥ã¯ãä»åŸã®LLMéçºã«ããããã ãŒã¢ã®æ³åãã®æ°ããªå¶çŽèŠå ãšããŠèªèãããã¹ãã§ãããã
ãã®ææ³ãæ¢åã®ãã®ãšæ¯èŒããåªäœæ§ã»å£äœæ§
åªäœæ§
粟床ãšå¹çã®ç»æçãªäž¡ç«
ãåŸæ¥ã®åžžèã§ãã£ããå¹çåã®ããã®ç²ŸåºŠäœäžããšãããã¬ãŒããªããæç ŽããSOTAã¬ãã«ã®ç²ŸåºŠãšæ°ååã®å¹çæ§ãåæã«å®çŸããŸããã
ã€ãããŒã·ã§ã³ã³ã¹ãã®åçãªåæž
ãPostNASã¯ãæ¢åã®åŠç¿æžã¿ã¢ãã«ãåå©çšããããšã§ãæ°ããã¢ãŒããã¯ãã£ã®èšèšãšæ€èšŒã«ãããèšç®ã³ã¹ããšæéãæ¡éãã«åæžããŸããããã«ãããããå€ãã®ç ç©¶è ãçµç¹ãã¢ãŒããã¯ãã£é©æ°ã«åå ã§ããéãéããAIéçºã®ãšã³ã·ã¹ãã å šäœã®æŽ»æ§åã«è²¢ç®ããŸãã
äœç³»çãã€åçŸå¯èœãªæ¹æ³è«
ãPostNASã¯ãçŽæã詊è¡é¯èª€ã«é Œãã®ã§ã¯ãªããããŒã¿é§åçãªæ¢çŽ¢ãšããŒããŠã§ã¢ãæèããæé©åãçµã¿åããããäœç³»çã§åçŸæ§ã®é«ãèšèšãã€ãã©ã€ã³ãæäŸããŸãã
å£äœæ§ã»èª²é¡ç¹
ãŒãããã®åŠç¿æ§èœã¯æªç¥æ°
ãPostNASã«ãã£ãŠèšèšãããã¢ãŒããã¯ãã£ããå šãã®ãŒãããäºååŠç¿ããå Žåã«ãæãããŠãã«ã¢ãã³ã·ã§ã³ã¢ãã«ãšåçã®æ§èœã«å°éã§ããã®ãã¯ããã®ç ç©¶ã®ç¯å²ã§ã¯æ€èšŒãããŠããŸããã
ãPostNASã¯ãããŸã§ãäºåŸçãªæé©åãææ³ã§ããããã®èšèšãããŒãããã®åŠç¿ãã«ãããŠãæé©ã§ããä¿èšŒã¯ãããŸãã ã Â
é«å質ãªåºçºç¹ãžã®äŸå
ããã®ææ³ã®æåã¯ãåºçºç¹ãšãªãé«å質ãªäºååŠç¿æžã¿ãã«ã¢ãã³ã·ã§ã³ã¢ãã«ã®ååšã«å€§ããäŸåããŸãããããã®ãããªããŒã¹ã¢ãã«ãååšããªããã°ãPostNASãé©çšããããšèªäœãã§ããŸããã
ãããã¯ã巚倧ã©ããéçºããåºç€ã¢ãã«ã®äžã«ãããåºç¯ãªã³ãã¥ããã£ãå¹ççãªæŽŸçã¢ãã«ãæ§ç¯ãããšãããæ°ããªåæ¥ã»åæ¥ã¢ãã«ã瀺åããŠãããšãèšããŸãã
ãJet-Nemotronã®æåã¯ãåäžã®é©æ°çãªçºæãšããããããããããããŸã§ç«¶åããŠããç°ãªãç ç©¶åéã®æè¯ã®ã¢ã€ãã¢ããç¥çã«çµ±åã»æè¯ãããçµæã§ãããšæããããšãã§ããŸãã
ãTransformerã®åŒ·åãªè¡šçŸåãNAS/OFAã®å¹ççãªæ¢çŽ¢å²åŠãMambaãªã©ã«ä»£è¡šãããç·åœ¢ã¢ãã³ã·ã§ã³ã®å¹çæ§ããããŠããŒããŠã§ã¢ãæèãããšã³ãžãã¢ãªã³ã°ã
ãPostNASã¯ããããã®èŠçŽ ãçµã¿åãããããã®ãã¡ã¿æ¹æ³è«ããæäŸããããšã§ãåãªãåŸç¶è äºãã§ã¯ãªããTransformerèªèº«ãæ¡åŒµããç¹æ®åããããšãããããæçãããã€ããªããã¢ãŒããã¯ãã£ã®æä»£ãåãæããã®ã§ãã
çµè«
æ¬ç ç©¶ã®äž»èŠãªè²¢ç®ã¯ãäžã€ã®åŽé¢ã«éçŽãããŸãã
Jet-Nemotronã¢ãã«ãã¡ããªãŒã®åµåº
ãQwen3ãLlama3.2ãšãã£ãæå 端ã®ãã«ã¢ãã³ã·ã§ã³ã¢ãã«ã«å¹æµããããã¯åé§ãã粟床ãéæãã€ã€ãçæã¹ã«ãŒããããæå€§53.6åã«ãŸã§åäžããããšãããå¹çæ§ãšç²ŸåºŠã®æ°ããªããã³ãã£ã¢ãåãæããŸãã ã Â
PostNASãã€ãã©ã€ã³ã®ç¢ºç«
ãæ¢åã®åŠç¿æžã¿ã¢ãã«ãåå©çšããããšã§ãLLMã¢ãŒããã¯ãã£ã®æ¢æ±ã«äŒŽãè«å€§ãªã³ã¹ããšãªã¹ã¯ãåçã«åæžãããæ°ããç ç©¶éçºãã©ãã€ã ã確ç«ããŸãããããã¯ãAIåéã«ãããã€ãããŒã·ã§ã³ãå éãããããå€ãã®ãã¬ã€ã€ãŒã«éæžãéãå¯èœæ§ãç§ããŠããŸã ã Â
JetBlockã®èšèš
ãå ¥åã«å¿ããŠé©å¿ããåçç³ã¿èŸŒã¿ãç·åœ¢ã¢ãã³ã·ã§ã³ã«çµ±åããæ°ããã¢ãã³ã·ã§ã³ãããã¯ãJetBlockããéçºããæ¢åã®èšèšãäžåã衚çŸåãšæ§èœãå®çŸããŸãã ã Â
ãJet-NemotronãšPostNASã¯ãåã«é«æ§èœã§å¹ççãªã¢ãã«ãæäŸããã ãã§ãªããLLMã®æªæ¥ãèšèšããããã®æ°ããæèæ³ãšããŒããããã瀺ããŠããŸããããã¯ã巚倧ãªèšç®è³æºãæãããåä»»ããã®ã¢ãããŒããããã·ã¹ãã ã®çã®ããã«ããã¯ãç¹å®ããç¥çã«ãããè§£æ¶ããã粟å¯å·¥åŠããžã®ç§»è¡ã象城ããŠããŸãã
ããã®ç ç©¶ææã¯ãããå€ãã®äººã ã髿§èœAIã®æ©æµãåãããããæç¶å¯èœã§ã¢ã¯ã»ã¹ããããæªæ¥ã®å®çŸã«åãããéèŠãªäžæ©ãšãªãã§ãããã
ããšãã
ãJet-Nemotronã®æè¡é©æ°ã¯ãLLMã®éçºã«æ°ããªæ¹åæ§ã瀺ããŸããããããŸã§ã巚倧åã«ããæ§èœåäžããäž»æµã ã£ãåéã«ãããŠããæ¢åè³ç£ã®æé©åããšããéžæè¢ãå®èšŒãããããšã®æçŸ©ã¯å€§ããã§ãããã
ãPostNASãšããææ³ã¯ãèšå€§ãªèšç®è³æºãå¿ èŠãšããã«ã¢ãã«ãæ¹è¯ã§ããéçãéããããå€ãã®ç ç©¶è ãäŒæ¥ãAIéçºã«åç»ã§ããå¯èœæ§ãåºããŠããŸãã
ãæ¬çš¿ã§è§£èª¬ãã47åãšããé床åäžã¯ãåãªãæ°å€ä»¥äžã®æå³ãæã¡ãŸããé·æåŠçã®å®çšåããªã¢ã«ã¿ã€ã å¯Ÿè©±ã®æ¹åããšããžããã€ã¹ã§ã®åäœãªã©ãçæAIã®å¿çšç¯å²ã¯ç¢ºå®ã«æ¡å€§ããŠããã§ãããã
ãåæã«ãå¹çãšæ§èœã®äž¡ç«ãšãã課é¡ãžã®åãçµã¿ã¯ãæç¶å¯èœãªAIéçºãšãã芳ç¹ãããéèŠãªäžæ©ãšãããŸãã
ãæ¬çš¿ãéããŠåŸãããã¢ã€ãã¢ãç¥èããããªãã®ããžãã¹ã«å°ãã§ã圹ç«ã€ããšãé¡ã£ãŠããŸãããããæ¬çš¿ãåèã«ãªã£ããšæããŠããã ããŸãããããã²ããããããããã©ããŒããããŠããã ãããšå±ã¿ã«ãªããŸããä»åŸãå®è·µçãªããŠããŠãAIææ°ååãå ±æããŠãããŸãã®ã§ãåŒãç¶ããèªã¿ããã ãããšå¬ããã§ãã
泚èš
ãæ¬çš¿ã¯2025幎8æã«çºè¡šãããç ç©¶è«æãåºã«å·çããŸãããAIæè¡ã¯æ¥éã«é²åããŠãããæ¬çš¿ã®å
容ã¯å·çæç¹ã§ã®ææ°æ
å ±ã«åºã¥ããŠããŸããå®åãžã®é©çšãæ€èšãããéã¯ãææ°ã®æè¡ååãšèŠå¶èŠä»¶ã確èªããããšããå§ãããŸãã
Appendix
A.2ã§ã¯PostNASå·¥çšããã³Jet-Nemotron-2Bã¢ãã«ã®åŠç¿ã«èŠããèšç®ã³ã¹ãããŸãšããããŠãããææ¡ææ³ãåŸæ¥æ¯ã§ã©ã®çšåºŠå¹ççãã瀺ãããŠããŸãã
ä»é²A.3ã§ã¯åã¢ãã«ã®ã¹ã«ãŒãããæž¬å®æã«çšããããããµã€ãºã®äžèЧã瀺ãããå
¬å¹³ãªæ¯èŒã®ããã®æ¡ä»¶ãèšãããŠããŸãã
ä»é²Bã§ã¯è¿œå ã®ãã³ãããŒã¯çµæãšããŠãåŠç¿ããŒã¿ã»ããã®åœ±é¿ã調ã¹ãå¶åŸ¡å®éšïŒTable 14ïŒããJetson OrinãRTX 3090ãšãã£ãç°ãªãããŒããŠã§ã¢äžã§ã®ã¹ã«ãŒãããæž¬å®çµæïŒTable 15ïŒãå ±åãããŠããŸããäŸãã°Jet-Nemotron-2Bã¯ãšããžããã€ã¹ã§ããJetsonäžã§ãåŸæ¥ã¢ãã«ããé«ãã¹ã«ãŒãããã瀺ããå¹
åºãç°å¢ã§æçšã§ããããšã瀺åãããŠããŸããä»é²å
šäœãéããŠãèè
ãã®è©äŸ¡ãããã«åšå°ã«è¡ããããã䌺ããçµæã®ä¿¡é Œæ§ãé«ããŠããŸãã
References
ãæ¬çš¿ã®å 容ã«ããã«æ·±ãé¢å¿ãæãããèªè ã®ããã«ãåèæç®ãããã€ã玹ä»ããŸãã
Vaswani, A., et al., "Attention is all you need," Advances in neural information processing systems, 30 (2017).
ã LLMã®åºç€ãšãªã£ãTransformerã¢ãŒããã¯ãã£ãšèªå·±ã¢ãã³ã·ã§ã³ã¡ã«ããºã ãåããŠææ¡ããããã®åéã«ãããæãéèŠãªè«æã§ããJet-Nemotronã解決ããããšããŠãã課é¡ã®åç¹ãçè§£ããããã«äžå¯æ¬ ã§ãÂ
Gu, A., & Dao, T., "Mamba: Linear-Time Sequence Modeling with Selective State Spaces," arXiv preprint arXiv:2312.00752 (2023).
ãå¹ççãªã·ãŒã±ã³ã¹ã¢ããªã³ã°ã®ããäžã€ã®äž»èŠãªã¢ãããŒãã§ããç¶æ
空éã¢ãã«ïŒSSMïŒãææ¡ããç»æçãªè«æã§ããJet-Nemotronã®ç ç©¶ã§ãæ¯èŒå¯Ÿè±¡ãšããŠè©äŸ¡ãããŠãããTransformerãšã¯ç°ãªãå¹çåã®ææ³ãçè§£ããäžã§éåžžã«æçã§ã ã Â
Sun, Y., et al., "Retentive network: A successor to transformer for large language models," arXiv preprint arXiv:2307.08621 (2023).
ã䞊ååŠç¿ãå¹ççãªæšè«ãé«ãæ§èœãäž¡ç«ãããããšãç®æããå¥ã®æåãªã¢ãŒããã¯ãã£ãRetNetããææ¡ããŠããŸããMambaãšåæ§ã«ãç·åœ¢æéã¢ããªã³ã°ã«ããã倿§ãªã¢ãããŒããç¥ãããã®éèŠãªæç®ã§ã ã Â
Cai, H., et al., "Once-for-all: Train one network and specialize it for efficient deployment," International Conference on Learning Representations (2020).
ãPostNASã®ææ³çãªæºæµãšãªã£ãç ç©¶ã§ããåŠç¿ãšã¢ãŒããã¯ãã£æ¢çŽ¢ãåé¢ãããšããã¢ã€ãã¢ããã¹ãŒããŒãããã¯ãŒã¯ã®æŠå¿µã¯ãPostNASã«ãããã¢ãã³ã·ã§ã³å±€é
çœ®ã®æ¢çŽ¢ææ³ãçè§£ããããã®éµãšãªããŸã ã Â
â ïžãéèŠïŒChatGPTã䜿ãããªãã人ã¯å
šäœã®3å²ä»¥äž
ãªããïŒã質ååãã®å·®ã§ãã
å®ã¯ãAIãã10åã®äŸ¡å€ãåŒãåºããåãã®ç«ãŠæ¹ãã«ã¯ç§åŠçãªæ³åããããŸããGoogleã20%ã«ãŒã«ã§å®èšŒããAmazonã®ããŸã¹ãæšå¥šããæ¹æ³ãå
šãŠãAIæä»£ã®æåŒ·ã¹ãã«ã奜å¥å¿åããã§å
¬éäžã
AIãç¬æã«ãçãããè¿ãä»ãåè² ã®åããç®ã¯ãåããã«ããããã®ç§å¯ã𿿥ã®ä»äºã§äœ¿ãã13ã®å®è·µããŒã«ãåçž®ããæèãAIæä»£ã®æåŒ·ã¹ãã«ã奜å¥å¿åããããã²ãã§ãã¯ããŠã¿ãŠãã ããã
ãããªãã®èãããã£ãšæå¿«ã«äŒããŸãããïŒã
ãçºå£²ãã25幎以äžãæèæ³ã®"ãã€ãã«"ãšããŠèªã¿ç¶ãããäžæœã®åèãæ¬æžã¯ãã³ã³ãµã«ãã£ã³ã°ã®åéãããã³ãŒãŒã§éçºãããæèã®æŽçè¡ããã©ãããååãã®å
šãŠã解説ããŸãã
ãçµè«ããèããæ ¹æ ãæ§é åããæè¡ã身ã«ã€ããã°ãããªãã®ã¬ããŒãããã¬ãŒã³ã¯åçã«åããããããªããŸããè€éãªåé¡ãã·ã³ãã«ã«æãã説åŸåãé£èºçã«é«ããäžçã¢ãã®ã¹ãã«ãããã«ã
ãæä»£ãè·çš®ãåãããå
šãŠã®ããžãã¹ããŒãœã³å¿
èªã®äžåã§ãã
ãåãã®æ¬è³ªãèŠæããçŸç¶ææ¡âèª²é¡æœåºâ解決çç«æ¡ãšãã3ã¹ããããäœç³»åãæŠç¥ãã¡ãŒã ã®é¢æ¥å¯Ÿçã«ãéçšããâæ¬è³ªçãªè«çæèâãäºäŸãšæŒç¿ã§éããæ§æãç¹é·ã§ãã35äžéšè¶
ã®ãæ±å€§ããŒããã·ãªãŒãºææ°äœã
ãã³ã³ããã®å£²äžãäžããã«ã¯ïŒããšãã£ãå®è·µçãªã±ãŒã¹åé¡ãéããåãªãç¥èããã¬ãŒã ã¯ãŒã¯ã§ã¯ãªããç©äºã®æ¬è³ªãèŠæããæèã®ããã»ã¹ããã®ãã®ã培åºçã«éããŸãã
ãåãªãç¥èã§ã¯ãªããäžç䜿ãããèããåããæã«å
¥ãããæ¹ã«æé©ã就掻çããããã©ã³ããžãã¹ããŒãœã³ãŸã§ãæèåãæ¬¡ã®ã¬ãã«ã«åŒãäžããå¿
èªæžã
ãªãé«åŠæŽè
ãè©æ¬ºã«éããå°éå®¶ãåæ©çãªãã¹ãç¯ãã®ãïŒæ¬æžã¯ãç¥æ§ãé«ãã»ã©é¥ããããæèã®çœ ããšããè¡æçãªçå®ãç§åŠçã«è§£æããŸãã
èè
ãããœã³ã¯ãããŒãã«è³åè³è
ã®å€±æããå»åž«ã®èª€èšºãŸã§è±å¯ãªäºäŸãåæãIQãåŠæŽã ãã§ã¯é²ããªããèªç¥ãã€ã¢ã¹ãã®æ£äœãæŽããŸããèªåã®ç¥èãéä¿¡ããæ¹å€çæèãæ ãããšã§ããããè³¢ã人ã»ã©å€§ããªéã¡ãç¯ãããããšããé説ã
æ¬æžãæç€ºããã®ã¯åãªãèŠåã§ã¯ãããŸããããã¡ã¿èªç¥ããç¥çè¬èãããªã©ãçã®è³¢ãã身ã«ã€ããå
·äœçæ¹æ³ãæºèŒãããžãã¹ã§ã®æææ±ºå®ãæè³å€æãæ¥åžžç掻ãŸã§ãããããå Žé¢ã§åœ¹ç«ã€å®è·µçãªç¥æµãè©°ãŸã£ãŠããŸãã
ãèªåã¯å€§äžå€«ããšæã£ãŠãã人ã»ã©èªãã¹ãäžåãç¥æ§ã®éçãç¥ãããšã§ãæ¬åœã®ç¥æ§ã身ã«ã€ããŸãã
çæAIæä»£ã®çŸ
éç€ãšãªã決å®çãæ±äº¬å€§åŠæŸå°Ÿã»å²©æŸ€ç 究宀ãç·åãçµéããçæAIææ°ååã培åºè§£èª¬ãChatGPTç»å Žä»¥éãæ¥éã«é²åããçæAIæè¡ã®çŸåšå°ãšæªæ¥ããæ¥æ¬ã代衚ããAIç ç©¶è
ãã¡ãå
æ¬çã«åæããŠããŸãã
2025幎3æåè¡ãæè¡ã®æ žå¿ããããžãã¹æŽ»çšãåœå
å€ã®æ³èŠå¶ããããŠå®å
šæ§ã«è³ããŸã§ãä»ç¥ãã¹ãæ
å ±ãç¶²çŸ
çã«è§£èª¬ãAIãšãŒãžã§ã³ãããã«ãã¢ãŒãã«åãšãã£ãææ°ãã¬ã³ããæŒãããŠããŸãã
çæAIããžãã¹ã«é¢ãããã¹ãŠã®æ¹ã«ãšã£ãŠå¿
æºã®äžåãæ¿åããAIæ¥çã§ç¢ºããªå€æãäžãããã«ãä¿¡é Œã§ããæ
å ±æºã§ãã
- #AI
- #çæAI
- #ChatGPT
- #AI掻çš
- #ãã¯ãããžãŒ
- #人工ç¥èœ
- #ã¢ãã«
- #LLM
- #ã€ãããŒã·ã§ã³
- #æ©æ¢°åŠç¿
- #nvidia
- #ãã£ãŒãã©ãŒãã³ã°
- #å€§èŠæš¡èšèªã¢ãã«
- #æè¡è§£èª¬
- #AIæè¡
- #transformer
- #ã¢ãã³ã·ã§ã³
- #Nemotron
- #ç·åœ¢
- #ãã¥ãŒã©ã«ã¢ãŒããã¯ãã£æ¢çŽ¢
- #JetNemotron
- #PostNAS