
AIã¯æ¬åœã«è³¢ãã®ãïŒ283çš®é¡ã®LLMè©äŸ¡åºæºããèŠããæž¬å®ã®éçãšç²ç¹
ãLLMãæ¥éã«æ®åããäžããããã®AIã·ã¹ãã ã®èœåãæ£ç¢ºã«æž¬å®ããããšã¯æè¡éçºã®æ¹åæ§ãå·Šå³ããéèŠãªèª²é¡ãšãªã£ãŠããŸããããããçŸåšäœ¿çšãããŠããè©äŸ¡ææ³ã«ã¯çéã§ããªãæ§é çãªåé¡ãååšããŸãã
ãè©äŸ¡ããŒã¿ãåŠç¿ããŒã¿ã«æ··å ¥ãããããŒã¿æ±æãã«ãããå®å以äžã®ã¹ã³ã¢ãèšé²ãããäºäŸãè±èªåã®æåã䟡å€èгã«åã£ãè©äŸ¡åºæºã«ããäžå ¬å¹³ãªæž¬å®ããããŠãå®äžçã®åçãªç¶æ³ãžã®é©å¿åãæž¬ããªãéçãªè©äŸ¡èšèšããããã®åé¡ã¯ãAIã®çã®èœåãèŠèª€ãããéçºã®æ¹åæ§ãæªããå¯èœæ§ããããŸãã
ãæ¬çš¿ã§ã¯ã2025幎8æã«çºè¡šãããå æ¬ç調æ»ç ç©¶ãåºã«ãçŸåãã283çš®é¡ã®LLMãã³ãããŒã¯ãäœç³»çã«åæããŸããè©äŸ¡ææ³ã®å šäœåã俯ç°ããããšã§ãçŸåšã®AIè©äŸ¡ãæ±ããæ ¹æ¬çãªèª²é¡ãšãããé©åãªè©äŸ¡äœç³»ã®æ§ç¯ã«åããæéãèå¯ããŸãã
Spotifyã§ããããããé³å£°é ä¿¡ïŒããã¿ã®AIããã¯ã©ãžãªã
ãäžççµæžãã©ãŒã©ã ã®èª¿æ»ã§ã¯ã2030幎ã«å¿
èŠãªã¹ãã«ãã®ç¬¬5äœã«ã奜å¥å¿ãšç涯åŠç¿ããã©ã³ã¯ã€ã³ã奜å¥å¿ã¯è¶£å³ã§ã¯ãªãçµæžçå¿
é ã¹ãã«ã
æèãAIæä»£ã®æåŒ·ã¹ãã«ã奜å¥å¿åããã§è©³èª¬ããŠããŸãã
ãŸããã
ãããLLMãæ°åŠåé¡ãã³ãããŒã¯ã§90%ã®æ£ççãèšé²ãããšããŸãããã®æ°åã¯äžèŠå°è±¡çã§ãããå®ã¯ãã®åé¡ãšè§£çãã¢ãã«ã®åŠç¿ããŒã¿ã«å«ãŸããŠããå¯èœæ§ãããããšããåç¥ã§ããããããŸããåãã¢ãã«ãæ¥æ¬èªã®åžžèåé¡ã§ã¯60%ããæ£çã§ããªããšããäºå®ã¯ãäœãæå³ããã®ã§ããããã
ãLLMã®è©äŸ¡ã¯ãåãªãæçžŸæž¬å®ãè¶ ããæå³ãæã¡ãŸããè©äŸ¡åºæºã¯ã¢ãã«éçºã®åªå é äœã決å®ããç ç©¶æè³ã®æ¹åæ§ãå·Šå³ããŸããGLUEãSuperGLUEãšãã£ãåæã®ãã³ãããŒã¯ãèšèªçè§£ã®åäžãä¿ããããã«ãè©äŸ¡ææ³ãã®ãã®ãæè¡é²åã®éçã圢äœã£ãŠããŸãããããããã¢ãã«ã®èœåãé£èºçã«åäžããå¿çšç¯å²ãæ¡å€§ããäžã§ãæ¢åã®è©äŸ¡äœç³»ã¯éçãé²åãå§ããŠããŸãã
ãæ¬çš¿ã§ç޹ä»ãã調æ»ç ç©¶ã¯ãä¹±ç«ããè©äŸ¡åºæºãåããŠå æ¬çã«æŽçããäžã€ã®äœç³»çã«ããŽãªãŒã«åé¡ããŸããããã®åé¡ã«ãããåãã³ãããŒã¯ãäœã枬å®ããäœãèŠèœãšããŠããã®ããæç¢ºã«ãªããŸããAIã®å®åãæ£ããçè§£ããããã«ããŸãè©äŸ¡ææ³ãã®ãã®ãæ€èšŒããããšããå§ããŸãããã
èæ¯
解決ãã課é¡
ãLLMã®è©äŸ¡ã¯ãåãªãæè¡çãªæž¬å®ãè¶ ãããç§åŠãšç€ŸäŒã®æ¥ç¹ã«ãããéèŠãªèª²é¡ãšãªã£ãŠããŸãã客芳çãªè©äŸ¡åºæºã®ç¢ºç«ã¯ãæè¡éçºã®æ¹åæ§ãå®ããæœåšçãªã¹ã¯ãå®éåãã瀟äŒçä¿¡é Œãæ§ç¯ããäžã§äžå¯æ¬ ã§ãã
ããã³ãããŒã¯ã¯ããã®è€éãªèª²é¡ã«å¯Ÿããäžæ žçãªããŒã«ãšããŠæ©èœããŸããã¢ãã«ã®èœåãå®éçã«æž¬å®ããæè¡é©æ°ãä¿é²ããçŸ éç€ã®åœ¹å²ãæããã®ã§ãã
ãç ç©¶è ãéçºè ã«ãšã£ãŠããã³ãããŒã¯ã¯ç°ãªãã¢ãã«ã®é·æãšçæã客芳çã«æ¯èŒããæè¡çããã«ããã¯ãç¹å®ããããã®ããŒã¿ãæäŸããŸãã
ãããã«ããã¢ã«ãŽãªãºã ã®æé©åãã¢ãŒããã¯ãã£èšèšãä¿é²ãããŸãããŸããæšæºåãããè©äŸ¡çµæã¯ãã¢ãã«ãå®å šæ§ãå ¬å¹³æ§ãšãã£ã瀟äŒçèŠç¯ãéµå®ããŠããããšã瀺ãããŠãŒã¶ãŒã®ä¿¡é Œéžæã«è²¢ç®ããŸãã
ãããããLLMã®æ¥éãªèœååäžã¯ã枬å®å±æ©ããšãåŒã¶ã¹ãç¶æ³ãçã¿åºããŸãããã¢ãã«ãæçšããããã匷åãªèœåããéã«ãã®å šäœåãå æ¬çã«è©äŸ¡ããããšãå°é£ã«ããŠããã®ã§ãã
ããã®ç¶æ³ããããæŽç·Žããå€è§çãªæ°ãããã³ãããŒã¯ã次ã ãšçã¿åºãããèŠå ãšãªã£ãŠããŸããåãããŠããã®ã¯ããã®ã¢ãã«ã¯ã©ã®ãããåªããŠããããã ãã§ãªãããããã»ã©è€éãªååšã«ãšã£ãŠãåªããŠããããšã¯äœãæå³ããã®ãããšããæ ¹æºçãªåãã§ãã
æ¢åç ç©¶ã®æµã
ãLLMè©äŸ¡ã®æŽå²ã¯ãã¢ãã«ã®èœååäžãšè©äŸ¡ææ³ã®é²åãšã®éã®ãããªãããè»æ¡ç«¶äºãã®ãããªæ§çžãåããŠããŸãããã®å€é·ã¯ãç§ãã¡ãAIã«æ±ãããç¥æ§ãã®å®çŸ©ãã©ã®ããã«æ·±åããŠããããæ ãåºãé¡ã§ããããŸãããã®é²åã®è»è·¡ã¯ãå³1ã®ã¿ã€ã ã©ã€ã³ã«èŠèŠçã«ç€ºãããŠããŸã ã Â

ãã§ãŒãº1ïŒæ±çšèšèªçè§£ã®æä»£
ãè©äŸ¡ã®åææ®µéã§ã¯ãèªç¶èšèªçè§£ïŒNLUïŒèœåãæž¬ãããšãäžå¿ã§ããã2018幎ã«ç»å ŽããGLUE (General Language Understanding Evaluation) ã¯ãææ
åæãæéã®è«çé¢ä¿ãåãããã¹ã嫿èªèãªã©ã9ã€ã®å€æ§ãªNLUã¿ã¹ã¯ãçµ±åããç»æçãªãã³ãããŒã¯ã§ãã ã
ãããã«ãããæççã ã£ãè©äŸ¡ãçµ±äžãããç ç©¶ã®æ¹åæ§ãæç¢ºã«ãªããŸãããããããBERTã®ãããªåŒ·åãªã¢ãã«ã®ç»å Žã«ãããGLUEã¯ç¬ãéã«äººéãè¶ ããã¹ã³ã¢ãéæããããè§£ãããŠãããŸããŸãããããåããããè€éãªæšè«ãèŠããã¿ã¹ã¯ãéãã Â
ãSuperGLUEã2019幎ã«ç»å ŽããŸããããããããŸããã¢ãã«æ§èœã®æ¥éãªåäžã«ãã£ãŠé£œåç¶æ ã«è¿ã¥ããŠãããŸãã ã Â
ãã§ãŒãº2ïŒå€§èŠæš¡ã»ãã«ãã¿ã¹ã¯è©äŸ¡ã®å°é
ãæ¬¡ã«èšªããã®ã¯ãè©äŸ¡ãã©ãã€ã ã®å€§ããªè»¢ææã§ããã2021幎ã«çºè¡šãããMMLU (Massive Multask Language Understanding) ã¯ããã®è±¡åŸŽã§ã ãMMLUã¯ãåçæ°åŠããç±³åœã®æŽå²ãã³ã³ãã¥ãŒã¿ãµã€ãšã³ã¹ãæ³åŸã«è³ããŸã§ã57ãã®åºç¯ãªå°éåéã«ãããå€è¢éžæåé¡ãåé²ããŠããŸãã
ããã®æå€§ã®ç¹åŸŽã¯ãã¢ãã«ã«å€éšæ å ±ãžã®ã¢ã¯ã»ã¹ãèš±ãããZero-shotãŸãã¯Few-shotãšããå³ããæ¡ä»¶äžã§ãã¢ãã«å éšã«ãã©ã¡ãŒã¿åãããç¥èã®ã¿ãé Œãã«è§£çãããç¹ã«ãããŸãã
 ããã«ãããã¿ã¹ã¯ç¹åã®ãã¡ã€ã³ãã¥ãŒãã³ã°ã§ã¯ãªããäºååŠç¿ã§ç²åŸããåºç¯ãªç¥èãšåé¡è§£æ±ºèœåãã®ãã®ãåãããããã«ãªããLLMè©äŸ¡ã®æ°ããªæšæºã確ç«ããŸããã Â
ãã§ãŒãº3ïŒå
æ¬çã»åçè©äŸ¡ã®æä»£ãž
ãMMLUã§ãããæ§èœã飜åãå§ãããšãè©äŸ¡ã®æ çµã¿ã¯ããã«å
æ¬çãã€åçãªãã®ãžãšé²åããŸããã¹ã¿ã³ãã©ãŒã倧åŠãéçºããHELM (Holistic Evaluation of Language Models) ã¯ããã®ä»£è¡šæ Œã§ã ã
ãHELMã¯ãåäžã®ã¹ã³ã¢ã§ã¢ãã«ãè©äŸ¡ããã®ã§ã¯ãªããæ£è§£çãé 奿§ãå ¬å¹³æ§ããã€ã¢ã¹ãæ¯æ§ãå¹çæ§ãªã©ãè€æ°ã®éèŠãªåŽé¢ããã¢ãã«ãå€è§çã«è©äŸ¡ããŸãã
ããŸããæ°ããªã¿ã¹ã¯ãã¢ãã«ãç¶ç¶çã«è¿œå ããŠããããªãã³ã°ãã³ãããŒã¯ããšããæŠå¿µãå°å ¥ãããã³ãããŒã¯èªäœã®é³è åãé²ãä»çµã¿ãåãå ¥ããŠããŸããåæ§ã«ãBIG-Benchã¯ã400人以äžã®ç ç©¶è ãååããŠäœæãã204ã®ææŠçãªã¿ã¹ã¯ããæ§æãããåœæã®ã¢ãã«ã®èœåãæå³çã«è¶ ãããããªãæªç¥ã®èœåãæž¬ãããšãç®æããŸãã ã Â
ããã®æŽå²çå€é·ã瀺ãã®ã¯ãAIã«æ±ãããããç¥æ§ãã®å®çŸ©ã®æ¡å€§ã§ããããã¯ãçãèšèªèœåã®ãã¹ãïŒGLUEïŒããå§ãŸããåºç¯ãªäºå®ç¥èã®ç¢ºèªïŒMMLUïŒãžãšåºããããããŠä»ããä¿¡é Œæ§ãå ¬å¹³æ§ãå¹çæ§ãšãã£ã瀟äŒå®è£ ã«äžå¯æ¬ ãªèŠçŽ ãŸã§ãå«ãå æ¬çãªæŠå¿µïŒHELMïŒãžãšæçãã€ã€ããã®ã§ãã
ãã®ç ç©¶ã解決ãã課é¡ã»ã©ã解決ããã®ã
ãLLMè©äŸ¡ãšããåéã¯ããããŸã§æ°å€ãã®ãã³ãããŒã¯ãä¹±ç«ããå šäœåãäœç³»çã«ææ¡ããããšãå°é£ãªãè€éã§æ··æ²ãšããç¶æ³ã«ãããŸãããæ¬çš¿ã®åºç€ãšãªããµãŒãã€è«æã¯ããã®æ··æ²ãšããé åã«åããŠäœç³»çãªå°å³ãäžããããšãã詊ã¿ã§ã ã Â
ããã®ç ç©¶ã¯ã283ãã®ä»£è¡šçãªLLMãã³ãããŒã¯ãç¶²çŸ çã«åæããããããåããŠ3ã€ã®æç¢ºãªã«ããŽãªã«åé¡ããŸããããã®å顿³ããããæ¬ç ç©¶ãæç€ºããæ žå¿çãªè§£æ±ºçã§ãã
ãç¡ç§©åºã«èŠããæ°çŸã®ãã³ãããŒã¯ã®ãªã¹ãããæå³ã®ããæ§é çãªãã¬ãŒã ã¯ãŒã¯ãžãšæŽçãçŽãããšã§ãåéå šäœã®ååã課é¡ãããæ·±ããäœç³»çã«åæããããšãå¯èœã«ããŠããŸãã
ãããã«ããã®ç ç©¶ã¯åãªãæŽçã«çãŸããŸãããå°æ¥ã®ãã³ãããŒã¯éçºã®ããã®èšèšãã©ãã€ã ãæç€ºãããšãšãã«ãçŸä»£ã®LLMè©äŸ¡ãçŽé¢ããŠãã3ã€ã®é倧ãªåé¡ç¹ãéãææããŠããŸãã
ãããªãã¡ããããŒã¿æ±æã«ããã¹ã³ã¢ã®ã€ã³ãã¬ãããæåçã»èšèªçãã€ã¢ã¹ã«ããäžå ¬å¹³ãªè©äŸ¡ãããããŠããããã»ã¹ã®ä¿¡é Œæ§ããšãåçç°å¢ãã«ãããè©äŸ¡ã®æ¬ åŠãã§ã ã
ããããã®èª²é¡ãæããã«ããããšã§ãæ¬ç ç©¶ã¯ã次äžä»£ã®ããä¿¡é Œæ§ã®é«ãLLMè©äŸ¡ææ³ãæ§ç¯ããããã®ç¥çåºç€ãæäŸããŠããã®ã§ãã Â
æ¹æ³è«
ææ¡ææ³
ããã®ãµãŒãã€è«æãæç€ºããæãéèŠãªè²¢ç®ã¯ãLLMè©äŸ¡ãã³ãããŒã¯ã®åºå€§ãªäžçãçè§£ããããã®äœç³»çãªå顿³ãããªãã¡ãã¿ã¯ãœãããŒãã§ãããã®ãã¬ãŒã ã¯ãŒã¯ã¯ãç¡æ°ã«ååšãããã³ãããŒã¯ãããã®è©äŸ¡ç®çã«åºã¥ããŠ3ã€ã®äž»èŠãªæ±ã«æŽçããŸãããã®å顿³ã¯ãæ¬çš¿ã®æ®ãã®éšåãèªã¿è§£ãããã®ããŒãããããšããŠæ©èœããŸãã
3ã€ã®æ±ãšã¯ä»¥äžã®éãã§ã ã Â
æ±çšèœåãã³ãããŒã¯
ãLLMãæã€èšèªãç¥èãæšè«ãšãã£ãåºç€çãã€æ±çšçãªç¥çèœåãæž¬å®ããŸãã
ãã¡ã€ã³ç¹ååãã³ãããŒã¯
ãèªç¶ç§åŠã人æç€ŸäŒç§åŠãå·¥åŠæè¡ãšãã£ãç¹å®ã®å°éåéã«ãããLLMã®å°éç¥èãšå¿çšèœåãè©äŸ¡ããŸãã
ã¿ãŒã²ããç¹ååãã³ãããŒã¯
ãå®å šæ§ãä¿¡é Œæ§ããšãŒãžã§ã³ããšããŠã®èªåŸè¡åèœåãªã©ãç¹å®ã®ç®çãæ§è³ªã«çŠç¹ãåœãŠãŠè©äŸ¡ããŸãã
ããã®3ã€ã®ã«ããŽãªã¯ãåãªãåé¡ä»¥äžã®æå³ãæã£ãŠããŸããããã¯ãç§ãã¡ãLLMã«å¯ŸããŠæãããããå²åŠçã«ç°ãªã3çš®é¡ã®åããåæ ããŠããã®ã§ãã
æ±çšèœå
ããã®AIã¯ãæ±çšçãªç¥æ§ãšããŠãã©ã®çšåºŠããŸãæèããç¥èãæã¡ãæšè«ããããšãã§ããã®ãïŒ
ãã¡ã€ã³ç¹å
ããã®AIã¯ãç§åŠè ãæ³åŸå®¶ããšã³ãžãã¢ãšãã£ãå°éå®¶ãšããŠãã©ã®çšåºŠåªããããã©ãŒãã³ã¹ãçºæ®ã§ããã®ãïŒ
ã¿ãŒã²ããç¹å
ããã®AIãå®äžçã§éçšãããšããããã¯å®å šã§ãä¿¡é Œã§ãã責任ããäž»äœãšããŠè¡åããããšãã§ããã®ãïŒ
ããã®å顿³ãèŠèŠçã«ç€ºãããã®ããå³2ã®ã¿ã¯ãœãããŒã§ãã以éã®ã»ã¯ã·ã§ã³ã§ã¯ããã®å³ã«æ²¿ã£ãŠãåã«ããŽãªã®è©³çްãæãäžããŠãããŸãã

ææ¡ææ³ã®çŽæçãªèª¬æ
ããã®3å±€ã®å顿³ã¯ãLLMã®èœåãè©äŸ¡ããéã®èŠç¹ã®é²åãçŽæçã«ç€ºããŠããŸãããŸããç§ãã¡ã¯AIã®ãåºç€åŠåããç¥ããããšèããŸããããããæ±çšèœåãã³ãããŒã¯ãã§ããèšèªãæ£ããæãããïŒèšèªèœåïŒãåºæ¬çãªç¥èãæã£ãŠãããïŒç¥èïŒãè«ççã«èãããããïŒæšè«èœåïŒãšãã£ããããããå¿çšã®åå°ãšãªãèœåãæž¬ããŸãã
ãæ¬¡ã«ãåºç€åŠåã身ã«ã€ããAIããç¹å®ã®ãå°éè·ããšããŠéçšãããã©ããã詊ããŸãããããããã¡ã€ã³ç¹ååãã³ãããŒã¯ãã§ããæ°åŠã®åé¡ãè§£ããããïŒæ°åŠè ïŒãæ³åŸã«é¢ããå©èšãæ±ãããïŒæ³åŸå®¶ïŒãããã°ã©ã ã³ãŒããæžããããïŒãšã³ãžãã¢ïŒããããšã§ãå°éåéã§ã®å®è·µçãªèœåãè©äŸ¡ããŸãã
ããããŠæåŸã«ãå°éçãªã¹ãã«ãæã€AIããã瀟äŒã®äžå¡ããšããŠè²¬ä»»ããè¡åããšãããã©ãããæ€èšŒããŸããããããã¿ãŒã²ããç¹ååãã³ãããŒã¯ãã§ããåãã€ããªããïŒãã«ã·ããŒã·ã§ã³ïŒãå±éºãªæç€ºã«åŸããªããïŒå®å šæ§ïŒãèªåŸçã«ã¿ã¹ã¯ãéè¡ã§ãããïŒãšãŒãžã§ã³ãèœåïŒãšãã£ããå®ç€ŸäŒã§ã®ä¿¡é Œæ§ã«é¢ããéèŠãªåŽé¢ãè©äŸ¡ããŸãã
ããã®ããã«ãè©äŸ¡ã®å¯Ÿè±¡ã¯ãLLMãšãããçã®ç¥æ§ãããããèšç·Žãããå°éå®¶ãããããŠæçµçã«ã¯ã瀟äŒã§ä¿¡é Œãããååšããžãšããã®èŠæ±æ°Žæºãé«ããŠããã®ã§ãããã®éå±€çãªèŠç¹ã¯ãLLMè©äŸ¡ã®å šäœåãçè§£ããäžã§éåžžã«æå¹ãªæ çµã¿ãšãªããŸãã
ææ¡ææ³è©³çŽ°
ãããã§ã¯ãåè¿°ãã3ã€ã®ã«ããŽãªãæ±çšèœåãããã¡ã€ã³ç¹åããã¿ãŒã²ããç¹åãã«ã€ããŠãããããã®å ·äœçãªå 容ãšä»£è¡šçãªãã³ãããŒã¯ã詳述ããŸãã
æ±çšèœåãã³ãããŒã¯
ãLLMã®æãæ ¹æºçãªèœåãããªãã¡èšèªãçè§£ããç¥èãèšæ¶ããè«ççã«æèããèœåãæž¬å®ããã®ãããã®ã«ããŽãªã§ãã
1. èšèªèœå
ãèšèªèœåã®è©äŸ¡ã¯ãLLMãæ§æãæå³ãæèïŒèªçšè«ïŒãã©ãã ãæ·±ãçè§£ããŠããããæž¬ãããšãç®çãšããŠããŸãããã®é²åã¯ãåçŽãªæ£èª€å€å®ãããããè€éã§ãæµå¯Ÿçã§ãå€èšèªã«ãããè©äŸ¡ãžãšé²ãã§ããŸããã代衚çãªãã³ãããŒã¯ã®æŠèŠã衚1ã«ç€ºããŸã ã Â
ãåæã®GLUEãSuperGLUEãNLUã¿ã¹ã¯ã®çµ±äžçãªè©äŸ¡åºç€ãç¯ããåŸãã¢ãã«ã衚é¢çãªçµ±èšçæãããã«é ŒãåŸåãæããã«ãªããŸãã ã
ãããã«å¯Ÿæããããã人éã«ã¯å®¹æã ãã¢ãã«ã«ã¯é£ãããå·§åŠãªéžæè¢ãæã€HellaSwagã®ãããªãæµå¯Ÿçããã³ãããŒã¯ãç»å ŽããŸãã ã
ããŸããè©äŸ¡ã®å¯Ÿè±¡ãè±èªåã«åã£ãŠãããšãã課é¡ã«å¯Ÿããäžåœèªã®CLUEãã40èšèªãã«ããŒããXtremeã®ãããªå€èšèªãã³ãããŒã¯ãéçºãããèšèªã®å€æ§æ§ãèæ ®ããè©äŸ¡ãéèŠèŠãããããã«ãªããŸãã ã
ãè¿å¹Žã§ã¯ãçæã¿ã¹ã¯ã®è©äŸ¡ã«ãããŠãåèªã®äžèŽçã ãã§ãªãæå³çãªé¡äŒŒæ§ãè©äŸ¡ããBERTScoreããè©äŸ¡èªäœãLLMã«ä»»ããMT-Benchã®ãããªææ³ãç»å Žããè©äŸ¡ã®ç²Ÿç·»åãé²ãã§ããŸã ã Â
2. ç¥è
ãLLMããã®å éšã«ã©ãã ãåºç¯ã§æ£ç¢ºãªäžçã®ç¥èãä¿æããŠããããæž¬å®ããŸããè©äŸ¡ã¯ãå€éšè³æã®åç §ãçŠãããã¯ããŒãºãããã¯ïŒæã¡èŸŒã¿äžå¯ïŒè©Šéšãã®åœ¢åŒã§è¡ãããã®ãäžè¬çã§ãã代衚çãªãã³ãããŒã¯ã®æŠèŠã衚2ã«ç€ºããŸã ã Â

ããã®åéã®è©äŸ¡ãã©ãã€ã ã確ç«ããã®ãMMLUã§ã ã57ã®å€æ§ãªåŠååéã«ãããå€è¢éžæåé¡ãéããŠãã¢ãã«ã®ãã©ã¡ãŒã¿åãããç¥èãçŽæ¥åããŸããã¢ãã«ã®æ§èœåäžã«äŒŽããMMLUã飜åãèŠãããããã«ãªããšãããé£æåºŠã®é«ããã³ãããŒã¯ãç»å ŽããŸããã
ãäŸãã°ãéžæè¢ãå¢ãããããè€éãªæšè«ãèŠããåé¡ã远å ãã MMLU-Proãããã¡ã€ã³ã®å°éå®¶ãWebæ€çŽ¢ã§ã¯å®¹æã«çããèŠã€ãããªãããã«èšèšãããGoogle-ProofããªGPQAãããã«ãããæ°çŸã®å€§åŠé¢ã¬ãã«ã®å°éåéã«æ¡åŒµããSuperGPQAãªã©ãéçºãããŠããŸã ã
ããŸãã人éã®è©ŠéšïŒå€§åŠå ¥è©Šãªã©ïŒãçŽæ¥å©çšããAGIEvalããäžåœã®å€§åŠçµ±äžå ¥è©Šãé«èããããŒã¹ã«ããGAOKAO-Benchãªã©ã人éã®ç¥èœåºæºã«åãããè©äŸ¡ãè¡ãããŠããŸã ã Â
3. æšè«
ã圢åŒè«çã«åºã¥ãå³å¯ãªæšè«ãããåžžèçãªå€æãããã«ã¯å¿çšçãªåé¡è§£æ±ºãŸã§ãLLMã®æèèœåãå€è§çã«è©äŸ¡ããŸãã代衚çãªãã³ãããŒã¯ã®æŠèŠã衚3ã«ç€ºããŸã ã Â

ãæšè«èœåã®è©äŸ¡ã¯ã倧ãã3ã€ã®ã«ããŽãªã«åããããŸãã 第äžã«ãè«ççæšè«ã§ããããã¯ãäžéè¿°èªè«çã®æ€èšŒãè¡ãLogicNLIãã蚌æã®çæãè©äŸ¡ããProofWriterãå¶çŽå è¶³åé¡ãè§£ãZebraLogicãªã©ã圢åŒè«çã®èŠåã«åã£ãŠæ£ããæèã§ããããåããŸã ã Â
ã第äºã«ãå°éçã»åžžèçæšè«ã§ããããã«ã¯ãæ°åŠçãªåé¡ãè§£ãMathQAãå æé¢ä¿ãæšè«ããCorr2CauseããããŠäººéã®åé¡è§£æ±ºã«äžå¯æ¬ ãªæé»çã»å€æ®µéã®æšè«ã詊ãStrategyQAãªã©ãå«ãŸããŸã ã Â
ã第äžã«ãå¿çšã»æèçæšè«ã§ããããã¯ãè€æ°ã®æ å ±ãçµ±åããŠçµè«ãå°ãåºãèœåãè©äŸ¡ããŸããäŸãã°ãè€æ°ã®ææžã«ãŸãããæ å ±ãç¹ãåãããŠçããèŠã€ãåºãããã«ããããæšè«ããèŠæ±ããHotpotQAããç§åŠçãªç¥èã®å¿çšãåãARCã代衚çã§ã ã Â
ãã¡ã€ã³ç¹ååãã³ãããŒã¯
ãLLMãæ±çšçãªèœåã ãã§ãªããç¹å®ã®å°éåéã§ãå°éå®¶ããšããŠæ©èœã§ããããè©äŸ¡ããŸãã
1. èªç¶ç§åŠ
ãè«ççå³å¯æ§ãšæ§é åãããç¥èãæ±ããããèªç¶ç§åŠã¯ãLLMã®èœåãè©Šãæ Œå¥œã®èå°ã§ãã代衚çãªãã³ãããŒã¯ã®æŠèŠã衚4ã«ç€ºããŸã ã Â

æ°åŠ
ãå°åŠæ ¡ã¬ãã«ã®æç« åé¡ã§ããGSM8Kããã髿 ¡ã»å€§åŠã¬ãã«ã®ç«¶ææ°åŠåé¡ãéããMATHãããã«ã¯æ°åŠãªãªã³ããã¯ã¬ãã«ã®åé¡ã圢åŒçå®ç蚌æã«çŠç¹ãåœãŠãOmni-MATHãMiniF2FãŸã§ãé£æåºŠã¯å€å²ã«ããããŸã ã Â
ç©çåŠ
ã倧åŠã¬ãã«ã®ç§åŠåé¡å šè¬ãæ±ãSciBenchã«å ããç©çåŠã«ç¹åããUGPhysicsããå³è¡šã®èªè§£ãäžå¯æ¬ ãªãã«ãã¢ãŒãã«åé¡ãå«ãPhysics Arenaãªã©ããããŸã ã Â
ååŠ
ãåºç€ç¥èãåãChemEvalãç§åŠè«æã®èªè§£åã詊ãChemistryQAããããŠç¹ã«éèŠãªã®ããå±éºãªååŠç©è³ªã®åææ³ãªã©ã«é¢ããå¿çã®å®å šæ§ãè©äŸ¡ããChemSafety Benchã§ããååŠåéã§ã¯ãæ£ããã ãã§ãªãå®å šæ§ã極ããŠéèŠãªè©äŸ¡è»žãšãªããŸã ã Â
çç©åŠ
ãçç©å»åŠåéã®è«æèªè§£ãè©äŸ¡ããPubMedQAããéºäŒåãã¿ã³ãã¯è³ªã®è€éãªçžäºäœçšãããã¯ãŒã¯ïŒçç©åŠççµè·¯ïŒã«é¢ããæšè«èœåãåãBioMazeãªã©ãå°éæ§ã®é«ããã³ãããŒã¯ãååšããŸã ã Â
2. 人æç€ŸäŒç§åŠ
ãæ°å€çãªæ£è§£ãååšããªãããšãå€ããã®åéã§ã¯ãæèã®çè§£ããã¥ã¢ã³ã¹ã®ææ¡ãå«ççãªå€æãšãã£ãããã人éçãªèœåãåãããŸãã代衚çãªãã³ãããŒã¯ã®æŠèŠã衚5ã«ç€ºããŸã ã Â

æ³åŸ
ãæ³çç¥èã®èšæ¶ã»çè§£ã»å¿çšãè©äŸ¡ããLawBenchïŒäžåœæ³ïŒãã162ãã®å€æ§ãªæ³çæšè«ã¿ã¹ã¯ãå«ãLegalBenchïŒç±³åœæ³ïŒãªã©ãååœã®æ³å¶åºŠã«ç¹åãããã³ãããŒã¯ãéçºãããŠããŸã ã Â
æè²
ãå®éã®æè²çŸå Žã®ã·ããªãªã«åºã¥ããçåŸåãã»æåž«åãã®ã¿ã¹ã¯ãå¹ åºãã«ããŒããEduBenchãªã©ãç»å ŽããŠããŸã ã Â
å¿çåŠ
ãå¿çåŠã®è©Šéšåé¡ãããŒã¹ã«ããCPsyExamããå®éã®ã«ãŠã³ã»ãªã³ã°å¯Ÿè©±ãè©äŸ¡ããCPsyCounãªã©ãå°éç¥èãšå¯Ÿè©±èœåã®äž¡é¢ããè©äŸ¡ãè¡ãããŸã ã Â
éè
ãéèãã¥ãŒã¹ã®åæããæ ªäŸ¡äºæž¬ãéèç¥èãåãåé¡ãŸã§ãç¶²çŸ ããFinEvalãPIXIUãªã©ãå®è·µçãªå¿çšåã詊ãããŸã ã Â
3. å·¥åŠæè¡
ããã®åéã§ã¯ãèšèªçãªæµæ¢ã以äžã«ãè«ççãªå³å¯æ§ãæ©èœçãªæ£ããããããŠå°éç¥èã®æ£ç¢ºãªé©çšãæ±ããããŸããè©äŸ¡ã¯ãã°ãã°ãçæãããã³ãŒããèšèšãå®éã«åäœãããã©ãããšãããå®è¡ããŒã¹ãã§è¡ãããŸãã代衚çãªãã³ãããŒã¯ã®æŠèŠã衚6ã«ç€ºããŸã ã Â

ãœãããŠã§ã¢å·¥åŠ
ãèªç¶èšèªã®æç€ºããPythonã³ãŒããçæããèœåãè©äŸ¡ããHumanEvalã¯ããã®åéã®èåãçååšã§ããããè€éãªãã®ãšããŠãå®éã®GitHubäžã®èª²é¡ã解決ãããSWE-benchããããŒã¿ããŒã¹èšèªSQLãžã®å€æèœåãæž¬ãSpiderãªã©ãéçºã©ã€ããµã€ã¯ã«ã®æ§ã
ãªåŽé¢ãè©äŸ¡å¯Ÿè±¡ãšãªããŸã ãÂ
å°éå·¥åŠåé
ããœãããŠã§ã¢ä»¥å€ã«ããåå°äœèšèšã«äœ¿ãããããŒããŠã§ã¢èšè¿°èšèªïŒVerilogïŒã®çæèœåãæž¬ãVerilogEvalãæ©æ¢°èšèšã®ããã®CADã¹ã¯ãªããçæãè©äŸ¡ããCADBenchãèªç©ºå®å®å·¥åŠã®å°éç¥èãåãAviation-Benchmarkãªã©ã極ããŠå°éæ§ã®é«ããã³ãããŒã¯ãååšããŸã ã Â
ã¿ãŒã²ããç¹ååãã³ãããŒã¯
ãLLMãå®ç€ŸäŒã§è²¬ä»»ãã圢ã§éçšããããã«äžå¯æ¬ ãªãç¹å®ã®æ§è³ªãèœåã«çŠç¹ãåœãŠãè©äŸ¡ã§ãã
1. ãªã¹ã¯ãšä¿¡é Œæ§
ãLLMãããããæœåšçãªãªã¹ã¯ãç¹å®ãããã®ä¿¡é Œæ§ãå®éåããŸããããã¯ãLLMã®ç€ŸäŒå®è£ ã«ãããæéèŠèª²é¡ã®äžã€ã§ãã代衚çãªãã³ãããŒã¯ã®æŠèŠã衚7ã«ç€ºããŸã ã Â

å®å šæ§
ãã¢ãã«ãæå®³ãéå«ççããããã¯å±éºãªã³ã³ãã³ããçæããªãããè©äŸ¡ããŸããç¹ã«ãã¢ãã«ã®å®å šå¯Ÿçãåé¿ããããšããå·§åŠãªããã³ããïŒæµå¯Ÿçããã³ããããžã§ã€ã«ãã¬ã€ã¯ãšåŒã°ããïŒã«å¯Ÿããèæ§ããã¹ãããJailbreakBenchãHarmBenchãéèŠã§ã ã Â
ãã«ã·ããŒã·ã§ã³
ãã¢ãã«ãäºå®ã«åºã¥ããªãããã£ãšããããåãçæããçŸè±¡ããã«ã·ããŒã·ã§ã³ãšåŒã³ãŸããTruthfulQAã¯ã人éãé¥ããã¡ãªèª€è§£ãã¢ãã«ãæš¡å£ããªããããã¹ãããããšã§ããã®åé¡ã®æ·±å»åºŠã枬å®ããŸã ã Â
é 奿§
ãããã³ããã®å ããªèšãæãããæå³çã«äœããããã€ãºïŒæµå¯Ÿçå ¥åïŒã«å¯ŸããŠãã¢ãã«ã®åºåãå®å®ããŠããããè©äŸ¡ããŸããAdvGLUEã¯ãæ§ã ãªæµå¯Ÿçæ»æææ³ãçšããŠã¢ãã«ã®è匱æ§ãäœç³»çã«è©äŸ¡ãããã³ãããŒã¯ã§ã ã Â
ããŒã¿æŒæŽ©
ãã¢ãã«ãåŠç¿ããŒã¿ã«å«ãŸããå人æ å ±ãæ©å¯æ å ±ãæå³ããåºåããŠããŸããªã¹ã¯ãè©äŸ¡ããŸããWikiMIAãªã©ããã®çš®ã®è©äŸ¡ã«çšããããŸã ã Â
2. ãšãŒãžã§ã³ã
ãLLMãåãªãå¿ççæåšãšããŠã§ã¯ãªããèªåŸçã«ç®æšãèšå®ããèšç»ãç«ãŠãããŒã«ã䜿ããç°å¢ãšå¯Ÿè©±ããªããã¿ã¹ã¯ãéè¡ããããšãŒãžã§ã³ãããšããŠè©äŸ¡ãããæ°ãããã©ãã€ã ã§ãã代衚çãªãã³ãããŒã¯ã®æŠèŠã衚8ã«ç€ºããŸã ã Â

ããšãŒãžã§ã³ãèœåã®è©äŸ¡ã¯ãè€æ°ã®æ¬¡å ã«ããããŸããAgentBenchã¯ãOSæäœãããŒã¿ããŒã¹ã¯ãšãªãWebãã©ãŠãžã³ã°ãªã©8ã€ã®ç°ãªãç°å¢ã§LLMãšãŒãžã§ã³ãã®ç·åçãªèœåãè©äŸ¡ããå æ¬çãªãã³ãããŒã¯ã§ã ã
ãè©äŸ¡è»žã«ã¯ãèšç»èœåãããŒã«äœ¿çšãšãã£ã  ç¹å®ã®èœåè©äŸ¡ãè€æ°ã®èœåãå調ãããŠè€éãªã¿ã¹ã¯ã解決ããçµ±åçèœåè©äŸ¡ãç¹å®ã®å°éé åïŒç§åŠç ç©¶ãéèãªã©ïŒã§ã®ã¿ã¹ã¯éè¡èœåãæž¬ããã¡ã€ã³ç¿ç床è©äŸ¡ããããŠæµå¯Ÿçãªç¶æ³äžã§ã®å®å šæ§ãåãå®å šæ§ã»ãªã¹ã¯è©äŸ¡ãªã©ãå«ãŸããŸã ã Â
ææ¡ææ³ã®æ§æã³ã³ããŒãã³ãããä»çµã¿ã®è©³çް
ããã®ãµãŒãã€è«æãææ¡ããææ³ãããªãã¡3ã€ã®ã«ããŽãªããæãã¿ã¯ãœãããŒã¯ãåãªãåé¡ãªã¹ãã§ã¯ãããŸãããããã¯ãLLMã®èœåãè©äŸ¡ããããã®å€å±€çãªèŠç¹ãæäŸãããæ§é åãããæ çµã¿ã§ãã
æ§æã³ã³ããŒãã³ã
ããã®æ çµã¿ã¯ã3ã€ã®äž»èŠã«ããŽãªïŒæ±çšèœåããã¡ã€ã³ç¹åãã¿ãŒã²ããç¹åïŒãšããã®äžã«é£ãªããµãã«ããŽãªïŒäŸïŒèšèªèœåãç¥èãæšè«ïŒãããã«ãã®äžã«ããåå¥ã®ãã³ãããŒã¯çŸ€ïŒäŸïŒGLUE, MMLU, AgentBenchïŒããæ§æãããŠããŸãã
ããã®éå±€æ§é ã«ããã倧å±çãªèŠç¹ããåå¥ã®è©äŸ¡ææ³ã®è©³çްãŸã§ãã·ãŒã ã¬ã¹ã«çè§£ãæ·±ããããšãå¯èœã«ãªããŸãã
ä»çµã¿
ããã®ã¿ã¯ãœãããŒã®ä»çµã¿ã¯ãè©äŸ¡ã®ãç®çãã«åºã¥ããŠãã³ãããŒã¯ãæŽçããããšã«ãããŸãããäœã枬ãããã®ãïŒããšããåããèµ·ç¹ã«ããããšã§ãããããã®ãã³ãããŒã¯ãæã€æçŸ©ãéçãæç¢ºã«ãªããŸãã
ãäŸãã°ããã¢ãã«ã®åºç€çãªèšèªçè§£åãç¥ããããã®ã§ããã°æ±çšèœåã«ããŽãªã®GLUEãåç §ãããã¢ãã«ãèªåŸçã«Webã§æ å ±åéã§ããããç¥ããããã®ã§ããã°ã¿ãŒã²ããç¹åã«ããŽãªã®AgentBenchãåç §ããããšãã£ãå ·äœçãªæéãåŸãããŸãã
ããã®æ§é åãããã¢ãããŒãããããæ··æ²ãšããŠããLLMè©äŸ¡ã®äžçã«ç§©åºããããããç ç©¶è ãéçºè ãèªèº«ã®ç®çã«åã£ãé©åãªè©äŸ¡ææ³ãéžæããããã«ã¯æ°ããªè©äŸ¡ææ³ãèšèšããããã®åŒ·åãªåºç€ãšãªãã®ã§ãã
å®é𿹿³
ãLLMã®è©äŸ¡ã¯ãå ·äœçã«ã©ã®ããã«è¡ãããã®ã§ããããããã®ããã»ã¹ã¯ã人éã®è©Šéšã«äŒŒããäœç³»çãªæé ã«åºã¥ããŠããŸãã
ããŸããè©äŸ¡ã®åå°ãšãªãã®ããæšæºåãããããŒã¿ã»ããã§ããããã¯ãç¹å®ã®èœåãæž¬ãããã«æ³šææ·±ãèšèšããã質åãã¿ã¹ã¯ã®éåäœã§ããäŸãã°ãMMLUã§ããã°57åéã®å€è¢éžæåé¡ãHumanEvalã§ããã°ããã°ã©ãã³ã°ã®åé¡ãšãããæ€èšŒããããã®åäœãã¹ããå«ãŸããŸã ã Â
ãæ¬¡ã«ãè©äŸ¡å¯Ÿè±¡ã®LLMã«å¯ŸããŠããã®ããŒã¿ã»ããããã®åé¡ãæç€ºãããŸãããã®éãå€ãã®å ŽåãZero-shotãŸãã¯Few-shotãšããèšå®ãçšããããŸããããã¯ãã¢ãã«ãç¹å®ã®ã¿ã¹ã¯åœ¢åŒã«éå°ã«é©å¿ïŒãã¡ã€ã³ãã¥ãŒãã³ã°ïŒããããšãªããäºååŠç¿ã§ç²åŸããæ±çšçãªèœåãçºæ®ã§ãããã確ãããããã§ã ã Â
ãã¢ãã«ã¯æç€ºãããåé¡ã«å¯ŸããŠå¿çãçæããŸãããã®åºåã¯åéããããããããçšæãããæ£è§£ãšæ¯èŒãããŸãããããŠãè©äŸ¡ææšãçšããŠã¹ã³ã¢ãç®åºãããŸããå€è¢éžæåé¡ã§ããã°æ£è§£çãæ å ±æ€çŽ¢ã§ããã°F1ã¹ã³ã¢ïŒé©åçãšåçŸçã®èª¿åå¹³åïŒãªã©ãäžè¬çã«çšããããŸã ã Â
ãããããLLMã®èœåãåäžããèŠçŽã察話ã®ãããªèªç±åœ¢åŒã®å¿çãå¢ããã«ã€ããåçŽãªæ£è§£ãšã®äžèŽåºŠã ãã§ã¯è©äŸ¡ãå°é£ã«ãªããŸãããããã§ç»å Žããã®ãããLLM-as-a-JudgeïŒå¯©æ»å¡ãšããŠã®LLMïŒããšãã驿°çãªææ³ã§ã ã
ãããã¯ãè©äŸ¡å¯Ÿè±¡ã®ã¢ãã«ãçæããå¿çããGPT-4ã®ãããªéåžžã«é«æ§èœãªå¥ã®LLMã«æç€ºããããã®å¿çã¯è«ççãããæç€ºã«å¿ å®ãããäžå¯§ãªèšèé£ããããšãã£ãããããããå®çŸ©ãããåºæºã«åºã¥ããŠæ¡ç¹ãããæ¹æ³ã§ãã
ã人éã«ããè©äŸ¡ã¯è³ªãé«ãäžæ¹ã§ã³ã¹ããšæéãããããããLLM-as-a-Judgeã¯ããã®ã¹ã±ãŒã©ããªãã£ãšæè»æ§ããããªãŒãã³ãšã³ããªã¿ã¹ã¯è©äŸ¡ã®æ°ããªæšæºãšãªãã€ã€ãããŸãã
ããã®ææ³ã®èæ¯ã«ã¯ãLLMãçæèœåã ãã§ãªããé«åºŠãªè©äŸ¡èœåããåããŠãããšããçºèŠããããŸããã€ãŸããAIã®èœåãæž¬ãããã«ãAIãã®ãã®ãå©çšãããšãããèªå·±èšåçãªãããéåžžã«åŒ·åãªãã©ãã€ã ã確ç«ãããã®ã§ãã Â
å®éšçµæ
283çš®é¡ã®ãã³ãããŒã¯ãæããã«ããLLMã®å®æ
ãæ¬ç ç©¶ã®å æ¬çåæã«ãããçŸä»£ã®LLMã瀺ãèœåååžã®è©³çްãªå°å³ãåããŠæããã«ãªããŸããã283ã®ãã³ãããŒã¯ãéããŠåŸãããçµæã¯ãåŸæ¥ã®æççãªçè§£ã倧ããæŽæ°ãããã®ã§ãã
åºç€èœåã«ãããæ¥éãªé£œåçŸè±¡
ãæãé¡èãªçºèŠã¯ãåºç€çèšèªçè§£ã¿ã¹ã¯ã«ãããè©äŸ¡ã®ã倩äºå¹æãã§ããGLUEãã³ãããŒã¯ã§ã¯ã2018幎ã®ç»å Žããå ã2幎ã§ãããã¢ãã«ã人éã®å¹³åã¹ã³ã¢ïŒ87.1ïŒãè¶ ããçŸåšã§ã¯è€æ°ã®ã¢ãã«ã90ç¹ä»¥äžãèšé²ããŠããŸãã
ãSuperGLUEã§ãåæ§ã®åŸåãèŠããããã¯ãå·®å¥åèŠå ãšããŠæ©èœããŠããŸãããMMLUã«ãããŠããGPT-4ãClaude 3ãªã©ã®ææ°ã¢ãã«ã¯86.4%ãšããé«ã¹ã³ã¢ãéæããè©äŸ¡åºæºãšããŠã®æå¹æ§ãéçã«è¿ã¥ããŠããŸãã
å°éé åã«ãããé¡èãªæ§èœæ Œå·®
ãäžæ¹ã§ãå°éæ§ã®é«ãé åã§ã¯å€§ããªæ§èœæ Œå·®ã芳å¯ãããŸãããç¹ã«æ³šç®ãã¹ãã¯ãæ°åŠåéã®FrontierMathãã³ãããŒã¯ã«ãããçµæã§ãããã®ãã³ãããŒã¯ã¯çŸåœ¹ã®æ°åŠç ç©¶è ãäœæããæå 端ã®åé¡ã§æ§æãããŠãããGPT-4ã§ãã2%æªæºã®æ£ççããéæã§ããŸããã§ããã
ã人éã®æ°åŠè ãå¹³åããŠæ°æéãããŠè§£ãåé¡ã«å¯ŸããLLMã¯åºæ¬çãªæŠå¿µçè§£ã®æ®µéã§èºããŠããŸããåæ§ã«ã倧åŠé¢ã¬ãã«ã®å°éç¥èãåãGPQAïŒGraduate-level Google-Proof Q&AïŒã§ã¯ãæé«æ§èœã¢ãã«ã§ã39%ã®æ£ççã«çãŸãã該åœåéã®å士å·ä¿æè ã®å¹³åïŒ65%ïŒã倧ããäžåããŸããã
ãããã¯ãLLMã衚é¢çãªç¥èã®èšæ¶ã§ã¯åªããŠãããã®ã®ãæ·±ãå°éççè§£ãšå¿çšèœåã«ãããŠæ ¹æ¬çãªéçãæ±ããŠããããšã瀺ããŠããŸãã
å®å šæ§è©äŸ¡ã«ãããè匱æ§ã®äœç³»ç確èª
ãJailbreakBenchã«ããè©äŸ¡ã§ã¯ã調æ»å¯Ÿè±¡ãšãªã£ãäž»èŠãª10ã¢ãã«å šãŠããå°ãªããšã1çš®é¡ä»¥äžã®æµå¯Ÿçããã³ããã«ãã£ãŠå®å šæ©èœãåé¿ãããããšã倿ããŸãããç¹ã«æžå¿µãããã®ã¯ãæåçã®é«ãã§ãã
ãå·§åŠã«èšèšãããããã³ããã¯å¹³åããŠ42%ã®ç¢ºçã§å®å šãã£ã«ã¿ãŒãçªç Žããæå®³ãªå 容ãçæãããããšã«æåããŠããŸãã
ãããã«ãAdvGLUEãçšããé 奿§è©äŸ¡ã§ã¯ãå ¥åããã¹ãã«äººéã«ã¯èªèã§ããªãçšåºŠã®æåãå ããã ãã§ãã¢ãã«ã®æ§èœãå¹³å35%äœäžããããšã確èªãããŸããããã®è匱æ§ã¯ãå®äžçã§ã®éçšã«ãããŠæ·±å»ãªã»ãã¥ãªãã£ãªã¹ã¯ãããããå¯èœæ§ããããŸãã
ãšãŒãžã§ã³ãèœåã«ãããåçšã»ãªãŒãã³ãœãŒã¹éã®æ±ºå®çæ Œå·®
ãAgentBenchã«ãã8ç°å¢ã§ã®ç·åè©äŸ¡ã§ã¯ãè峿·±ãäºæ¥µåã芳å¯ãããŸãããGPT-4ã¯å šã¿ã¹ã¯ã§å¹³å88.4%ã®æåçã瀺ããäžæ¹ãæé«æ§èœã®ãªãŒãã³ãœãŒã¹ã¢ãã«ïŒLLaMA-2 70BïŒã¯35.8%ã«çãŸããŸããã
ãç¹ã«é¡èãªå·®ãçŸããã®ã¯ãè€æ°ã¹ãããã®èšç»ç«æ¡ãèŠããã¿ã¹ã¯ã§ãåçšã¢ãã«ãå¹³å73%ã®æåçã瀺ããã®ã«å¯ŸãããªãŒãã³ãœãŒã¹ã¢ãã«ã¯18%ãšããäœãçµæãšãªããŸããã
æåçãã€ã¢ã¹ã®å®éçæž¬å®
ãå€èšèªè©äŸ¡ã«ãããŠéèŠãªçºèŠããããŸãããåäžã¢ãã«ãè±èªçMMLUã§82%ã®ã¹ã³ã¢ãèšé²ããªãããäžåœèªçïŒC-EvalïŒã§ã¯61%ãã¢ã©ãã¢èªçã§ã¯53%ãŸã§äœäžããããšã確èªãããŸããã
ããã®å·®ã¯åãªã翻蚳ã®åé¡ã§ã¯ãªããèšç·ŽããŒã¿ã®åãã«èµ·å ããæ§é çãªãã€ã¢ã¹ã§ããããšãã詳现ãªèª€çåæããæããã«ãªããŸããã
åçç°å¢ã§ã®æ§èœå£å
ãéçãªäžåäžç圢åŒãšåçãªå¯Ÿè©±åœ¢åŒã§ã®è©äŸ¡æ¯èŒã§ã¯ãè¡æçãªçµæãåŸãããŸãããMT-Benchã«ããå€ã¿ãŒã³å¯Ÿè©±è©äŸ¡ã§ã¯ãåäžã¿ãŒã³ã§90%以äžã®æ£ççã瀺ããã¢ãã«ãã5ã¿ãŒã³ä»¥äžã®å¯Ÿè©±ã§ã¯å¹³åããŠ47%ãŸã§æ§èœãäœäžããŸããã
ãç¹ã«æèã®ä¿æãšäžè²«æ§ã®ç¶æã«ãããŠãã¿ãŒã³æ°ã®å¢å ã«äŒŽãææ°é¢æ°çã«æ§èœãå£åããåŸåã芳å¯ãããŸããã
ããŒã¿æ±æã®åœ±é¿åºŠæž¬å®
ãWikiMIAãçšããåæã«ãããè©äŸ¡ããŒã¿ã»ããã®æ±æåºŠãå®éåãããŸããã調æ»å¯Ÿè±¡ãšãªã£ãäž»èŠãã³ãããŒã¯ã®13%ã«ãããŠããã¹ãåé¡ã®äžéšãŸãã¯å šéšãäž»èŠãªLLMã®èšç·ŽããŒã¿ã«å«ãŸããŠããå¯èœæ§ãé«ãããšã倿ããŸãããæ±æãçããããã³ãããŒã¯ã§ã¯ãå¹³åããŠå®éã®èœåãã15-20%é«ãã¹ã³ã¢ãèšé²ãããŠãããšæšå®ãããŸãã
è©äŸ¡ãã©ãã€ã ã®è»¢æç¹
ããããã®çµæã瀺ãã®ã¯ãLLMè©äŸ¡ãéèŠãªè»¢æç¹ã«å·®ãæãã£ãŠãããšããäºå®ã§ããåŸæ¥ã®éçã§åäžæ¬¡å çãªè©äŸ¡ã§ã¯ããã¯ãã¢ãã«ã®çã®èœåãéçãæããããšãã§ããŸãããç¹ã«ãå®äžçã§ã®å¿çšãèæ ®ãããšãçŸåšã®è©äŸ¡äœç³»ã«ã¯æ ¹æ¬çãªåèšèšãå¿ èŠã§ããããšãæç¢ºã«ãªããŸããã
ãæ¬ç ç©¶ã§æããã«ãªã£ãæãéèŠãªç¥èŠã¯ãLLMããäžåäžãªèœåååžãæã€ç¹æ®ãªç¥çã·ã¹ãã ãã§ãããšããç¹ã§ããåºç¯ãªç¥èãšç¹å®ã¿ã¹ã¯ã§ã®åè¶ããæ§èœã瀺ãäžæ¹ã§ãæ·±ãçè§£ãè«ççäžè²«æ§ãå®å šæ§ã«ãããŠè匱æ§ãæ±ããŠããŸãã
ããã®ããã³ã€ãªå€©æããšããŠã®ç¹æ§ãæ£ç¢ºã«çè§£ããé©åã«è©äŸ¡ããããšããä»åŸã®AIéçºãšç€ŸäŒå®è£ ã«ãããŠäžå¯æ¬ ãšãªããŸãã
èå¯
ãªããã®ææ³ãåªããŠããã®ã
ãçŸä»£ã®LLMè©äŸ¡ãã©ãã€ã ãç¹ã«HELMãAgentBenchã«ä»£è¡šãããæ°ããäžä»£ã®ãã³ãããŒã¯ãããªãåŸæ¥ã®åäžã¿ã¹ã¯ã®ãªãŒããŒããŒããããåªããŠããã®ã§ããããããã®åªäœæ§ã¯ãè©äŸ¡ã®èŠç¹ããç·æ¬çãããã蚺æçããžãšæ ¹æ¬çã«ç§»è¡ããç¹ã«ãããŸãã
ãåŸæ¥ã®ãã³ãããŒã¯ã¯ãã¢ãã«ã«æçµçãªãæçžŸããã€ããããšïŒç·æ¬çè©äŸ¡ïŒãäž»ç®çãšããŠããŸãããããããHELMã®ãããªçŸä»£ã®ãã¬ãŒã ã¯ãŒã¯ã¯ãåäžã®æ£è§£çã ãã§ãªããé 奿§ãå ¬å¹³æ§ããã€ã¢ã¹ãå¹çæ§ãšãã£ãå€é¢çãªåŽé¢ããã¢ãã«ãè©äŸ¡ããŸã ã
ãããã«ãããã¢ãã«ã®ç·åçãªãããã¡ã€ã«ãæããã«ãªããç¹å®ã®åŒ·ã¿ãšåŒ±ç¹ãç¹å®ããããšïŒèšºæçè©äŸ¡ïŒãå¯èœã«ãªããŸããäŸãã°ãããã¢ãã«ã¯ç¥èåé¡ã«ã¯åŒ·ãããæµå¯Ÿçå ¥åã«éåžžã«åŒ±ãããšãã£ãå ·äœçãªæŽå¯ãåŸãããŸãã Â
ãããã«ãAgentBenchã®ãããªãã³ãããŒã¯ã¯ãè©äŸ¡ã®å Žãéçãªè³ªçå¿çãããåçã§ã€ã³ã¿ã©ã¯ãã£ããªç°å¢ãžãšç§»ããŸãã ã
ãããã«ãããã¢ãã«ãåã«ç¥èãåçããã ãã§ãªããç°å¢ããã®ãã£ãŒãããã¯ã«é©å¿ããããŒã«ã䜿ãããªããé·æçãªç®æšã«åãã£ãŠè¡åãèšç»ã»å®è¡ããèœåãããªãã¡å®äžçã§äŸ¡å€ãçã¿åºãããã«äžå¯æ¬ ãªèœåãæž¬å®ã§ããããã«ãªããŸããã Â
ããã®ã蚺æçãã¢ãããŒããžã®è»¢æããããçŸä»£ã®ãã³ãããŒã¯ã®æå€§ã®äŸ¡å€ã§ããããã¯ãåã«ã¢ãã«éã®åªå£ã決ããã ãã§ãªããç ç©¶éçºã«å¯ŸããŠã次ã«äœãæ¹åãã¹ããããšããå ·äœçã§å®è¡å¯èœãªæéãäžããŸããAIã責任ãã圢ã§éçºããŠããäžã§ããã®ãããªæ·±ãæŽå¯ãæäŸããè©äŸ¡ææ³ã¯äžå¯æ¬ ãªã®ã§ãã
ãã®ææ³ãæ¢åã®ãã®ãšæ¯èŒããåªäœæ§ã»å£äœæ§
ãLLMè©äŸ¡ã®åéã¯ç®èŠãŸãã鲿©ãéããŠããŸãããããã®æ£åœæ§ãšä¿¡é Œæ§ãæ ¹åºããæºãããã3ã€ã®é倧ãªèª²é¡ã«çŽé¢ããŠããŸãããããã®èª²é¡ãçè§£ããããšã¯ããã³ãããŒã¯ã®ã¹ã³ã¢ãéµåã¿ã«ããããã®èåŸã«ããè€éããæ¹å€çã«æããäžã§æ¥µããŠéèŠã§ãã
1. ããŒã¿æ±æã«ããã¹ã³ã¢ã®ã€ã³ãã¬
ãæãæ·±å»ãã€èå»¶ããŠããåé¡ããããŒã¿æ±æãã§ã ãããã¯ããã³ãããŒã¯ã®è©äŸ¡çšããŒã¿ïŒè©Šéšåé¡ïŒããæå³ããããŠã¢ãã«ã®åŠç¿çšããŒã¿ã«å«ãŸããŠããŸãçŸè±¡ãæããŸãã
ãLLMã¯Webå šäœããèšå€§ãªããŒã¿ãåŠç¿ãããããå ¬éãããŠãããã³ãããŒã¯ã®ããŒã¿ã»ãããåŠç¿ã³ãŒãã¹ã«æ··å ¥ããå¯èœæ§ã¯åžžã«ååšããŸããÂ
ããã®æ±æãèµ·ãããšãè©äŸ¡ã¯ã¢ãã«ã®çã®èœåãæž¬ããå®å詊éšããããåãªããèšæ¶åãã¹ãããžãšå€è³ªããŠããŸããŸããã¢ãã«ã¯åé¡ãè§£ããŠããã®ã§ã¯ãªããåŠç¿æã«èŠããçãããæãåºããŠããã ããããããŸããã
ããã®çµæããã³ãããŒã¯ã®ã¹ã³ã¢ã¯äººçºçã«ã€ãäžãããïŒã€ã³ãã¬ïŒãç ç©¶è ã瀟äŒã«AIã®èœåã«é¢ãã誀ã£ãèªèãäžãã鲿©ã®å¹»æ³ãçã¿åºããŠããŸããŸãã
ããã®åé¡ã«å¯ŸåŠãããããç ç©¶è ãã¡ã¯æ§ã ãªå¯Ÿçãè¬ããŠããŸããäžã€ã¯ãGPQAã®ããã«ãå°éå®¶ãWebæ€çŽ¢ã§ã¯çããèŠã€ãããªãããã«æå³çã«äœæãããGoogle-Proofããªåé¡ãçšããã¢ãããŒãã§ã ã
ãããäžã€ã¯ãLiveBenchã®ããã«ãè©äŸ¡ã®çŽåã«äœæããããã¢ãã«ãéå»ã«èŠãå¯èœæ§ã®ãªãæ°ããããŒã¿ïŒäŸãã°ãé²è¡äžã®ããã°ã©ãã³ã°ã³ã³ãã¹ãã®åé¡ïŒãçšãããåçãã³ãããŒã¯ãã§ã ãããŒã¿æ±æãšã®æŠãã¯ãä¿¡é Œã§ããè©äŸ¡ãç¶æããããã®ãçµããã®ãªãåªåãªã®ã§ãã Â
2. æåçã»èšèªçãã€ã¢ã¹ã«ããäžå ¬å¹³ãªè©äŸ¡
ãçŸåšã®ãã³ãããŒã¯ã®å€ãã¯ãæé»ã®åæãšããŠè±èªãäžå¿ãšããæ¬§ç±³ã®æåçèŠç¹ãåæ ããŠæ§ç¯ãããŠããŸã ãäŸãã°ãç±³åœã®æŽå²ãæ³åŸã«é¢ããç¥èãåãåé¡ã¯ããã®æååã§åŠç¿ããã¢ãã«ã«æå©ã«åããŸãã Â
ããã®åãã¯ã2ã€ã®å€§ããªåé¡ãåŒãèµ·ãããŸãã第äžã«ãAIã®é²æ©ã«é¢ããåã£ããäžå®å šãªèŠæ¹ãçã¿åºããŸããè±èªå以å€ã®èšèªãæååã§åªããèœåãæã€ã¢ãã«ããäžåœã«äœãè©äŸ¡ãããå¯èœæ§ããããŸãã
ã第äºã«ãæ¢åã®ç€ŸäŒçäžå¹³çãAIã®äžçã§åçç£ããå¢å¹ ããããªã¹ã¯ããããŸããç¹å®ã®æåã䟡å€èгã«åºã¥ããAIããæšæºããšèŠãªãããããšã§ãæè¡ã®æ©æµãäžçã«å ¬å¹³ã«è¡ãæž¡ããªããªãæããããã®ã§ãã
ããã®èª²é¡ã«å¯Ÿãã解決çã¯ãè©äŸ¡ã®å€æ§æ§ã確ä¿ããããšã§ããäžåœèªã®ç·åçãªãã³ãããŒã¯ã§ããC-Evalãã40ãã®èšèªã察象ãšããXtremeã®ãããªãå€èšèªã»å€æåãã³ãããŒã¯ã®éçºãšæ®åã¯ãããå ¬å¹³ã§ã°ããŒãã«ãªAIè©äŸ¡ãå®çŸããããã®äžå¯æ¬ ãªäžæ©ã§ã ã Â
3. ãããã»ã¹ã®ä¿¡é Œæ§ããšãåçç°å¢ãã«ãããè©äŸ¡ã®æ¬ åŠ
ãåŸæ¥ã®ãã³ãããŒã¯ã®å€ãã¯ãã¢ãã«ãå°ãåºãããæçµçãªçãããæ£ãããã©ããã®ã¿ãè©äŸ¡ããŸãããããããã®çãã«è³ããæèããã»ã¹ããè«ççã§ä¿¡é Œã§ãããã®ãã¯åãããŸãããã¢ãã«ã¯ã誀ã£ãæšè«ã®æ«ã«å¶ç¶æ£ããçãã«ãã©ãçãããããããããã¯çã®ç¥æ§ã®èšŒãšã¯èšããŸããã
ãå ããŠãã»ãšãã©ã®ãã³ãããŒã¯ã¯ãäžåäžç圢åŒã®ãéçããªãã®ã§ããããããå®äžçã«ãããç¥æ§ã®çºæ®ã¯ãç°å¢ãšå¯Ÿè©±ãããã£ãŒãããã¯ããåŠã³ãè¡åãä¿®æ£ããŠãããåçããªããã»ã¹ã§ããéçãªãã¹ãã§ã¯ãã¢ãã«ããšãŒãžã§ã³ããšããŠé©å¿çã«æ¯ãèãèœåãè©äŸ¡ããããšã¯ã§ããŸãã ã Â
ããã®å顿èãããè©äŸ¡ã®çŠç¹ã¯ãçµæããããããã»ã¹ããžãšç§»ãã€ã€ãããŸããã¢ãã«ã«CoTã説æããããã®è«ççãªæ£ãããè©äŸ¡ããã¢ãããŒããäžè¬åããŠããŸããããã«ãAgentBenchã«ä»£è¡šãããã€ã³ã¿ã©ã¯ãã£ããªãšãŒãžã§ã³ãããŒã¹ã®ãã³ãããŒã¯ã¯ãåçãªç°å¢ã§ã®åé¡è§£æ±ºèœåãè©äŸ¡ããæ¬¡ãªãããã³ãã£ã¢ãšããŠæ³šç®ãããŠããŸã ã Â
ãããã3ã€ã®èª²é¡ã¯ãåãªãæè¡çãªåé¡ã«çãŸããŸãããããã¯ãç§ãã¡ããAIã®ç¥æ§ãã«ã€ããŠäœãç¥ã£ãŠããã®ãããããŠãã®ç¥èã®åºç€ãã©ãã»ã©ç¢ºããªã®ããåããæ ¹æºçãªãèªèè«ç屿©ããæ§æããŠããŸãã
ãããç§ãã¡ã®æž¬å®ããŒã«ãæ±æãããåããéçã§ãããªãã°ãAIã®é²æ©ã«é¢ããæã ã®çè§£å šäœããèãåå°ã®äžã«ç¯ãããŠããããšã«ãªããŸãããããã£ãŠããè©äŸ¡ã®åé¡ãã解決ããããšã¯ãããåªããã¢ãã«ãæ§ç¯ããããšãšå°ãªããšãåçã«éèŠãªèª²é¡ã§ãããšèšããã§ãããã
çµè«
ãæ¬çš¿ã§ã¯ã283ã®LLMãã³ãããŒã¯ãå æ¬çã«åæãã調æ»ç ç©¶ãéããŠãè©äŸ¡äœç³»ã®å šäœåãšãã®æ§é ç課é¡ãæããã«ããŸããã3ã€ã®ã«ããŽãªïŒæ±çšèœåããã¡ã€ã³ç¹åãã¿ãŒã²ããç¹åïŒãžã®äœç³»çåé¡ã«ãããåŸæ¥ã¯æççã ã£ãè©äŸ¡ç ç©¶ãåããŠçµ±åçãªå°å³ãšããŠå¯èŠåãããŸããã
ããã®åæããæµ®ãã³äžãã£ãã®ã¯ãçŸä»£ã®LLMè©äŸ¡ãæ±ããæ ¹æ¬çãªãã©ããã¯ã¹ã§ããè©äŸ¡ææ³ãæŽç·Žãããã«ã€ããæž¬å®ã®éçããŸãé®®æã«ãªããšããé説çç¶æ³ãçããŠããŸãã
ãæ±çšæ§ã远æ±ããã°æ·±ããç ç²ã«ãªããå°éæ§ã远æ±ããã°é©çšç¯å²ãçãŸããæè¡çå³å¯ããæ±ããã°å®äžçãšã®ä¹é¢ãçããå®çšæ§ãéèŠããã°ç§åŠçåŠ¥åœæ§ãæãªãããããããã®ãã¬ãŒããªãã¯ãåçŽãªæè¡çæ¹è¯ã§ã¯è§£æ±ºã§ããªãæ§é çãªèª²é¡ã§ãã
è©äŸ¡ãã©ãã€ã ã®è»¢æç¹
ãæ¬ç ç©¶ãæããã«ããæãéèŠãªç¥èŠã¯ãLLMè©äŸ¡ããèœåã®æž¬å®ããããæ¯ãèãã®è©äŸ¡ããžãšè³ªç転æãè¿«ãããŠãããšããäºå®ã§ããããŒã¿æ±æã«ããæž¬å®ã®ä¿¡é Œæ§äœäžãæåçååã«ããäžå ¬å¹³æ§ãéçè©äŸ¡ãšåççŸå®ã®ã®ã£ãããšãã3ã€ã®æ ¹æ¬ç課é¡ã¯ãåŸæ¥ã®è©äŸ¡æ çµã¿ã®éçãé²åãããŸããã
ãç¹ã«æ³šç®ãã¹ãã¯ãåäžã¿ãŒã³ã§90%ã®æ£ççã瀺ãã¢ãã«ãã5ã¿ãŒã³ä»¥äžã®å¯Ÿè©±ã§ã¯47%ãŸã§æ§èœãäœäžãããšããäºå®ã§ãããã®åçãªå·®ã¯ãçŸåšã®è©äŸ¡äœç³»ãå®äžçã§ã®AIæ§èœã系統çã«é倧è©äŸ¡ããŠããå¯èœæ§ã瀺åããŸããããã«ãè±èªã§ã®è©äŸ¡ãšéè±èªã§ã®è©äŸ¡ã«ããã20-30%ã®æ§èœå·®ã¯ãæè¡è©äŸ¡ã«ãããæåçå ¬å¹³æ§ãšããæ°ããªèª²é¡ãæèµ·ããŠããŸãã
次äžä»£è©äŸ¡äœç³»ãžã®èŠæ±äºé
ä»åŸã®LLMè©äŸ¡ã«æ±ããããã®ã¯ã4ã€ã®æ¬è³ªçç¹æ§ã§ãã
åçé©å¿æ§
ãéçãªäžåäžçãããç°å¢å€åã«å¿ããç¶ç¶çè©äŸ¡ãžã®ç§»è¡ãã¢ãã«ãæèãä¿æãããã£ãŒãããã¯ããåŠç¿ããè¡åãä¿®æ£ããèœåã®æž¬å®ã
ããã»ã¹ã®éææ§
ãæçµçãªçãã®æ£ç¢ºãã ãã§ãªããæšè«éçšã®è«ççåŠ¥åœæ§ãšèª¬æå¯èœæ§ã®è©äŸ¡ãå¶ç¶ã®æ£è§£ãšçã®çè§£ã®åºå¥ã
æåçå ææ§
ãç¹å®ã®èšèªãæååã«åããªããçã«ã°ããŒãã«ãªè©äŸ¡åºæºã®ç¢ºç«ãæè¡æ Œå·®ã®åºå®åãé²ãå ¬å¹³ãªæž¬å®äœç³»ã
ãªã¹ã¯èæ§
ãäºæãã¬å ¥åãæªæãã䜿çšã«å¯Ÿããå ç¢æ§ã®äœç³»çæ€èšŒã瀟äŒå®è£ ã«ãããå®å šæ§ã®äºåè©äŸ¡ã
ãLLMè©äŸ¡ã®çã®èª²é¡ã¯ãæž¬å®æè¡ã®æ¹è¯ã ãã§ãªãããç¥æ§ãšã¯äœãããAIã«äœãæåŸ ãããããšããæ ¹æ¬çãªåããšåãåãããšã«ãããŸãã283ã®ãã³ãããŒã¯ã瀺ã倿§æ§ã¯ããã®åãã«å¯Ÿããåäžã®çããååšããªãããšãç©èªã£ãŠããŸãã
ãæ¬ç ç©¶ãæããã«ããã®ã¯ãå®ç§ãªè©äŸ¡ã¯ååšããªããšããçŸå®ãšãããã§ããªããããè¯ãè©äŸ¡ã远æ±ãç¶ããå¿ èŠæ§ã§ããè©äŸ¡ã®éçãèªèãã€ã€ããã®éçãæŒãåºããŠããããã®æ¹å€çãã€å»ºèšçãªå§¿å¢ããããAIã®è²¬ä»»ããçºå±ãæ¯ããåºç€ãšãªããŸãã
ãã¢ãã«ããäœãã§ãããããæž¬å®ããæä»£ããããã©ã®ããã«ç€ŸäŒãšå ±åãã¹ããããè©äŸ¡ããæä»£ãžããã®è»¢ææã«ãããŠãæ¬èª¿æ»ãæäŸããå æ¬çãªåæãšäœç³»çãªåé¡ã¯ãä»åŸã®è©äŸ¡ç ç©¶ã®ç€ç³ãšãªãã§ãããã
ããšãã
ãLLMã®é©ç°çãªçºå±ã«äŒŽãããäœããã£ãŠè¯ãã¢ãã«ãšãããããšããåãã¯ãŸããŸãéèŠã«ãªã£ãŠããŸããè©äŸ¡ãã³ãããŒã¯ã¯ãã®åãã«çããçŸ éç€ã§ãããæã«åé¡ç¹ãæµ®ã圫ãã«ããæã«æ°ããªæ¹åæ§ã瀺ãååšã§ãã
ãæ¬çš¿ã§åãäžãããµãŒãã€ã¯ããŸãã«çŸç¶ã®çŸ éç€ãåç¹æ€ãæ¹è¯ããããã®è²Žéãªã¬ã€ãã©ã€ã³ãæäŸããŠãããŸãããä»åŸãLLMã瀟äŒã«åºã溶ã蟌ãã«ã€ããè©äŸ¡ã®ããæ¹ã瀟äŒãšæè¡ã®æ©æž¡ããšããŠé²åããŠããããšã§ãããã
ãæ¬çš¿ãéããŠåŸãããã¢ã€ãã¢ãç¥èããããªãã®ããžãã¹ã«å°ãã§ã圹ç«ã€ããšãé¡ã£ãŠããŸãããããæ¬çš¿ãåèã«ãªã£ããšæããŠããã ããŸãããããã²ããããããããã©ããŒããããŠããã ãããšå±ã¿ã«ãªããŸããä»åŸãå®è·µçãªããŠããŠãAIææ°ååãå ±æããŠãããŸãã®ã§ãåŒãç¶ããèªã¿ããã ãããšå¬ããã§ãã
泚èš
ãæ¬çš¿ã¯2025幎8æã«çºè¡šãããç ç©¶è«æãåºã«å·çããŸãããAIæè¡ã¯æ¥éã«é²åããŠãããæ¬çš¿ã®å
容ã¯å·çæç¹ã§ã®ææ°æ
å ±ã«åºã¥ããŠããŸããå®åãžã®é©çšãæ€èšãããéã¯ãææ°ã®æè¡ååãšèŠå¶èŠä»¶ã確èªããããšããå§ãããŸãã
References
ãæ¬çš¿ã§æ±ã£ãããŒãã«ã€ããŠãããã«æ·±ãæ¢æ±ãããèªè ã®ããã«ãåºç€ãšãªãè«æãããã€ã玹ä»ããŸãã
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N.,... & Polosukhin, I. (2017). "Attention is all you need." Advances in neural information processing systems, 30.
ãçŸä»£ã®LLMã®åºç€ãšãªã£ãŠããTransformerã¢ãŒããã¯ãã£ãææ¡ãããç»æçãªè«æã§ãã
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018). "GLUE: A multi-task benchmark and analysis platform for natural language understanding." arXiv preprint arXiv:1804.07461.
ãè€æ°ã®NLUã¿ã¹ã¯ãçµ±åããæ±çšèšèªçè§£ã¢ãã«ã®è©äŸ¡ã«ãããæšæºã確ç«ããGLUEãã³ãããŒã¯ãææ¡ããè«æã§ãã
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2021). "Measuring massive multitask language understanding." International Conference on Learning Representations.
ã57ã®å°éåéã«ãããåºç¯ãªç¥èããŒãã·ã§ãã/ãã¥ãŒã·ã§ããèšå®ã§è©äŸ¡ããMMLUãã³ãããŒã¯ãå°å
¥ããLLMè©äŸ¡ã®ãã©ãã€ã ã転æãããè«æã§ãã
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M.,... & Wu, Y. (2022). "Holistic evaluation of language models." arXiv preprint arXiv:2211.09110.
ãæ£è§£çã ãã§ãªããé 奿§ãå
¬å¹³æ§ããã€ã¢ã¹ãªã©è€æ°ã®åŽé¢ããã¢ãã«ãå
æ¬çã«è©äŸ¡ããããªãã³ã°ãã³ãããŒã¯ãã§ããHELMãææ¡ããè«æã§ãã
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H.,... & Dong, Y. (2024). "Agentbench: Evaluating llms as agents." International Conference on Learning Representations.
ãLLMãèªåŸçãªãšãŒãžã§ã³ããšããŠè©äŸ¡ããããã®ã8ã€ã®å€æ§ãªç°å¢ãããªãå
æ¬çãªãã³ãããŒã¯AgentBenchãææ¡ããè«æã§ãã
â ïžãéèŠïŒChatGPTã䜿ãããªãã人ã¯å
šäœã®3å²ä»¥äž
ãªããïŒã質ååãã®å·®ã§ãã
å®ã¯ãAIãã10åã®äŸ¡å€ãåŒãåºããåãã®ç«ãŠæ¹ãã«ã¯ç§åŠçãªæ³åããããŸããGoogleã20%ã«ãŒã«ã§å®èšŒããAmazonã®ããŸã¹ãæšå¥šããæ¹æ³ãå
šãŠãAIæä»£ã®æåŒ·ã¹ãã«ã奜å¥å¿åããã§å
¬éäžã
AIãç¬æã«ãçãããè¿ãä»ãåè² ã®åããç®ã¯ãåããã«ããããã®ç§å¯ã𿿥ã®ä»äºã§äœ¿ãã13ã®å®è·µããŒã«ãåçž®ããæèãAIæä»£ã®æåŒ·ã¹ãã«ã奜å¥å¿åããããã²ãã§ãã¯ããŠã¿ãŠãã ããã
ãããªãã®èãããã£ãšæå¿«ã«äŒããŸãããïŒã
ãçºå£²ãã25幎以äžãæèæ³ã®"ãã€ãã«"ãšããŠèªã¿ç¶ãããäžæœã®åèãæ¬æžã¯ãã³ã³ãµã«ãã£ã³ã°ã®åéãããã³ãŒãŒã§éçºãããæèã®æŽçè¡ããã©ãããååãã®å
šãŠã解説ããŸãã
ãçµè«ããèããæ ¹æ ãæ§é åããæè¡ã身ã«ã€ããã°ãããªãã®ã¬ããŒãããã¬ãŒã³ã¯åçã«åããããããªããŸããè€éãªåé¡ãã·ã³ãã«ã«æãã説åŸåãé£èºçã«é«ããäžçã¢ãã®ã¹ãã«ãããã«ã
ãæä»£ãè·çš®ãåãããå
šãŠã®ããžãã¹ããŒãœã³å¿
èªã®äžåã§ãã
ãåãã®æ¬è³ªãèŠæããçŸç¶ææ¡âèª²é¡æœåºâ解決çç«æ¡ãšãã3ã¹ããããäœç³»åãæŠç¥ãã¡ãŒã ã®é¢æ¥å¯Ÿçã«ãéçšããâæ¬è³ªçãªè«çæèâãäºäŸãšæŒç¿ã§éããæ§æãç¹é·ã§ãã35äžéšè¶
ã®ãæ±å€§ããŒããã·ãªãŒãºææ°äœã
ãã³ã³ããã®å£²äžãäžããã«ã¯ïŒããšãã£ãå®è·µçãªã±ãŒã¹åé¡ãéããåãªãç¥èããã¬ãŒã ã¯ãŒã¯ã§ã¯ãªããç©äºã®æ¬è³ªãèŠæããæèã®ããã»ã¹ããã®ãã®ã培åºçã«éããŸãã
ãåãªãç¥èã§ã¯ãªããäžç䜿ãããèããåããæã«å
¥ãããæ¹ã«æé©ã就掻çããããã©ã³ããžãã¹ããŒãœã³ãŸã§ãæèåãæ¬¡ã®ã¬ãã«ã«åŒãäžããå¿
èªæžã
ãªãé«åŠæŽè
ãè©æ¬ºã«éããå°éå®¶ãåæ©çãªãã¹ãç¯ãã®ãïŒæ¬æžã¯ãç¥æ§ãé«ãã»ã©é¥ããããæèã®çœ ããšããè¡æçãªçå®ãç§åŠçã«è§£æããŸãã
èè
ãããœã³ã¯ãããŒãã«è³åè³è
ã®å€±æããå»åž«ã®èª€èšºãŸã§è±å¯ãªäºäŸãåæãIQãåŠæŽã ãã§ã¯é²ããªããèªç¥ãã€ã¢ã¹ãã®æ£äœãæŽããŸããèªåã®ç¥èãéä¿¡ããæ¹å€çæèãæ ãããšã§ããããè³¢ã人ã»ã©å€§ããªéã¡ãç¯ãããããšããé説ã
æ¬æžãæç€ºããã®ã¯åãªãèŠåã§ã¯ãããŸããããã¡ã¿èªç¥ããç¥çè¬èãããªã©ãçã®è³¢ãã身ã«ã€ããå
·äœçæ¹æ³ãæºèŒãããžãã¹ã§ã®æææ±ºå®ãæè³å€æãæ¥åžžç掻ãŸã§ãããããå Žé¢ã§åœ¹ç«ã€å®è·µçãªç¥æµãè©°ãŸã£ãŠããŸãã
ãèªåã¯å€§äžå€«ããšæã£ãŠãã人ã»ã©èªãã¹ãäžåãç¥æ§ã®éçãç¥ãããšã§ãæ¬åœã®ç¥æ§ã身ã«ã€ããŸãã
çæAIæä»£ã®çŸ
éç€ãšãªã決å®çãæ±äº¬å€§åŠæŸå°Ÿã»å²©æŸ€ç 究宀ãç·åãçµéããçæAIææ°ååã培åºè§£èª¬ãChatGPTç»å Žä»¥éãæ¥éã«é²åããçæAIæè¡ã®çŸåšå°ãšæªæ¥ããæ¥æ¬ã代衚ããAIç ç©¶è
ãã¡ãå
æ¬çã«åæããŠããŸãã
2025幎3æåè¡ãæè¡ã®æ žå¿ããããžãã¹æŽ»çšãåœå
å€ã®æ³èŠå¶ããããŠå®å
šæ§ã«è³ããŸã§ãä»ç¥ãã¹ãæ
å ±ãç¶²çŸ
çã«è§£èª¬ãAIãšãŒãžã§ã³ãããã«ãã¢ãŒãã«åãšãã£ãææ°ãã¬ã³ããæŒãããŠããŸãã
çæAIããžãã¹ã«é¢ãããã¹ãŠã®æ¹ã«ãšã£ãŠå¿
æºã®äžåãæ¿åããAIæ¥çã§ç¢ºããªå€æãäžãããã«ãä¿¡é Œã§ããæ
å ±æºã§ãã