
å€§èŠæš¡èšèªã¢ãã«ã®ããã®åŒ·ååŠç¿
以äžã®èšäºãé¢çœãã£ãã®ã§ã軜ãèŠçŽããŸããã
1. ã¯ããã«
ãChatGPTããšããã«ç¶ããå€§èŠæš¡èšèªã¢ãã«ã(LLM)ã®ãªãªãŒã¹ã«äŒŽãããRLHFãã®éèŠæ§ãè°è«ãããŸããããããããªãã匷ååŠç¿ãããæåž«ããåŠç¿ããããèšèªã¢ãã«ã®åŠç¿ã«é©ããŠããã®ãçåã«æããŸããããæåž«ããåŠç¿ã (Instructionãã¥ãŒãã³ã°) ã§ååã§ã¯ãªãã§ããããïŒ
ç§ã¯ãããçšåºŠçŽåŸã®ããçè«çãªè°è«ãæãã€ããŸããããããŠãç¹ã«ChatGPTã®ãããªã¢ãã«ã«ã¯ã匷ååŠç¿ã®ã±ãŒã¹ãæ¯æããã ãã§ãªãããããå¿ èŠãšãã远å ã®è«æ ãããããšã«æ°ãã€ããŸããããã®è¿œå ã®è°è«ã¯ãOpenAIã®John Schulmanã«ããè¬æŒã®ïŒååéšåã§ïŒç¶ŽãããŠããŸãããã®æçš¿ã¯ã圌ã®äž»åŒµãããå€ãã®èšèã§ç¹°ãè¿ãããŸããæç¢ºã«èšããªãã£ãããšãããã€ãä»ãå ããŠããŸãã
2. æåž«ããåŠç¿ vs 匷ååŠç¿
ãæåž«ããåŠç¿ããšã匷ååŠç¿ãã®2ã€ã®ã·ããªãªãç°¡åã«èª¬æããŸãã
â äºååŠç¿
ã¯ããã«ã倧éã®ããã¹ããããšã«èšèªã¢ãã«ã®äºååŠç¿ãè¡ããŸãããã®åŠç¿ã¯ãããã¹ãã«ç¶ã次ã®ããã¹ããäºæž¬ããããšãç®æšãšããŠããŸãããã®çµæãèšèªã®å
éšè¡šçŸãååŸããŸãããããããã³ãã¥ãã±ãŒã·ã§ã³ãã«ã¯åããŠããŸãããæš¡ç¯ãšãªãæ¯ãèãã«æ²¿ã£ãŠãã¢ãã«ã調æŽããããšãããã¡ã€ã³ãã¥ãŒãã³ã°ããšåŒã³ãŸãã
â æåž«ããåŠç¿
ãæåž«ããåŠç¿ãã§ã¯ããæç€ºãã«ç¶ãæãŸãããåçããå«ãåŠç¿ããŒã¿ãã人éãäœæããŸããã匷ååŠç¿ãã¯ãã¢ãã«ããæç€ºãã«å¯ŸããŠãã®ãåçããæš¡å£ããããšãç®æšãšããŠããŸãã
â 匷ååŠç¿
ã匷ååŠç¿ãã§ã¯ããæç€ºãã®ã¿äººéãäœæãããåçãã¯æäŸããŸããã代ããã«ãã¢ãã«ããåçããçæããã¹ã³ã¢ãªã³ã°ã¡ã«ããºã ïŒäŸãã°äººéïŒããã®ãåçããè¯ããã©ããã倿ããŸããã匷ååŠç¿ãã¯ãã¢ãã«ãé«åŸç¹ãåŸãããšãã§ãããåçããçæããããšãç®æšãšããŠããŸãã
ã匷ååŠç¿ãã«ã¯ããæåž«ããåŠç¿ããããé£ããç¹ãããã€ããããŸãã1ã€ã¯ãä¿¡çšå²ãåœãŠãã§ããã¢ãã«ã¯ããŒã¯ã³ã®ã·ãŒã±ã³ã¹ãçæããã·ãŒã±ã³ã¹ã®æåŸã«ã®ã¿ã¹ã³ã¢ãä»ããããŸãããåçã®ã©ã®éšåãè¯ããæªããã倿ããã®ãé£ããã§ãããŸããåçãè©äŸ¡ããããã®ã¹ã³ã¢ãªã³ã°ã¡ã«ããºã ãå¿ èŠã§ãããèªåè©äŸ¡ãé£ããã§ãããã®ãããååŠç¿ã¹ãããã§ã人éã®ãã£ãŒãããã¯ããå¿ èŠã«ãªããããã¯éåžžã«è²»çšãšæéãããããŸãã
3. 倿§æ§ã®è°è«
èšèªã¢ãã«ã«ããããæåž«ããåŠç¿ãã«å¯ŸããæãçŽæçãªåè«ã¯ãçŸå®ã®äººéã®èšèªã«ã¯åãã¡ãã»ãŒãžãäŒããããã®å€æ§ãªæ¹æ³ãããããããããã¹ãŠæå¹ãªçãã§ããå¯èœæ§ãããããšã§ãããã®ããããåçããšããŠæå®ãããããã¹ããããããã«éžè±ããã¢ãã«ãã眰ãããããšãããããããã¢ãã«ãæ··ä¹±ããããããããŸããããŸããã¢ãã«ãåŠã³ã«ãã衚çŸã匷å¶ããäžæ¹ã§ãã¢ãã«ã¯ãã§ã«ä»£æ¿ã®çããçæããæ¹æ³ãç¥ã£ãŠãããããããŸããããããã£ãŠã匷ååŠç¿ã«ãã£ãŠæäŸããã倿§æ§ãæãŸãããšèããããŸãã
ããã¯éåžžã«çŽæçãªè°è«ã§ãããæåž«ããåŠç¿ãå®éã«éåžžã«ããŸãæ©èœããããšãã匷ååŠç¿ã¢ãã«ãåŠç¿ããäžã§ã®èª²é¡ãèæ ®ãããšãããã»ã©èª¬åŸåã®ãããã®ã§ã¯ãããŸããããããååã«æ žå¿çãªåé¡ã§ãããšã¯æããããŸã çŽåŸããŠããŸããã
4. çè«çãªè°è«
èšèªã¢ãã«ã§ã®ã匷ååŠç¿ããšãæåž«ããåŠç¿ãã®æ¯èŒã«ãããŠã説åŸåã®ãããæ£åœåã®ç¬¬äžæ©ã¯ããæåž«ããåŠç¿ãã§ã¯è¯å®çãªãã£ãŒãããã¯ã®ã¿ãå¯èœã§ããã®ã«å¯Ÿããã匷ååŠç¿ãã§ã¯åŠå®çãªãã£ãŒãããã¯ãå¯èœã§ããç¹ã§ããæ£åŒãªåŠç¿çè«ã®èгç¹ããèŠããšãäž¡è ã®éã«ã¯å€§ããªéãããããåŠå®çãã£ãŒãããã¯ã®æ¹ãã¯ããã«åŒ·åã§ãã
ãæåž«ããåŠç¿ãã§ã¯ã¢ãã«ãåãã€ãããšãåŠãã§ããŸãããããã®çš®ã®ã€ã³ã¿ã©ã¯ã·ã§ã³ã«ã¯ã匷ååŠç¿ããå¿ èŠã§ããã¢ãã«ã¯çãããç¥ã£ãŠãããå Žåãšãç¥ããªããå ŽåããããŸããåŸè ã®å Žåããæåž«ããåŠç¿ãã¯ã¢ãã«ã質åãšçããé¢é£ä»ããããã«ä¿ãããããç¹å®ã®è³ªåãšåçãã¢ãèŠãããããããã¯ã¢ãã«ãã§ããããªããšãäœãåºããŠåãã€ãããã«ãªããŸããã¢ãã«ãäœãç¥ã£ãŠãããããããªãããããæåž«ããåŠç¿ãã§ã¯é¿ããããªãåé¡ã«ãªããŸãã
ã匷ååŠç¿ãã¯ãã¢ãã«ã«åãã€ãããšãç©æ¥µçã«ä¿ãããé·æçã«ã¯ã§ããããªåçã«å¯ŸããŠæªãã¹ã³ã¢ãäžããå éšç¥èã«äŸåããããªã·ãŒããåçãæ§ããããšãåŠã³ãŸãã
6. æ žå¿çãªè°è«
èšèªã¢ãã«ãšã®å¯Ÿè©±ã«ã¯ãå°ãªããšã3ã€ã®ã¢ãŒãããããŸãã
(a) ããã¹ãããŒã¹ : ã¢ãã«ã«ãããã¹ãããšãæç€ºããæäŸã ãæäŸãããããã¹ããããšã«ããåçãæåŸ ãããŸãã
ã»ãã®ããã¹ããèŠçŽããŠ
ã»ãã®ããã¹ããã¹ãã€ã³èªã«ç¿»èš³ããŠ
ã»ãã®ããã¹ãã«ç»å Žãã人ç©åã¯ïŒ
(b) ç¥èããŒã¹ : ã¢ãã«ã«ã質åãããæç€ºããæäŸããã¢ãã«ã®å éšç¥èã«åºã¥ããïŒçå®ã®ïŒçããæåŸ ãããŸãã
ã»ã€ã³ãã«ãšã³ã¶ã®äžè¬çãªåå ã¯äœã§ãã
(c) ã¯ãªãšã€ãã£ã : ã¢ãã«ã«ã質åãããæç€ºããæäŸããäœããã®åµé çãªåºåãæåŸ ãããŸãã
ã»ïœã«ã€ããŠã®ç©èªãæžããŠãã ãã
ã匷ååŠç¿ãã®è°è«ã¯ã(b)ã®ããåããããªãã¡çå®ã§ç¢ºä¿¡ãæãŠãçããæåŸ ãããç¥èãæ±ãã質åãšãäžç¢ºããªç¶æ³ã§ãããããªãããšçãããåçãæåŠããã¢ãã«ã®èœåã«åºã¥ããŠããŸãã
ãã®çš®ã®ããåãã§ã¯ãã匷ååŠç¿ãã®åŠç¿ãå¿ èŠã§ãããªããªãããæåž«ããåŠç¿ãã§ã¯ã¢ãã«ã«åãã€ãããããã§ããæ ¹æ¬çãªåé¡ã¯ãã¢ãã«ãå éšã®ç¥èã«åºã¥ããŠçããããã«ä¿ãããã®ã«ããã®å éšç¥èãäœãå«ãã§ãããããããªãããšã§ãããæåž«ããåŠç¿ãã§ã¯ãã¢ãã«ã«è³ªåãšæ£ããçããæç€ºããæäŸãããçããåçŸããããã«åŠç¿ããŸãã
ããã§2ã€ã®ã±ãŒã¹ããããŸãã
(1) ã¢ãã«ãçãããç¥ã£ãŠãããå Žå
ãæåž«ããåŠç¿ãã¯çããšè³ªåãæ£ããé¢é£ä»ããããã«ä¿ããå°æ¥é¡äŒŒã®è³ªåã«çããããã®é¡äŒŒã®æé ãå®è¡ããããã«ä¿é²ããŸããããã¯æãŸããåäœã§ãã
(2)ã¢ãã«ãçããç¥ããªãå Žå
ãæåž«ããåŠç¿ãã¯ãšã«ããçããšè³ªåãé¢é£ä»ããããã«ã¢ãã«ãä¿ããŸãããã®å Žåã2ã€ã®éžæè¢ããããŸããã¢ãã«ãç¹å®ã®è³ªå-çãã®ãã¢ãèšæ¶ããããã«ä¿ããããããŸãããããã¯æå®³ã§ã¯ãããŸãããã广çã§ããããŸããããªããªããã¢ãã«ãäžè¬åããŠã©ããªè³ªåã«ãçããããããã«åŠã¶ããšãç®çã§ãããæç€ºã®åŠç¿ããŒã¿å
ã®è³ªåã ãã§ãªãããã§ããã¢ãã«ãäžè¬åããããã®ã§ããããããã®ã±ãŒã¹ã§ã¢ãã«ãäžè¬åãããããšã«æåããå Žåãäºå®äžãã¢ãã«ã«ã§ããããäœãåºãããããšã«ãªããŸããã¢ãã«ã«ãåããã€ãããããã«ç©æ¥µçã«ä¿ããŠããŸãã®ã§ããããã¯æªãããšã§ãã
ã¢ãã«ãäœãç¥ã£ãŠããã®ãããããªãããããæåž«ããåŠç¿ãã«ãããå®éã®æ·±å»ãªåé¡ã§ãã(2)ãé¿ããããšã¯ã§ããŸãããçå®ã®çããåºãããã«çŽç²ãªãæåž«ããåŠç¿ãã䜿ãããšã¯ã§ããããã®ããã匷ååŠç¿ããå¿ èŠã§ãã
7. ã¢ãã«ã®èžçãžã®åœ±é¿
ãOpenAIãã¯ãèšèªã¢ãã«ã®åŒ·ååŠç¿åã«å€å€§ãªåªåãæããŠããŸãããã®åæ©ã®äžã€ã¯ãã¢ãã«ãçããç¥ããªãå Žåã«åçãé¿ããããšã§ãäºå®æ§ãç宿§ã確ä¿ããããšã§ãã
æè¿ã®åŸåãšããŠãå ¬éãããŠããä»ã®èšèªã¢ãã«ãããGPTãã®åºåã§åŠç¿ããGPTã®æ¯ãèããåçŸããããšãããã®ããããŸãã
ããã¯ããæåž«ããåŠç¿ã(Instructionãã¥ãŒãã³ã°)ã«äŒŒãŠããŸããã¢ãã«ã¯ãGPTã®åçãæ£ç¢ºã«åçŸããããã«åŠç¿ãããŸããããã¯ãã¢ãã«ã«æç€ºãå®è¡ãããããã«ããŸãæ©èœããã§ããããããããç¥èãæ±ããã¯ãšãªã«åçããã±ãŒã¹ïŒbïŒã§ã¯ããŸãæ©èœããŸãããå ¬éãããŠããåºæ¬ã¢ãã«ã¯ãOpenAIã¢ãã«ãšã¯ç°ãªãäžé£ã®äºå®ãç¥ã£ãŠãããGPTã®åçãåçŸããåŠç¿ã¯ããæåž«ããåŠç¿ããæ±ããåé¡ãšåãåé¡ãåŒãèµ·ãããŸãã
解決çã¯ããããã®ã¢ãã«ãã匷ååŠç¿ãã§åŠç¿ããããšã§ããããããããã¯é«ãããã®ã§ã¯ãªãã§ããããïŒ
8. 人éã®ãã£ãŒãããã¯ã®ãªã匷ååŠç¿ã«åããŠ
é·ãéãã匷ååŠç¿ããçšããèšèªã¿ã¹ã¯ã®åŠç¿ã¯ãä¿¡é Œæ§ã®ããèªåè©äŸ¡ææšãæ¬ ããŠãããããã»ãšãã©ã®ç ç©¶è ã«ãšã£ãŠéçŸå®çã§ãããã匷ååŠç¿ãã®åŠç¿ã§ã¯ããã¹ãŠã®åŠç¿ãµã³ãã«ã«å¯ŸããŠäººéã®ãã£ãŒãããã¯ãå¿ èŠã§ãããã¯è²»çšããããã ãã§ãªããç¹ã«äœåãäœåäžãã®äŸãåŠç¿ããå¿ èŠãããã¢ãã«ã«ãšã£ãŠã¯éåžžã«é ãã§ãã
ããããã匷ååŠç¿ãã®åŠç¿ãå®çšçã«ãªããšããŸãå€§èŠæš¡ãªäºååŠç¿æžã¿èšèªã¢ãã«ãå°ãªãäŸããäœããã®æ¹æ³ã§åŠç¿ã§ããããã«ãªããŸããããããããéèŠãªã®ã¯ã人éã匷ååŠç¿ã®ã«ãŒãããé€å€ããéãéãããšã§ãã
ããã¯ãããã¹ãã«åºã¥ãã¿ã¹ã¯ã§ã¯ãæåž«ããåŠç¿ãã®ãã©ãã€ã ãéåžžã«å¹æçã§ãããå€§èŠæš¡èšèªã¢ãã«ãããã€ãã®ã¿ã¹ã¯ãéåžžã«ããŸãå®è¡ã§ããããã«ãªãããšã«ããšã¥ããŠããŸãããã®ãããªã¿ã¹ã¯ã®1ã€ã¯ããããã®2ã€ã®ããã¹ãã¯åãæå³ããã§ããã1ã€ã¯ãããã¹ãAã«ã¯ããã¹ãBã«çŸããªãäºå®ãããããã§ãã
çµéšçã«ã¯ãå€§èŠæš¡ (ãŸãã¯äžèŠæš¡) èšèªã¢ãã«ã¯ããæåž«ããåŠç¿ããçšããŠãã®ãããªã¿ã¹ã¯ãæ¯èŒç確å®ã«åŠç¿ã§ããããã§ããããã«ãããã匷ååŠç¿ãã§äœ¿çšã§ãã广çãªèªåè©äŸ¡ææšãæäŸã§ããŸãã人éãæäŸããæç€º-å¿çãã¢ã§åŠç¿ããŸããã人éãçŽæ¥æäŸããå¿çãåçŸããããšããã®ã§ã¯ãªããã¢ãã«ãç¬èªã®å¿çãçæãããæåž«ããåŠç¿ãã§åŠç¿ãããå°çšã®ããã¹ãæ¯èŒã¢ãã«ã䜿çšããŠãã¢ãã«ãçæããå¿çãšäººéãæäŸãããã®ãæ¯èŒããŸãã