
RLHF (人éã®ãã£ãŒãããã¯ããã®åŒ·ååŠç¿) ã®å³è§£
以äžã®èšäºãé¢çœãã£ãã®ã§ã軜ãèŠçŽããŸããã
ã»Illustrating Reinforcement Learning from Human Feedback (RLHF)
1. ã¯ããã«
èšèªã¢ãã«ã¯ã人éã®å ¥åããã³ãããã倿§ã§èª¬åŸåã®ããããã¹ããçæããããšã§ãããæ°å¹Žãç®èŠãŸããææããããŠããŸãããããããè¯ããããã¹ããã©ããã¯ã䞻芳çã§æèã«äŸåãããããå®çŸ©ããããšãå°é£ã§ãã
ãè¯ããããã¹ããçæããããã®æå€±é¢æ°ã®èšèšã¯é£ãããã»ãšãã©ã®èšèªã¢ãã«ã¯ããŸã åçŽãªæ¬¡ã®ããŒã¯ã³äºæž¬æå€±ïŒã¯ãã¹ãšã³ããããŒãªã©ïŒã§åŠç¿ããŠããŸãããã®æå€±èªäœã®æ¬ ç¹ãè£ãããã«ãBLEUãROUGEãªã©ã®äººéã®å¥œã¿ãããããæããããã«èšèšãããææšãå®çŸ©ãããŠããŸããããããããã¯ãèœå枬å®ã«ãããŠæå€±é¢æ°ããé©ããŠãŸãããçæãããããã¹ããåçŽãªã«ãŒã«ã§åç §æ¯èŒãããããå¶éããããŸãã
çæãããããã¹ãã«äººéã®ãã£ãŒãããã¯ãæ§èœã®å°ºåºŠãšããŠäœ¿çšããããããã«äžæ©é²ãã§ãã®ãã£ãŒãããã¯ãæå€±ãšããŠäœ¿çšããŠã¢ãã«ãæé©åãããããããšã¯ã§ããªãã§ããããïŒãããããRLHFãã®ã¢ã€ãã¢ã§ãã匷ååŠç¿ã䜿çšããŠã人éã®ãã£ãŒãããã¯ã§èšèªã¢ãã«ãçŽæ¥æé©åããŸãããRLHFãã¯ãèšèªã¢ãã«ãäžè¬çãªã³ãŒãã¹ã§åŠç¿ããã¢ãã«ããè€éãªäººéã®äŸ¡å€ã«åãããããšãå¯èœã«ããŸãã
2. RLHFã®3ã€ã®ã¹ããã
ãRLHFãã¯ãè€æ°ã®åŠç¿ããã»ã¹ã§æ§æããããææŠçãªæŠå¿µã§ãããã®èšäºã§ã¯ãåŠç¿ããã»ã¹ã次ã®3ã€ã®ã¹ãããã«åé¡ããŸãã
(1) èšèªã¢ãã«ã®äºååŠç¿
(2) å ±é ¬ã¢ãã«ã®åŠç¿
(3) 匷ååŠç¿ã«ããèšèªã¢ãã«ã®ãã¡ã€ã³ãã¥ãŒãã³ã°
2-1. èšèªã¢ãã«ã®äºååŠç¿
ãRLHFãã¯åºçºç¹ãšããŠãå€å žçãªäºååŠç¿ã¢ãã«ããããŒã¹ã¢ãã«ããšããŠäœ¿çšããŸãã
ã»OpenAI : ããŒã¹ã¢ãã«ãšããŠãGPT-3ãã®å°åçã䜿çšããŠãæåã®äººæ°ããRLHFã¢ãã«ãInstructGPTããäœæã
ã»Anthropic : ããŒã¹ã¢ãã«ãšããŠ1000äžãã520åã®ãã©ã¡ãŒã¿ã®Transformerã¢ãã«ã䜿çšã
ã»DeepMind : ããŒã¹ã¢ãã«ãšããŠ2800åã®ãã©ã¡ãŒã¿ã®ãGopherãã䜿çšã
ãã®ããŒã¹ã¢ãã«ã¯ã远å ã®ããã¹ããæ¡ä»¶ã§ãã¡ã€ã³ãã¥ãŒãã³ã°ããããšãã§ããŸãããå¿ ãããå¿ èŠã¯ãããŸããã
ã»OpenAI : ã奜ãŸããããšãã人éãäœæããããã¹ãã§ãã¡ã€ã³ãã¥ãŒãã³ã°ã
ã»Anthropic : ã圹ã«ç«ã€ãæ£çŽãç¡å®³ããšããåºæºã§ãæèãæãããã«ãªãªãžãã«ã®èšèªã¢ãã«ãèžçã

2-2. å ±é ¬ã¢ãã«ã®åŠç¿
人éã®å¥œã¿ã«åãããŠãã¥ãŒãã³ã°ãããå ±é ¬ã¢ãã«ã ãçæããããšããããRLHFãã®ç ç©¶ãå§ãŸããŸããåºæ¬çãªç®æšã¯ãããã¹ããã人éã®å¥œã¿ãè¡šãæ°å€ (ã¹ã«ã©ãŒå ±é ¬) ãååŸããããšã§ãã
ãå ±é ¬ã¢ãã«ãã¯ãå¥ã®ãã¡ã€ã³ãã¥ãŒãã³ã°ãããèšèªã¢ãã«ããŸãã¯å¥œã¿ããŒã¿ã«åºã¥ããŠãŒãããåŠç¿ããèšèªã¢ãã«ã®äž¡æ¹ãå©çšã§ããŸãã匷ååŠç¿ã®ããã³ããçæãã¢ã®åŠç¿ããŒã¿ã»ããã¯ããããããå®çŸ©ãããããŒã¿ã»ããããããã³ããã®ã»ããããµã³ããªã³ã°ããŠçæãããŸãã
ãã¥ãŒãã³ã»ã¢ãããŒã¿ãŒã¯ãèšèªã¢ãã«ããçæãããããã¹ãåºåãã©ã³ã¯ä»ãããŸããæåã¯ããå ±é ¬ã¢ãã«ããçæããããã«ã人éãã¹ã³ã¢ãããã¹ãã®åéšåã«çŽæ¥é©çšããå¿ èŠããããšèãããããããŸããããå®éã«ã¯ãããè¡ãã®ã¯å°é£ã§ãã人éã®äŸ¡å€èгã®éãã«ããããããã®ã¹ã³ã¢ã¯èª¿æŽãããããã€ãºãå€ããªããŸãã代ããã«ãã©ã³ãã³ã°ã䜿çšããŠè€æ°ã®ã¢ãã«ã®åºåãæ¯èŒããããåªããæ£èŠåããŒã¿ã»ãããäœæããŸãã
ããã¹ãã®ã©ã³ã¯ä»ãã«ã¯è€æ°ã®æ¹æ³ããããŸãã æåããæ¹æ³ã®1ã€ã¯ãåãããã³ããã§æ¡ä»¶ä»ãããã2ã€ã®èšèªã¢ãã«ããçæãããããã¹ãããŠãŒã¶ãŒã«æ¯èŒãããããšã§ããçŽæ¥å¯ŸæŠã§ã¢ãã«ã®åºåãæ¯èŒããããšã«ãããEloã·ã¹ãã ã䜿çšããŠãã¢ãã«ãšåºåã®çžå¯Ÿçãªã©ã³ãã³ã°ãçæã§ããŸãããããã®ããŸããŸãªã©ã³ãã³ã°æ¹æ³ã¯ãåŠç¿çšã®ã¹ã«ã©ãŒå ±é ¬ã·ã°ãã«ã«æ£èŠåãããŸãã

2-3. 匷ååŠç¿ã«ããèšèªã¢ãã«ã®ãã¡ã€ã³ãã¥ãŒãã³ã°
ã匷ååŠç¿ãã«ããèšèªã¢ãã«ã®åŠç¿ã¯ãé·ãéãå·¥åŠçãã¢ã«ãŽãªãºã çãªçç±ããäžå¯èœãšæãããŠããŸãããçŸåšå®çšåãããŠããã®ã¯ããPPOããçšããŠãããŒã¹ã¢ãã«ã®ã³ããŒã®ãã©ã¡ãŒã¿ã®äžéšãŸãã¯å šéšããã¡ã€ã³ãã¥ãŒãã³ã°ã§ãã10Bã100B以äžã®ãã©ã¡ãŒã¿ãæã€ã¢ãã«å šäœã®ãã¡ã€ã³ãã¥ãŒãã³ã°ã¯æ³å€ãªã³ã¹ããããããããèšèªã¢ãã«ã®ãã©ã¡ãŒã¿ã¯åçµããŠåŠç¿ããŸãïŒè©³ããã¯LoRAãSparrow LMãåç §ïŒã
PPOã¯æ¯èŒçæãã䜿ãããŠããæçããŠããããããRLHFãã®åæ£åŠç¿ãšããæ°ããã¢ããªã±ãŒã·ã§ã³ã®ã¹ã±ãŒã«ã¢ããã«æå©ãªéžæã§ããããRLHFããå®çŸããããã®åŒ·ååŠç¿ã®äžæ žçãªé²æ©ã®å€ãã¯ã倧ããªã¢ãã«ãäœ¿ãæ £ããã¢ã«ãŽãªãºã ã§æŽæ°ããæ¹æ³ãèŠã€ãåºãããšã§ããã
ãã®ãã¡ã€ã³ãã¥ãŒãã³ã°ã®ã¿ã¹ã¯ãã匷ååŠç¿ããšããŠå®åŒåããŠã¿ãŸãã
ããªã·ãŒÂ ã¯ãããã³ãããåãåãããã¹ãã®ã·ãŒã±ã³ã¹ïŒãŸãã¯ããã¹ãã«å¯Ÿããåãªã確çååžïŒãè¿ãèšèªã¢ãã«ã§ãããã®ããªã·ãŒã®Â è¡å空é ã¯èšèªã¢ãã«ã®èªåœã«å¯Ÿå¿ãããã¹ãŠã®ããŒã¯ã³ (50kããŒã¯ã³ã»ã©)ã芳å¯ç©ºé ã¯å¯èœãªå ¥åããŒã¯ã³åã®ååžã§ããå ±é ¬é¢æ° ã¯ãããªãã¡ã¬ã³ã¹ã¢ãã«ãšããªã·ãŒã·ããã®å¶çŽãçµã¿åããããã®ã«ãªããŸãã
å ±é ¬é¢æ° ã¯ããããŸã§èª¬æãããã¹ãŠã®ã¢ãã«ã1ã€ã®ãRLHFãã«çµ±åãããã®ã§ããããŒã¿ã»ããããããã³ãã x ãäžãããããšã2ã€ã®ããã¹ã y1, y2 ãçæãããŸããçŸåšã®ããªã·ãŒã®ããã¹ãã¯ããªãã¡ã¬ã³ã¹ã¢ãã«ã«æž¡ãããããªãã¡ã¬ã³ã¹ã¢ãã«ã¯ãpreferabilityããšããã¹ã«ã©ãŒæŠå¿µãè¿ããŸãããã®ããã¹ãã¯ãããŒã¹ã¢ãã«ããã®ããã¹ããšæ¯èŒãããäž¡è ã®å·®ã«å¯Ÿããããã«ãã£ãèšç®ããŸããOpenAIãAnthropicãDeepMindãªã©ã®è«æã§ã¯ããã®ããã«ãã£ã¯ãããŒã¯ã³äžã®ãããã®ã·ãŒã±ã³ã¹éã®ãKL divergenceãã®ã¹ã±ãŒãªã³ã°ããŒãžã§ã³ãšããŠèšèšãããŠããŸãã
 KL divergence ã¯ãååŠç¿ãããã§åŒ·ååŠç¿ãããªã·ãŒãããŒã¹ã¢ãã«ãã倧ããé¢ããããšã眰ãããã®ã§ãã¢ãã«ãé©åºŠã«ãŸãšãŸã£ãããã¹ããåºåããŠããããšã確èªããã®ã«æçšã§ãããã®ããã«ãã£ããªããšãæé©åã«ãã£ãŠå ±é ¬ã¢ãã«ãéšããŠãæå³äžæãªããã¹ããçæãããããã«ãªããŸãã
RLHFã·ã¹ãã ã®äžã«ã¯ãå ±é ¬é¢æ°ã«æ©èœè¿œå ãããã®ããããŸããäŸãã°ãOpenAIã¯InstructGPTã«ãããŠãPPOã®æŽæ°ã«ãŒã«ã«äºååŠç¿åŸé ã远å ããŠå®éšããæåãåããŸããããRLHFãã®ç ç©¶ãé²ãã«ã€ããŠããã®å ±é ¬é¢æ°ã®å®åŒåãé²åããŠãããšæãããŸãã
æåŸã«æŽæ°ã«ãŒã«ã¯ãçŸåšã®ããŒã¿ãããã®å ±é ¬ã¡ããªãã¯ãæå€§åããPPOããã®ãã©ã¡ãŒã¿æŽæ°ã«ãªããŸããPPOã¯ä¿¡é Œé åæé©åã¢ã«ãŽãªãºã ã§ãåŸé ã«å¯Ÿããå¶çŽã䜿çšããŠãæŽæ°ã¹ããããåŠç¿ããã»ã¹ãäžå®å®ã«ããªãããã«ããŸãã

3. RLHFã®ããã®ãªãŒãã³ãœãŒã¹ããŒã«
çŸåšãPyTorchã«ã¯ãRLHFãçšã®ããã€ãã®ã¢ã¯ãã£ããªãªããžããªãååšããŸãã
äž»ãªãªããžããªã¯ã次ã®3ã€ã§ãã
ã»TRL (Transformers Reinforcement Learning)
ã»TRLX (Transformers Reinforcement Learning X)Â
ã»RL4LMs (Reinforcement Learning for Language models)
ãTRLãã¯ãHuggingFaceã§PPO ã䜿çšããŠäºååŠç¿æžã¿ã®èšèªã¢ãã«ããã¡ã€ã³ãã¥ãŒãã³ã°ããããã«èšèšãããŠããŸãã
ãTRLXãã¯ããCarperAIãã«ãã£ãŠæ§ç¯ããããTRLãã®æ¡åŒµãã©ãŒã¯ã§ããªã³ã©ã€ã³ããã³ãªãã©ã€ã³ã®åŠç¿çšã®å€§èŠæš¡ãªã¢ãã«ãåŠçããŸããçŸæç¹ã§ã¯ããTRLXãã«ã¯ãLLMã®å±éã«å¿ èŠãªèŠæš¡ (ããšãã°ã33Bãã©ã¡ãŒã¿) ã§ãPPOãšILQLãåããããRLHFããå¯èœãªAPIããããŸããå°æ¥çã«ã¯ãæå€§200Bãã©ã¡ãŒã¿ã®èšèªã¢ãã«ãå¯èœã«ãªããŸãããã®ããããTRLXããšã®ã€ã³ã¿ãŒãã§ãŒã¹ã¯ããã®èŠæš¡ã§ã®çµéšãæã€æ©æ¢°åŠç¿ãšã³ãžãã¢åãã«æé©åãããŠããŸãã
ãRL4LMsãã¯ãããŸããŸãªåŒ·ååŠç¿ã¢ã«ãŽãªãºã (PPOãNLPOãA2Cãããã³ TRPO)ãå ±é ¬é¢æ°ãã¡ããªãã¯ã䜿çšããŠãLLMããã¡ã€ã³ãã¥ãŒãã³ã°ããã³è©äŸ¡ããããã®ã¢ãžã¥ãŒã«ãæäŸããŸãã çŸåšã®èšç»ã«ã¯ããã倧ããªã¢ãã«ã®åæ£åŠç¿ãšæ°ãã匷ååŠç¿ã¢ã«ãŽãªãºã ãå«ãŸããŠããŸãã
ãŸãããAnthropicãã«ãã£ãŠäœæãããå€§èŠæš¡ãªããŒã¿ã»ãããHuggingFaceã§æäŸãããŠããŸãã