
DPO ã«ãã Llama 2 ã®ãã¡ã€ã³ãã¥ãŒãã³ã°
以äžã®èšäºãé¢çœãã£ãã®ã§ãããããŸãšããŸããã
1. ã¯ããã«
ãRLHFãã¯ãGPT-4ããClaudeããªã©ã®LLMã®äºå®äžã®æåŸã®åŠç¿ã¹ããããšãªã£ãŠãããLLMåºåã®é¥èããå®å šãã人éã®æåŸ ãšäžèŽããŠããããšã確èªããŸãããã ããRLã®è€éããæã¡èŸŒãŸããŸããé©åãªå ±é ¬é¢æ°ãèšå®ããç¶æ ãæšå®ããããã«ã¢ãã«ãåŠç¿ããå¿ èŠããããŸããåæã«ãå ã®ã¢ãã«ããé¢ããããªãããæ³šæããå¿ èŠããããŸãããã®ãããªããã»ã¹ã¯éåžžã«è€éã§ãæ£ããè¡ãã®ã¯å®¹æã§ã¯ãããŸããã
RafailovãSharmaãMitchellãã«ããæè¿ã®è«æãDirect Preference Optimizationãã§ã¯ãæ¢åã®ææ³ã§äœ¿çšãããŠããRLããŒã¹ã®ç®æšããåçŽãªãã€ããªã¯ãã¹ãšã³ããããŒæå€±ãä»ããŠçŽæ¥æé©åã§ããç®æšã«åãæ¿ããããšãææ¡ããŠããŸããããã«ãããLLMãæ¹è¯ãããã®ããã»ã¹ãå€§å¹ ã«ç°¡çŽ åã§ããŸãã
ãã®èšäºã§ã¯ãTRL ã§å©çšã§ããããã«ãªã£ããDPOã(Direct Preference Optimization) ã䜿ã£ãŠãstack-exchange preference ããŒã¿ã»ããã§ ãLlama v2 7Bãããã¡ã€ã³ãã¥ãŒãã³ã°ããæé ã玹ä»ããŸãã
2. DPO vs PPO
RLã«ãã£ãŠäººéã®å¥œã¿ãæé©åããäŒçµ±çãªã¢ãã«ã§ã¯ãè£å©çãªå ±é ¬ã¢ãã«ã䜿çšããRLã«ãã£ãŠäžããããå ±é ¬ãæå€§åããããã«ã察象ã®ã¢ãã«ããã¡ã€ã³ãã¥ãŒãã³ã°ããã®ãäžè¬çã§ããçŽæçã«ã¯ãå ±é ¬ã¢ãã«ã䜿çšããŠãæé©åããŠããã¢ãã«ã«ãã£ãŒãããã¯ãäžããé«å ±é ¬ã®ãµã³ãã«ãããé »ç¹ã«çæããäœå ±é ¬ã®ãµã³ãã«ãããé »ç¹ã«çæããªãããã«ããŸããåæã«ãçæããããã®ã倧ããéžè±ããªãããã«ããŸãçæã®å€æ§æ§ãç¶æãç¶ããããã«ãåºå®ãããåç §ã¢ãã«ã䜿çšããŸããããã¯éåžžãåç §ã¢ãã«ãä»ããŠå®å šãªå ±é ¬æå€§åç®æšã«KLããã«ãã£ã远å ããããšã§è¡ãããã¢ãã«ãå ±é ¬ã¢ãã«ãããŸããããšãåŠç¿ããªãããã«ããŸãã
DPOã®å®åŒåã¯ãå ±é ¬ã¢ããªã³ã°ã®ã¹ãããããã€ãã¹ããéèŠãªæŽå¯ãä»ããŠå奜ããŒã¿ã®èšèªã¢ãã«ãçŽæ¥æé©åããŸããã€ãŸããå ±é ¬é¢æ°ããæé©ãªRLããªã·ãŒãžã®åæãããã³ã°ã«ãããäœæè ã¯å ±é ¬ã¢ãã«ãšåç §ã¢ãã«ã«å¯Ÿãã RL æå€±ã倿ã§ããŸããåç §ã¢ãã«ãçŽæ¥äžåãæå€±ãçºçããŸãããã®ãããã³ã°ã¯ãç¹å®ã®å ±é ¬é¢æ°ãç¹å®ã®å奜ããŒã¿ãšã©ã®çšåºŠäžèŽããŠããããçŽæçã«æž¬å®ããŸãããããã£ãŠãDPOã¯RLHFæå€±ã®æé©è§£ããéå§ãã倿°ã®å€æŽãéããŠåç §ã¢ãã«ã®ã¿ã®æå€±ãå°ãåºããŸãã
ãã®ç®æšã¯ãå ±é ¬ã¢ãã«ãå¿ èŠãšããããŸãã¯æœåšçã«é¢åãªRLããŒã¹ã®æé©åãå®è¡ããå¿ èŠããªããæé©åã§ããŸãã
3. TRLã䜿ã£ãåŠç¿æ¹æ³
éåžžãRLHF ãã€ãã©ã€ã³ã¯æ¬¡ã®ç°ãªãããŒãã§æ§æãããŸãã
(1) æåž«ãããã¡ã€ã³ãã¥ãŒãã³ã° (SFT)
(2) ããŒã¿ãžã®èšå®ã©ãã«ã®ä»å
(3) å奜ããŒã¿ã«åºã¥ããŠå ±é ¬ã¢ãã«ãåŠç¿
(4) RLæé©å
TRLã«ã¯ããããã¹ãŠã®ããŒãã®ãã«ããŒãä»å±ããŠããŸãããDPOã§ã¯å ±é ¬ã¢ããªã³ã°ãš RL ((3)(4)) ãäžèŠã«ãªããããªãã¡ã¬ã³ã¹ã®æ³šéä»ãããŒã¿ã«åºã¥ã㊠DPOãªããžã§ã¯ããçŽæ¥æé©åãããŸãã
(1) ãå®è¡ããå¿ èŠããããŸããã(3)(4) ã®ä»£ããã«ãéåžžã«ç¹æ®ãªåœ¢åŒãã€ãŸã次ã®3ã€ã®ããŒãå«ãèŸæžãæã€(2)ã®èšå®ããŒã¿ãTRLã®ãDPOTrainerãã«æž¡ããå¿ èŠããããŸãã
ã»prompt : ããã¹ãçæã®æšè«æã«ã¢ãã«ã«äžããããããã³ãã
ã»chosen : ããã³ããã«å¯Ÿãã奜ãŸããå¿ç
ã»rejected : ããã³ããã«å¯Ÿãã奜ãŸãããªãå¿ç
stack-exchange preference pairs ããŒã¿ã»ããã®å Žåãæ¬¡ã®ãã«ããŒãä»ããŠç®çã®èŸæžãè¿ããå ã®åããã¹ãŠåé€ããããã«ããŒã¿ã»ãããšã³ããªããããã§ããŸãã
def return_prompt_and_responses(samples) -> Dict[str, str, str]:
return {
"prompt": [
"Question: " + question + "\n\nAnswer: "
for question in samples["question"]
],
"chosen": samples["response_j"], # rated better than k
"rejected": samples["response_k"], # rated worse than j
}
dataset = load_dataset(
"lvwerra/stack-exchange-paired",
split="train",
data_dir="data/rl"
)
original_columns = dataset.column_names
dataset.map(
return_prompt_and_responses,
batched=True,
remove_columns=original_columns
)ããŒã¿ã»ãããäžŠã¹æ¿ãããšãDPOæå€±ã¯åºæ¬çã«ãåç §ã¢ãã«ãä»ããŠæé»çãªå ±é ¬ãååŸããæåž«ããæå€±ãšãªããããé«ã¬ãã«ã§ã¯ããDPOTrainerãã«ã¯æé©åããããããŒã¹ã¢ãã«ããšãåç §ã¢ãã«ããå¿ èŠã§ãã
dpo_trainer = DPOTrainer(
model, # ããŒã¹ã¢ãã«
model_ref, # åç
§ã¢ãã« (ããŒã¹ã¢ãã«ã®ã³ããŒ)
beta=0.1, # DPO枩床
train_dataset=dataset, # ããŒã¿ã»ãã
tokenizer=tokenizer, # ããŒã¯ãã€ã¶ãŒ
args=training_args, # åŠç¿ãã©ã¡ãŒã¿ (ããããµã€ãºãåŠç¿çãªã©)
)ããã§ããbetaãã¯DPOæå€±ã®æž©åºŠã§ãããé垞㯠0.1 ïœ 0.5 ã®ç¯å²ã«ãããŸãã ããã¯ãbetaãå°ãããªãã»ã©åç §ã¢ãã«ãç¡èŠãããšããæå³ã§ãåç §ã¢ãã«ã«ã©ã®çšåºŠæ³šæãæãããæå®ããŸãããã¬ãŒããŒãåæåããããæ¬¡ã®ã¡ãœãããåŒã³åºãã ãã§ãåŠç¿éå§ã§ããŸãã
dpo_trainer.train()4. Llama 2 ã®ãã¡ã€ã³ãã¥ãŒãã³ã°
TRLã«DPO ãã¬ãŒããŒãå®è£ ããå©ç¹ã¯ãTRL ãšãã®äŸåã©ã€ãã©ãª (Peft ã Accelerate ãªã©) ã«ä»å±ããå€§èŠæš¡ãªLLMã®åŠç¿ã«é¢ããè¿œå æ©èœããã¹ãŠå©çšã§ããããšã§ãããããã®ã©ã€ãã©ãªã䜿çšãããšãbitsandbytesã«ãã£ãŠæäŸãããQLoRAã䜿çšããŠLlama v2ãåŠç¿ããããšãã§ããŸãã
4-1. æåž«ãããã¡ã€ã³ãã¥ãŒãã³ã°
ãLlama 2 7Bãã§ QLoRA ã䜿çšããæåž«ãããã¡ã€ã³ãã¥ãŒãã³ã°ãè¡ããŸãã
# 4bitéååã§ããŒã¹ã¢ãã«ãããŒã
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
base_model = AutoModelForCausalLM.from_pretrained(
script_args.model_name, # "meta-llama/Llama-2-7b-hf"
quantization_config=bnb_config,
device_map={"": 0},
trust_remote_code=True,
use_auth_token=True,
)
base_model.config.use_cache = False
# éååãããããŒã¹ã¢ãã«ã« LoRAã¬ã€ã€ãŒã远å
peft_config = LoraConfig(
r=script_args.lora_r,
lora_alpha=script_args.lora_alpha,
lora_dropout=script_args.lora_dropout,
target_modules=["q_proj", "v_proj"],
bias="none",
task_type="CAUSAL_LM",
)
...
trainer = SFTTrainer(
model=base_model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
peft_config=peft_config,
packing=True,
max_seq_length=None,
tokenizer=tokenizer,
args=training_args, # HFãã¬ãŒããŒåŒæ°
)
trainer.train()4-2. DPO ãã¬ãŒãã³ã°
SFTãå®äºããããçµæã®ã¢ãã«ãä¿åããDPOãã¬ãŒãã³ã°ã«é²ãããšãã§ããŸããéåžžè¡ãããŠããããã«ãåã®SFT ã¹ãããã§ä¿åããã¢ãã«ããDPOã®ããŒã¹ã¢ãã«ãšåç §ã¢ãã«ã®äž¡æ¹ã«å©çšããŸããæ¬¡ã«ããããã䜿çšããŠãäžã«ç€ºããããŒã¿ã® DPOç®æšã§ã¢ãã«ãåŠç¿ã§ããŸãã ã¢ãã«ã¯ LoRaã¢ããã¿ãŒçµç±ã§åŠç¿ããããããPeftã® AutoPeftModelForCausalLM çµç±ã§ããŒãããŸãã
model = AutoPeftModelForCausalLM.from_pretrained(
script_args.model_name_or_path, # ä¿åããã SFT ã¢ãã«ã®å Žæ
low_cpu_mem_usage=True,
torch_dtype=torch.float16,
load_in_4bit=True,
is_trainable=True,
)
model_ref = AutoPeftModelForCausalLM.from_pretrained(
script_args.model_name_or_path, # ã¡ã€ã³ãšåãã¢ãã«
low_cpu_mem_usage=True,
torch_dtype=torch.float16,
load_in_4bit=True,
)
...
dpo_trainer = DPOTrainer(
model,
model_ref,
args=training_args,
beta=script_args.beta,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
tokenizer=tokenizer,
peft_config=peft_config,
)
dpo_trainer.train()
dpo_trainer.save_model()ã¢ãã«ã4bitã§ããŒãããpeft_configåŒæ°ãä»ããŠQLoraã§åŠç¿ããŸãããã¬ãŒããŒã¯ãè©äŸ¡ããŒã¿ã»ããã«é¢ããŠãåŠç¿äžã®é²æç¶æ³ãè©äŸ¡ããããšãã° wandbãä»ããŠèšé²ããã³è¡šç€ºã§ããæé»çå ±é ¬ãªã©ã®å€ãã®éèŠãªææšãã¬ããŒãããŸãããã®åŸãæçµçã«åŠç¿ãããã¢ãã«ã HuggingFace Hub ã«ããã·ã¥ã§ããŸãã
5. ãããã«
SFTããã³DPOã®åŠç¿ã¹ã¯ãªããã®å®å šãªãœãŒã¹ã³ãŒãã¯ãexamples/stack_llama_2 ã«ãããããŒãžãããã¢ããã¿ãŒã䜿çšããŠåŠç¿ãããã¢ãã«ã¯ããã¡ãã«ãããŸãã
DPOãã¬ãŒãã³ã°å®è¡ã® wandbãã°ã¯ããã§åç §ã§ããŸããåŠç¿ãšè©äŸ¡äžã«ãDPOTrainerã¯æ¬¡ã®å ±é ¬ã¡ããªãã¯ãèšé²ããŸãã
ã»rewards/chosen : betaã§ã¹ã±ãŒã«ããããéžæãããå¿çã®ããªã·ãŒã¢ãã«ãšåç §ã¢ãã«ã®å¯Ÿæ°ç¢ºçéã®å¹³åå·®
ã»rewards/rejected : ããªã·ãŒ ã¢ãã«ãšæåŠãããå¿çã®åç §ã¢ãã«ã®å¯Ÿæ°ç¢ºçéã®å¹³åå·® (betaã§ã¹ã±ãŒã«)
ã»rewards/accuracies : éžæãããå ±é ¬ã察å¿ããæåŠãããå ±é ¬ãããé«ãé »åºŠã®å¹³å
ã»rewards/margins : éžæãããå ±é ¬ãšå¯Ÿå¿ããæåŠãããå ±é ¬ã®éã®å¹³åå·®
çŽæçã«ã¯ãåŠç¿äžã«ããŒãžã³ãå¢å ãã粟床ã 1.0 ã«ãªãããšãã€ãŸããéžæãããå ±é ¬ãæåŠãããå ±é ¬ããã倧ãããªãããã«ããããšèããŸã (ãŸãã¯ããŒãžã³ããŒããã倧ããå Žå)ã ãããã®ã¡ããªã¯ã¹ã¯ãäœããã®è©äŸ¡ããŒã¿ã»ããã«å¯ŸããŠèšç®ã§ããŸãã