
TRL - 匷ååŠç¿ã«ããLLMã®åŠç¿ã®ããã®ã©ã€ãã©ãª
ãTRLãã®ããã¥ã¡ã³ããç°¡åã«ãŸãšããŸããã
ã»trl v0.4.2
1. TRL
ãTRLã(Transformer Reinforcement Learning) ã¯ã匷ååŠç¿ã䜿çšããŠTransformerèšèªã¢ãã«ãåŠç¿ã§ããŸãããã®ã©ã€ãã©ãªã¯HuggingFace Transformersãšçµ±åãããŠããŸãã
ãTRLãã¯ããGPT-2ããBLOOMããGPT-Neoããªã©ã®ãã³ãŒãã¢ãã«ããµããŒãããŠãããããããã¹ãŠãPPOãã䜿çšããŠæé©åã§ããŸãã
2. ã¯ã€ãã¯ã¹ã¿ãŒã
2-1. åŠçã®æµã
ãPPOãã«ããèšèªã¢ãã«ã®ãã¡ã€ã³ãã¥ãŒãã³ã°ã¯ã3ã€ã®ã¹ãããã§æ§æãããŸãã
ã»ããŒã«ã¢ãŠã : èšèªã¢ãã«ã¯ãæã®å§ãŸããšãªãã¯ãšãªãããšã«ãå¿çãŸãã¯ç¶ç¶ãçæããŸãã
ã»è©äŸ¡ : ã¯ãšãªãšå¿çã¯ã颿°ãã¢ãã«ã人éã«ãããã£ãŒãããã¯ããŸãã¯ãããã®çµã¿åããã«ãã£ãŠè©äŸ¡ãããŸããéèŠãªããšã¯ããã®ããã»ã¹ã§ã¯åã¯ãšãª/å¿ç ãã¢ã®ã¹ã«ã©ãŒå€ãçæãããããšã§ããæé©åã¯ããã®å€ãæå€§åããããšãç®çãšããŸãã
ã»æé©å : æãè€éãªéšåã§ããæé©åã¹ãããã§ã¯ãã¯ãšãª/å¿ç ãã¢ã䜿çšããŠãã·ãŒã±ã³ã¹å ã®ããŒã¯ã³ã® log-probabilities ãèšç®ããŸããããã¯ãåŠç¿ã¢ãã«ãšåç §ã¢ãã« (éåžžã¯ãã¡ã€ã³ãã¥ãŒãã³ã°åã®äºååŠç¿ã¢ãã«) ã䜿çšããŠè¡ãããŸãã2ã€ã®åºåéã® KL-divergence ã¯ãçæãããå¿çãåç §èšèªã¢ãã«ãã倧ããéžè±ããªãããã«ããããã®è¿œå ã®å ±é ¬ä¿¡å·ãšããŠäœ¿çšãããŸãã æ¬¡ã«ãã¢ã¯ãã£ããªèšèªã¢ãã«ã PPO ã§åŠç¿ãããŸãã
å®å šãªåŠçã®æµãã¯ã次å³ã®ãšããã§ãã

2-2. æå°éã®ã³ãŒã
äžèšæé ã瀺ãæå°éã®ã³ãŒãäŸã¯ã次ã®ãšããã§ãã
# 0. ã€ã³ããŒã
import torch
from transformers import GPT2Tokenizer
from trl import AutoModelForCausalLMWithValueHead, PPOConfig, PPOTrainer
# 1. äºååŠç¿ã¢ãã«ã®èªã¿èŸŒã¿
model = AutoModelForCausalLMWithValueHead.from_pretrained("gpt2")
model_ref = AutoModelForCausalLMWithValueHead.from_pretrained("gpt2")
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token
# 2. ãã¬ãŒããŒã®åæå
ppo_config = {"batch_size": 1}
config = PPOConfig(**ppo_config)
ppo_trainer = PPOTrainer(config, model, model_ref, tokenizer)
# 3. ã¯ãšãªã®ãšã³ã³ãŒã
query_txt = "This morning I went to the "
query_tensor = tokenizer.encode(query_txt, return_tensors="pt").to(model.pretrained_model.device)
# 4. ã¢ãã«å¿çã®çæ
generation_kwargs = {
"min_length": -1,
"top_k": 0.0,
"top_p": 1.0,
"do_sample": True,
"pad_token_id": tokenizer.eos_token_id,
"max_new_tokens": 20,
}
response_tensor = ppo_trainer.generate([item for item in query_tensor], return_prompt=False, **generation_kwargs)
response_txt = tokenizer.decode(response_tensor[0])
# 5. å¿çã«å¯Ÿããå ±é
¬ã®å®çŸ©
reward = [torch.tensor(1.0, device=model.pretrained_model.device)]
# 6. PPOã§ã¢ãã«ãåŠç¿
train_stats = ppo_trainer.step([query_tensor[0]], [response_tensor[0]], reward)äžè¬ã«ãã¹ããã 3 ïœ 6 ã for ã«ãŒãã§å®è¡ããŸãã
2-3. åŠç¿æžã¿ã¢ãã«ã®äœ¿ç𿹿³
ãAutoModelForCausalLMWithValueHeadããåŠç¿åŸãtransformersã§ã¢ãã«ãçŽæ¥äœ¿çšã§ããŸãã
# .. PPOTrainer ãš AutoModelForCausalLMWithValueHead ã䜿çšããŠåŠç¿ããã¢ãã«ããããšä»®å®
# ã¢ãã«ãHubã«Push
model.push_to_hub("my-fine-tuned-model-ppo")
# ãŸãã¯ããŒã«ã«ã«ä¿å
model.save_pretrained("my-fine-tuned-model-ppo")
# Hubããã¢ãã«ãèªã¿èŸŒã
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("my-fine-tuned-model-ppo")åŠç¿ãç¶ç¶ããå Žåãªã©ãvalue headã䜿çšãããå Žåã¯ãAutoModelForCausalLMWithValueHeadã䜿çšããŠã¢ãã«ãèªã¿èŸŒãããšãã§ããŸãã
from trl.model import AutoModelForCausalLMWithValueHead
model = AutoModelForCausalLMWithValueHead.from_pretrained("my-fine-tuned-model-ppo")3. ã€ã³ã¹ããŒã«
ãTRLã㯠pipãŸãã¯ãœãŒã¹ããã€ã³ã¹ããŒã«ã§ããŸãã
â pip
pip install trlâ ãœãŒã¹
git clone https://github.com/lvwerra/trl.git
cd trl/
pip install -e .4. åŠç¿ã®ã«ã¹ã¿ãã€ãº
ãTRLãã§ã¯ããŠãŒã¶ãŒãããŒãºã«åãããŠåŠç¿ã«ãŒããå¹ççã«ã«ã¹ã¿ãã€ãºã§ãããããååãªã¢ãžã¥ãŒã«æ§ãæäŸããŸãã 以äžã«ãããŸããŸãªãã¯ããã¯ãé©çšããŠããã¹ãããæ¹æ³ã®äŸã玹ä»ããŸãã
4.1 è€æ°ã®GPU/ããŒãã§ã®å®è¡
accelerateã³ãå©çšããŠããŠãŒã¶ãŒãè€æ°ã® GPU ãŸãã¯ããŒãã§åŠç¿ã§ããããã«ããŸããæ¬¡ã®ã³ãã³ãã§Â accelerate config ãäœæããå¿ èŠããããŸãã
accelerate config次ã«ããã«ãGPU / ãã«ããŒããéžæãããŠããããšã確èªããŠãã ããã ãã®åŸã以äžãå®è¡ããã ãã§åŠç¿ã§ããŸãã
accelerate launch your_script.py詳现ã«ã€ããŠã¯ãexamples page ãåç §ã
4-2. ããŸããŸãªãªããã£ãã€ã¶ãŒã®äœ¿çš
ããã©ã«ãã§ã¯ããPPOTrainerãã¯ãtorch.optim.Adamããªããã£ãã€ã¶ãäœæããŸãã å¥ã®ãªããã£ãã€ã¶ãäœæããŠå®çŸ©ããããããPPOTrainerãã«æå®ããããšãã§ããŸãã
import torch
from transformers import GPT2Tokenizer
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
# 1. äºååŠç¿ã¢ãã«ã®èªã¿èŸŒã¿
model = AutoModelForCausalLMWithValueHead.from_pretrained('gpt2')
model_ref = AutoModelForCausalLMWithValueHead.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
# 2. ã³ã³ãã£ã°ã®å®çŸ©
ppo_config = {'batch_size': 1, 'learning_rate':1e-5}
config = PPOConfig(**ppo_config)
# 2. ãªããã£ãã€ã¶ã®äœæ
optimizer = torch.optim.SGD(model.parameters(), lr=config.learning_rate)
# 3. ãã¬ãŒããŒã®åæå
ppo_trainer = PPOTrainer(config, model, model_ref, tokenizer, optimizer=optimizer)ã¡ã¢ãªå¹çããã¡ã€ã³ãã¥ãŒãã³ã°ããããã«ãbitsandbytes ãã Adam8bit ãªããã£ãã€ã¶ãæå®ããããšãã§ããŸãã
import torch
import bitsandbytes as bnb
from transformers import GPT2Tokenizer
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
# 1. äºååŠç¿ã¢ãã«ã®èªã¿èŸŒã¿
model = AutoModelForCausalLMWithValueHead.from_pretrained('gpt2')
model_ref = AutoModelForCausalLMWithValueHead.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
# 2. ã³ã³ãã£ã°ã®å®çŸ©
ppo_config = {'batch_size': 1, 'learning_rate':1e-5}
config = PPOConfig(**ppo_config)
# 2. ãªããã£ãã€ã¶ã®äœæ
optimizer = bnb.optim.Adam8bit(model.parameters(), lr=config.learning_rate)
# 3. ãã¬ãŒããŒã®åæå
ppo_trainer = PPOTrainer(config, model, model_ref, tokenizer, optimizer=optimizer)4-3. LION ãªããã£ãã€ã¶ã®äœ¿çš
Google ã®æ°ãã LION ãªããã£ãã€ã¶ã䜿çšã§ããŸãããªããã£ãã€ã¶å®çŸ©ã®ãœãŒã¹ ã³ãŒããååŸãããªããã£ãã€ã¶ãã€ã³ããŒãã§ããããã«ã³ããŒããŸããããã¡ã¢ãªå¹çã®é«ãåŠç¿ãè¡ãããã«ã®ã¿ãåŠç¿å¯èœãªãã©ã¡ãŒã¿ãŒãèæ ®ããŠãªããã£ãã€ã¶ãåæåããŠãã ããã
optimizer = Lion(filter(lambda p: p.requires_grad, self.model.parameters()), lr=self.config.learning_rate)
...
ppo_trainer = PPOTrainer(config, model, model_ref, tokenizer, optimizer=optimizer)ããã§ææãããŠããããã«ãAdam ã«äœ¿çšããåŠç¿çã 3 ã§å²ã£ãå€ã䜿çšããããšããå§ãããŸãããã®ãªããã£ãã€ã¶ã䜿çšãããšãåŸæ¥ã® Adam ãšæ¯èŒããŠæ¹åãèŠãããŸããã

4-4. åŠç¿çã¹ã±ãžã¥ãŒã©ã®è¿œå
åŠç¿çã¹ã±ãžã¥ãŒã©ã远å ããŠãåŠç¿ã詊ãããšãã§ããŸãã
import torch
from transformers import GPT2Tokenizer
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
# 1. äºååŠç¿ã¢ãã«ã®èªã¿èŸŒã¿
model = AutoModelForCausalLMWithValueHead.from_pretrained('gpt2')
model_ref = AutoModelForCausalLMWithValueHead.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
# 2. ã³ã³ãã£ã°ã®å®çŸ©
ppo_config = {'batch_size': 1, 'learning_rate':1e-5}
config = PPOConfig(**ppo_config)
# 2. ãªããã£ãã€ã¶ã®äœæ
optimizer = torch.optim.SGD(model.parameters(), lr=config.learning_rate)
lr_scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.9)
# 3. ãã¬ãŒããŒã®åæå
ppo_trainer = PPOTrainer(config, model, model_ref, tokenizer, optimizer=optimizer, lr_scheduler=lr_scheduler)4-5. ã¬ã€ã€ãŒã®å ±æã«ããã¡ã¢ãªå¹çã®é«ããã¡ã€ã³ãã¥ãŒãã³ã°
ããã¡ã¢ãªå¹çã®é«ããã¡ã€ã³ãã¥ãŒãã³ã°ã«äœ¿çšã§ãããã 1 ã€ã®ããŒã«ã¯ãåç §ã¢ãã«ãšåŠç¿ã¢ãã«ã®éã§ã¬ã€ã€ãŒãå ±æããããšã§ãã
import torch
from transformers import AutoTokenizer
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead, create_reference_model
# 1. äºååŠç¿ã¢ãã«ã®èªã¿èŸŒã¿
model = AutoModelForCausalLMWithValueHead.from_pretrained('bigscience/bloom-560m')
model_ref = create_reference_model(model, num_shared_layers=6)
tokenizer = AutoTokenizer.from_pretrained('bigscience/bloom-560m')
# 2. ãã¬ãŒããŒã®åæå
ppo_config = {'batch_size': 1}
config = PPOConfig(**ppo_config)
ppo_trainer = PPOTrainer(config, model, model_ref, tokenizer)4-6. 8-bitåç §ã¢ãã«ã®æå®
ãTRLãã¯ãfrom_pretrained ã䜿çšããŠtransformersããã¢ãã«ãèªã¿èŸŒãæããã¹ãŠã®ãã©ã¡ãŒã¿ããµããŒããããããload_in_8bit ãæå®ããŠãããã¡ã¢ãªå¹çã®é«ããã¡ã€ã³ãã¥ãŒãã³ã°ãè¡ãããšãã§ããŸãã
8-bitã¢ãã«ã®èªã¿èŸŒã¿ã«ã€ããŠè©³ããã¯ããã¡ããåç §ã
# 0. ã€ã³ããŒã
# pip install bitsandbytes
import torch
from transformers import AutoTokenizer
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
# 1. äºååŠç¿ã¢ãã«ã®èªã¿èŸŒã¿
model = AutoModelForCausalLMWithValueHead.from_pretrained('bigscience/bloom-560m')
model_ref = AutoModelForCausalLMWithValueHead.from_pretrained('bigscience/bloom-560m', device_map="auto", load_in_8bit=True)
tokenizer = AutoTokenizer.from_pretrained('bigscience/bloom-560m')
# 2. ãã¬ãŒããŒã®åæå
ppo_config = {'batch_size': 1}
config = PPOConfig(**ppo_config)
ppo_trainer = PPOTrainer(config, model, model_ref, tokenizer)4-7. CUDA ãã£ãã·ã¥ ãªããã£ãã€ã¶ã®äœ¿çš
å€§èŠæš¡ãªã¢ãã«ãåŠç¿ããå Žåã¯ãCUDAãã£ãã·ã¥ãç¹°ãè¿ãã¯ãªã¢ããŠãããé©åã«åŠçããå¿ èŠããããŸããPPOConfigã§optimize_cuda_cache=True ãæå®ããŸãã
config = PPOConfig(..., optimize_cuda_cache=True)4-8. DeepSpeed stage 3 ãæ£ãã䜿çš
DeepSpeed stage 3 ãæ£ãã䜿çšããã«ã¯ãåŠç¿ã¹ã¯ãªããã«å°ããªèª¿æŽã远å ããå¿ èŠããããŸãã zero3_init_context_manager ã³ã³ããã¹ã ãããŒãžã£ã䜿çšããŠãæ£ããããã€ã¹ã§å ±é ¬ã¢ãã«ãé©åã«åæåããå¿ èŠããããŸãã gpt2-sentiment ã¹ã¯ãªããã«é©å¿ããäŸããæ¬¡ã«ç€ºããŸãã
ds_plugin = ppo_trainer.accelerator.state.deepspeed_plugin
if ds_plugin is not None and ds_plugin.is_zero3_init_enabled():
with ds_plugin.zero3_init_context_manager(enable=False):
sentiment_pipe = pipeline("sentiment-analysis", model="lvwerra/distilbert-imdb", device=device)
else:
sentiment_pipe = pipeline("sentiment-analysis", model="lvwerra/distilbert-imdb", device=device)4-9. torch distributed ã®äœ¿çš
torch.distributed ããã±ãŒãžã¯ããããã¯ãŒã¯ãè€æ°ã®ãã·ã³ã«åæ£ããããã® PyTorch ãã€ãã£ãã¡ãœãããæäŸããŸã (è€æ°ã®GPUããŒããããå Žåã«æã圹ç«ã¡ãŸã)ã åGPUã§ã¢ãã«ãã³ããŒããããããã§é æ¹åãšéæ¹åãå®è¡ãããã¹ãŠã®GPUã®åŸé ã®å¹³åãããããã«é©çšããŸãã
torch 1.XXã®å Žåãæ¬¡ã®ããã« torch.distributed.launch ãåŒã³åºãããšãã§ããŸãã
python -m torch.distributed.launch --nproc_per_node=1 reward_summarization.py --bf16torch 2.+ ã§ã¯ torch.distributed.launch ã¯éæšå¥šãšãªããæ¬¡ã®åŒã³åºããå¿ èŠã§ãã
torchrun --nproc_per_node=1 reward_summarization.py --bf16
# ãŸãã¯
python -m torch.distributed.run --nproc_per_node=1 reward_summarization.py --bf16torch 2.0ã§ python -m torch.distributed.launch --nproc_per_node=1 reward_summarization.py -bf16 ã䜿ããšã次ã®ããã«ãªããŸãã
ValueError: Some specified arguments are not used by the HfArgumentParser: ['--local-rank=0']
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 194889) of binary: /home/ubuntu/miniconda3/envs/trl/bin/python5. ãã®ã³ã°
5-1. ãã®ã³ã°
匷ååŠç¿ã¢ã«ãŽãªãºã ã¯æŽå²çã«ãããã°ãé£ããããããã®ã³ã°ã«çްå¿ã®æ³šæãæãããšãéèŠã§ããããã©ã«ãã§ã¯ãTRL PPOTrainer ã¯å€ãã®é¢é£æ å ±ã wandb ãŸã㯠tensorboard ã«ä¿åããŸãã
åæåæã«ã次ã®2ã€ã®ãªãã·ã§ã³ã®ããããã PPOConfig ã«æž¡ããŸãã
config = PPOConfig(
model_name=args.model_name,
log_with=`wandb`, # or `tensorboard`
)tensorboard ã䜿çšããŠãã°ãèšé²ããå Žåã¯ãkwargaccelerator_kwargs={"logging_dir": PATH_TO_LOGS} ã PPOConfig ã«è¿œå ããŸãã
5-2. éèŠãªãã©ã¡ãŒã¿
åŠç¿äžãå€ãã®ãã©ã¡ãŒã¿ãèšé²ãããŸãããããã§ã¯æãéèŠãªãã®ã玹ä»ããŸãã
ã»env/reward_mean, env/reward_std, env/reward_dist : ãç°å¢ãããã®å ±é ¬ååžã®ç¹æ§ã
ã»ppo/mean_scores : å ±é ¬ã¢ãã«ããçŽæ¥åºåãããå¹³åç¹ã
ã»ppo/mean_non_score_reward : åŠç¿äžã®KLããã«ãã£ã®å¹³åå€ (ã¹ãããã®ãããã§åç §ã¢ãã«ãšæ°ããããªã·ãŒã®éã®deltaã瀺ã)
5-3. åŠç¿ã®å®å®æ§ãã©ã¡ãŒã¿
å®å®æ§ãç£èŠããã®ã«åœ¹ç«ã€ããã€ãã®ãã©ã¡ãŒã¿ã次ã«ç€ºããŸãããããã®ãã©ã¡ãŒã¿ãçºæ£ããã0ã«åŽ©å£ããå Žåã¯ã倿°ã調æŽããŠã¿ãŠãã ããã
ã»ppo/loss/value : 䟡å€é¢æ° lossãããŸããããªãå Žåã¯spike / NaN ã«ãªããŸãã
ã»ppo/val/clipfrac : å€é¢æ°æå€±ã«ãããã¯ãªãããããå€ã®å²åã å€ãã®å Žåããã㯠0.3 ïœ 0.6 ã§ãã
ã»object/kl_coef : AdaptiveKLController ã§ã®ã¿ãŒã²ããä¿æ°ã å€ãã®å Žåãæ°å€ãäžå®å®ã«ãªãåã«å¢å ããŸãã