MiniGPT-4ãã«ãã¢ãŒãã«LLMãããŒã«ã«ã§åãã
ã€ã©ã¹ããåçãAIã«èª¬æãããããªãããšããåæ©ãããã«ãã¢ãŒãã«LLMãåãããŠã¿ãŸããããã«ãã¢ãŒãã«ã¯ChatGPT-4ã§ãã§ããŸãããingã§ãã§ããŸããã諞è¬ã®äºæ
ã§ãªãŒãã³ãœãŒã¹ãçšãããŠãã«ãŒã«ç°å¢ã§åããããšã«ããŸãããæè¿ã§ã¯rinna-4BãStabirityAIããã«ãã¢ãŒãã«LLMãçºè¡šããŠããŸãããçè
ã詊ããŸããããåçãããããã£ããããŠããŠç©è¶³ããªããæããŸãããããã§ã以åã«æ³šç®ããŠããMiniGPT-4ãåããããšã«ããŸããããã¡ãªã¿ã«rinna-4Bã§åãããŠãããã«ãã¢ãŒãã«ã¯MiniGPT-4ããŒã¹ã§ãã
ïŒïŒVicuna-v0ïœv1.3ãŸã§ã¯éåçšã§ããv1.5ã¯åçšã§å©çšã§ããããã§ãããã ããv1.5ã¯MiniGPT-4ã§ã¯äœ¿ããŸãããïŒ
è«æå·çè
ããŒã ããŒãžã¯ä»¥äžã«ãªããŸãã
https://minigpt-4.github.io/
ãã¢ãåãããŠã¿ã
ããã§ã¯ãªã³ã©ã€ã³ãã¢ã詊ãããšãã§ããAbstractã®äžã«äžŠãã ãã¿ã³ã®ïŒ³paceãããã¢ç»é¢ã«å ¥ãããšãã§ããŸããææã¡ã®ã€ã©ã¹ãã§ã©ã®ãããªåçãåŸãããã®ã詊ããŠèŠãŸããã

ããã³ããïŒexplain this picture
以äžã®ãããªåçããããŸããã
This image depicts a beautiful garden with pink and white flowers and a white gazebo in the background. The garden is surrounded by tall trees and a pathway leading to the gazebo. The sky is a light blue with clouds and a few birds flying in the sky. The overall atmosphere of the image is peaceful and serene.
æ¥æ¬èªã«èš³ããšãããªæã
ããã®ç»åã¯ãèæ¯ã«ãã³ã¯ãšçœã®è±ãšçœã西æŽé¢šãããŸããããçŸããåºåãæããŠããŸããåºåã¯é«ãæšã
ã«å²ãŸãã西æŽé¢šãããŸãã«ç¶ãå°éããããŸããç©ºã¯æ°Žè²ã§ãé²ããããæ°çŸœã®é³¥ã空ãé£ãã§ããŸããçµµå
šäœã®é°å²æ°ã¯ç©ããã§ç©ããã§ããã
æ¥æ¬èªã®ããã³ããã䜿ããŸããææ«ã«ãæ¥æ¬èªã§çããŠããšæžããšåçãæ¥æ¬èªã«ãªããŸãã
ããŒã«ã«ã§ã®åããæ¹
ã·ã¹ãã æ§æ
第12äžä»£ãi5
ã¡ã¢ãªãŒã32G
GPUãããGForce-3060-12G
OSããããUbuntu20.04-JP
ãœãŒã¹ã³ãŒãã¯ãã¡ãã§ãã
åããæ¹ã¯ãªããžããªã®READ.MEã«èšèŒãããŠãã®ã§ã倧ããªåé¡ã¯ãªããšæããŸãããçè ã¯å¿ å®ã«å®è¡ããããšããŠããæžæããŸããã
ç°å¢èšå®
git clone https://github.com/Vision-CAIR/MiniGPT-4.git
cd MiniGPT-4
conda env create -f environment.yml
conda activate minigpt4ãªããžããªéãã§åé¡ã¯ãªãã¯ãã§ãã
MiniGPT-4ã§ã¯LLMãšããŠVicuna-13BãçšããããŠããŸãããã¢ãšåãããã«åããã«ã¯Vicuna-13Bãå¿
èŠã§ãããVRAMã24Gå¿
èŠã ãšã®ããšã§ãVicuna-7Bã§åããããšãã§ããcheckpointãæºåãããŠããŸããããã¡ãã§ããš3060ãªã©ã
VRAM-12Gãããã°åããŸããä»åã¯Vicuna-7BçãåãããŸãã
Vicunaã¢ãã«ã®æºå
ãªããžããªã§ã¯v0ããŒãžã§ã³ãçšããããšèšèŒãããŠããã®ã§ãæç€ºéãVicuna-v0ãæºåããããã«ãªããžããªã«èšèŒãããäœææ¹æ³ã®ãªã³ã¯å ãªããžããªãã¿ãŸããã
https://github.com/Vision-CAIR/MiniGPT-4/blob/main/PrepareVicuna.md
ããã§åé¡çºçãVicuna-v0ãäœæããããã«ã¯ïŒetaã«LLAMA-7BãLLAMA-13Bã®å©çšç³è«ãããå¿ èŠãããã審æ»ã«éããŸã§5æ¥çšåºŠå¿ èŠã®ããã§ãã
v0ã§ç¡ããŠãåã
çµè«ããèšããŸããšVicuna-v0ã§ãªããŠãåããŸããVicunaã®ãªããžããªã«äžèЧãèšèŒãããŠããŸãã
https://github.com/lm-sys/FastChat/blob/main/docs/vicuna_weights_version.md
v0以å€ã¯ãã§ã«ã¢ãã«ãäœææžã¿ãªã®ã§ãåä»ãªæç¶ããæ§ç¯äœæ¥ãªãã«ããŠã³ããŒãã ãã§å©çšã§ããŸãã以äžã®ãããªãã£ã¬ã¯ããªæ§é ã«ããããæ±ããããŠããã®ã§ãMiniGPT-4ãã£ã¬ã¯ããªå
ã«vicuna-7b-v1.3ãã£ã¬ã¯ããªãäœæããV1.3 7å
ã®å
šãŠã®ãã¡ã€ã«ãããŠã³ããŒãããŸãã
vicuna-7b-v1.3
âââ config.json
âââ generation_config.json
âââ pytorch_model.bin.index.json
âââ pytorch_model-00001-of-00003.bin
...
ãã£ã¬ã¯ããªã/minigpt4/configs/models/minigpt4.yamlã®16è¡ç®ã«èšå®ããŸãã
minigpt4.yaml
model:
arch: mini_gpt4
# vit encoder
image_size: 224
drop_path_rate: 0
use_grad_checkpoint: False
vit_precision: "fp16"
freeze_vit: True
freeze_qformer: True
# Q-Former
num_query_token: 32
# Vicuna
llama_model: "./vicuna-7b-v1.3/"
# generation configs
prompt: ""
preprocess:
vis_processor:
train:
name: "blip2_image_train"
image_size: 224
eval:
name: "blip2_image_eval"
image_size: 224
text_processor:
train:
name: "blip_caption"
eval:
name: "blip_caption"次ã«checkpointãã£ã¬ã¯ããªãäœæããŠMiniGPT-4ã®checkpointãããŠã³ããŒãããŸããã衚ã«ã¯Checkpoint Aligned with Vicuna 13BãšCheckpoint Aligned with Vicuna 7BããããŸãããä»åã¯Checkpoint Aligned with Vicuna 7Bãcheckpointãã£ã¬ã¯ããªã«ããŠã³ããŒãããŸãããã®åŸã
eval_configs/minigpt4_eval.yaml
ã®11è¡ç®ã«checkpointãæå®ããŸãã
minigpt4_eval.yaml
model:
arch: mini_gpt4
model_type: pretrain_vicuna
freeze_vit: True
freeze_qformer: True
max_txt_len: 160
end_sym: "###"
low_resource: True
prompt_path: "prompts/alignment.txt"
prompt_template: '###Human: {} ###Assistant: '
ckpt: './checkpoint/pretrained_minigpt4_7b.pth'
datasets:
cc_sbu_align:
vis_processor:
train:
name: "blip2_image_eval"
image_size: 224
text_processor:
train:
name: "blip_caption"
run:
task: image_text_pretrain
eval_configs/minigpt4_eval.yamlããã§æºåå®äºã§ãã
ãã¢ãåãã
ãªããžããªéãã®ã³ãã³ããå®è¡ããŸãã
python demo.py --cfg-path eval_configs/minigpt4_eval.yaml --gpu-id 0åæåãå§ãŸããå
šãŠäžæããããšãURLããã¢ã¯ã»ã¹ã§ããããã«ãªããŸãã
ããã©ã«ãã§ã¯ã
http://127.0.0.1:7860/
ã«ãªããŸãã
Initializing Chat
Downloading (âŠ)solve/main/vocab.txt: 100%|ââââ| 232k/232k [00:00<00:00, 778kB/s]
Downloading (âŠ)okenizer_config.json: 100%|âââ| 28.0/28.0 [00:00<00:00, 2.42kB/s]
Downloading (âŠ)lve/main/config.json: 100%|ââââââ| 570/570 [00:00<00:00, 249kB/s]
Loading VIT
100%|ââââââââââââââââââââââââââââââââââââââ| 1.89G/1.89G [00:57<00:00, 35.4MB/s]
Loading VIT Done
Loading Q-Former
100%|ââââââââââââââââââââââââââââââââââââââââ| 413M/413M [00:12<00:00, 35.8MB/s]
Loading Q-Former Done
Loading LLAMA
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
Loading checkpoint shards: 100%|ââââââââââââââââââ| 2/2 [00:30<00:00, 15.22s/it]
Loading LLAMA Done
Load 4 training prompts
Prompt Example
###Human: <Img><ImageHere></Img> Please provide a detailed description of the picture. ###Assistant:
Load BLIP2-LLM Checkpoint: ./checkpoint/prerained_minigpt4_7b.pth
Initialization Finished
Running on local URL: http://127.0.0.1:7860
Running on public URL: https://74bc1a00161547e37d.gradio.live
This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
http://127.0.0.1:7860/ã«ã¢ã¯ã»ã¹ãããšã以äžã®ç»é¢ãéããŸãããªãªãžãã«ãšåããã¢ç»é¢ãåããããšãã§ããŸããã

詊ãã«ãµã€ãã®ãã¢çãšåãããã«äœ¿ã£ãŠã¿ãŸããã

ãµã€ããã¢ããã詳ãã説æãããŠãããŸãããæ¥æ¬èªã ãšä»¥äžã®ãããªæãã§ãã
ãããã¯ããã³ã¯ãšçœã®è±ãå Žé¢å šäœã«æ£ãã°ããããŠãããæºéã®å ¬åã®çµµã§ããèæ¯ã«ãçœãšãã³ã¯ã®å±æ ¹ãšå€§ããªããŒã ãæã€å»ºç©ãèŠããŸããå°é¢ã¯è±ã§èŠããããã³ã¯ãšçœã®è±å£ãããã€ããããŸããæ©éã«ã¯æšã ãæ€ããããŠãããè¿ãã«ã¯ãã³ããããã€ããããŸããé ãã«éãæ°Žãããããæ¹ããããŸãã
çµµã¯ãã³ã¯ãšçœã®è±ãå²ãåºåãèå°ã«ããŠããŸãã ãã®çµµã§ã¯ããã³ã¯ãšçœã®è±ãå²ãå ¬åãããçœãšãã³ã¯ã®å±æ ¹ãšå€§ããªããŒã ã®ãã建ç©ãŸã§å°éãç¶ããŠããŸããå°é¢ã¯è±ã§èŠããããã³ã¯ãšçœã®è±å£ãããã€ããããŸããæ©éã«ã¯æšã ãæ€ããããŠãããè¿ãã«ã¯ãã³ããããã€ããããŸããé ãã«éãæ°Žãããããæ¹ããããŸãã
çµµã¯ãã³ã¯ãšçœã®è±ãå²ãåºåãèå°ã«ããŠããŸãããã®éã¯ãçœãšãã³ã¯ã®å±æ ¹ãšå€§ããªããŒã ãæã€å»ºç©ã«éããŠããŸããå°é¢ã¯è±ã§èŠããããã³ã¯ãšçœã®è±å£ãããã€ããããŸããæ©éã«ã¯æšã ãæ€ããããŠãããè¿ãã«ã¯ãã³ããããã€ããããŸããé ãã«éãæ°Žãããããæ¹ããããŸãã
çµµã¯ãã³ã¯ãšçœã®è±ãå²ãåºåãèå°ã«ããŠããŸãããã®éã¯çœãšãã³ã¯ã®å±æ ¹ãšå€§ããªããŒã ã®ãã建ç©ã«éããŠããŸãã
éè€ãããŸããããªããªãããæãã§ãã
ãŸãšã
ä»åã¯Vicuna 7Bã§ãããã3090ã4090ã§ããã°åãæ¹æ³ã§Vicuna 13Bãåãã¯ãã§ãããrinna-4BãStabilityAIãããé«ç²ŸåºŠãªãã«ãã¢ãŒãã«ããããŒã«ã«ã§åããããšãã§ããããšã§çšéãåºãããŸãã
åçšå©çšã«ã€ããŠ
Vicuna-v0ïœv1.3ãŸã§ã¯éåçšã®å¯èœæ§ããããŸããv1.5ã¯éåçšã§ééããããŸããã