
ChatRWKVãæ¥æ¬èªã§è©Šã(Windows11, VRAM 11GBç°å¢ïŒ
ããŒã«ã«ã§åããããšãã§ãããœãŒã¹ã³ãŒããã¢ãã«ãApache Licenceã§æäŸãããŠããå€§èŠæš¡èšèªã¢ãã«ã®RWKVãççã«ã¢ããããŒããç¹°ãè¿ãããŠããããªãã¡ãã£ãŠChatGPTãããã®æ§èœãåºãããªæåŸ æãé«ãŸã£ãŠããŸãã
ãã®RWKVã§ãã£ããã®ããã«ãããšãã§ããChatRWKVãšããããã°ã©ã ããããŸãã
ããã«ããã®RWKVã®ã¢ãã«ãAlpaca, CodeAlpaca, Guanaco, GPT4Allã§ãã¡ã€ã³ãã¥ãŒã³ããRWKV-4 "Raven"-seriesãšããã¢ãã«ã®ã·ãªãŒãºãããããã®äžã«ã¯æ¥æ¬èªã䜿ããç©ãå«ãŸããŠããŸãããã¡ã€ã«åã«ãEngChnJpnããå«ãŸããŠãã以äžã®ãã¡ã€ã«ã§ããïŒ14.8GBããã®ã§ãã£ã¹ã¯å®¹éã«æ³šæïŒ
ãããã䜿ã£ãŠãããŒã«ã«ã§ChatGPTã£ãœããã®ãåãããŠã¿ãŸããç§ã®ç°å¢ã¯ãWindows11ã§ã䜿ã£ãŠããGPUã¯VRAMã11GBã®RTX2080Tiã§ãããã5幎è¿ãåã®GPUã§ããããããšè¯ãæãã«åããŸããã
ç°å¢ã®ã»ããã¢ãã
ããã¯è¯ã解説ãããã®ã§ããã£ãšæµããŸãããã®éšåã¯ããããã®åå¿è åãããŒãžãäžã®äžã«ãããŸãã®ã§ããã¡ããåèã«ããŠãã ããã
Python 3.10ãã€ã³ã¹ããŒã«
Windows powershellã§git clone https://github.com/BlinkDL/ChatRWKV.git ããŠChatRWKVã®ãªããžããªãã¯ããŒã³
cd ChatRWKVã§ã«ã¬ã³ããã£ã¬ã¯ããªãç§»å
venvç°å¢ãäœã£ãŠãvenvç°å¢ã«ãã
pip install ninjaã§ninjaãã€ã³ã¹ããŒã«
pip install rwkvã§rwkvãã€ã³ã¹ããŒã«
ã¢ãã«ãã¡ã€ã«ã®ã³ã³ããŒã
ããŠã³ããŒãããRWKV-4-Raven-7B-v6-EngChnJpn-20230401-ctx4096.pth ããChatRWKVãã©ã«ãã«çœ®ããŸãã
14.8GBã®ãã¡ã€ã«ã¯ç§ã®ç°å¢ã§ã¯VRAMã«åãŸããªããããã³ã³ããŒãããŠå®¹éãå°ããããŸãã
ã³ã³ããŒãã¯ãvenvç°å¢ã«ãªã£ãŠããChatRWKVãã©ã«ããããpowershellã§./v2/convert_model.pyã䜿ã£ãŠä»¥äžã®æ§ã«è¡ããŸããæåŸã®strategyãªãã·ã§ã³ã§ãå€ææ¹æ³ã決å®ããŠããŸãã
python .\v2\convert_model.py --in .\RWKV-4-Raven-7B-v6-EngChnJpn-20230331-ctx4096.pth --out .\cudafp16i8.pth --strategy 'cuda fp16i8'--strategyãªãã·ã§ã³ã«ã€ããŠã¯ãPyPIã®rwkvã®ããŒãžã«è©³çްããããŸããåèªã®ç°å¢ã«åãããŠãã¥ãŒãã³ã°ããããšãæšå¥šãããŠããŸããç§ã®å Žåã¯ãå šã¬ã€ã€ãŒãfp16i8ã«ããããšã§ã50ïŒ åæžããããšããããã§ãã
########################################################################################################
#
# Use '/' in model path, instead of '\'. Use ctx4096 models if you need long ctx.
#
# fp16 = good for GPU (!!! DOES NOT support CPU !!!)
# fp32 = good for CPU
# bf16 = worse accuracy, supports CPU
# xxxi8 (example: fp16i8, fp32i8) = xxx with int8 quantization to save 50% VRAM/RAM, slower, slightly less accuracy
#
# We consider [ln_out+head] to be an extra layer, so L12-D768 (169M) has "13" layers, L24-D2048 (1.5B) has "25" layers, etc.
# Strategy Examples: (device = cpu/cuda/cuda:0/cuda:1/...)
# 'cpu fp32' = all layers cpu fp32
# 'cuda fp16' = all layers cuda fp16
# 'cuda fp16i8' = all layers cuda fp16 with int8 quantization
# 'cuda fp16i8 *10 -> cpu fp32' = first 10 layers cuda fp16i8, then cpu fp32 (increase 10 for better speed)
# 'cuda:0 fp16 *10 -> cuda:1 fp16 *8 -> cpu fp32' = first 10 layers cuda:0 fp16, then 8 layers cuda:1 fp16, then cpu fp32
#
# Basic Strategy Guide: (fp16i8 works for any GPU)
# 100% VRAM = 'cuda fp16' # all layers cuda fp16
# 98% VRAM = 'cuda fp16i8 *1 -> cuda fp16' # first 1 layer cuda fp16i8, then cuda fp16
# 96% VRAM = 'cuda fp16i8 *2 -> cuda fp16' # first 2 layers cuda fp16i8, then cuda fp16
# 94% VRAM = 'cuda fp16i8 *3 -> cuda fp16' # first 3 layers cuda fp16i8, then cuda fp16
# ...
# 50% VRAM = 'cuda fp16i8' # all layers cuda fp16i8
# 48% VRAM = 'cuda fp16i8 -> cpu fp32 *1' # most layers cuda fp16i8, last 1 layer cpu fp32
# 46% VRAM = 'cuda fp16i8 -> cpu fp32 *2' # most layers cuda fp16i8, last 2 layers cpu fp32
# 44% VRAM = 'cuda fp16i8 -> cpu fp32 *3' # most layers cuda fp16i8, last 3 layers cpu fp32
# ...
# 0% VRAM = 'cpu fp32' # all layers cpu fp32
#
# Use '+' for STREAM mode, which can save VRAM too, and it is sometimes faster
# 'cuda fp16i8 *10+' = first 10 layers cuda fp16i8, then fp16i8 stream the rest to it (increase 10 for better speed)
#
# Extreme STREAM: 3G VRAM is enough to run RWKV 14B (slow. will be faster in future)
# 'cuda fp16i8 *0+ -> cpu fp32 *1' = stream all layers cuda fp16i8, last 1 layer [ln_out+head] cpu fp32
#
# ########################################################################################################./v2/chat.pyã®æžãæã
./v2/chat.pyã«ãstrategyãã¢ãã«ãã¡ã€ã«ãæå®ããç®æãããã®ã§ããããæžãæããŸãã
./v2/chat.pyã®44ïœ50è¡ç®ä»è¿ã§ãstrategyã®ãªãã·ã§ã³ãcuda fp16i8ã«ããŸãã
# args.strategy = 'cpu fp32'
args.strategy = 'cuda fp16'
# args.strategy = 'cuda:0 fp16 -> cuda:1 fp16'
# args.strategy = 'cuda fp16i8 *10 -> cuda fp16'
# args.strategy = 'cuda fp16i8'
# args.strategy = 'cuda fp16i8 -> cpu fp32 *10'
# args.strategy = 'cuda fp16i8 *10+'
âã倿Ž
# args.strategy = 'cpu fp32'
# args.strategy = 'cuda fp16'
# args.strategy = 'cuda:0 fp16 -> cuda:1 fp16'
# args.strategy = 'cuda fp16i8 *10 -> cuda fp16'
args.strategy = 'cuda fp16i8'
# args.strategy = 'cuda fp16i8 -> cpu fp32 *10'
# args.strategy = 'cuda fp16i8 *10+'
./v2/chat.pyã®61è¡ç®ä»è¿ã§ãèªã¿èŸŒãã¢ãã«ãå ã»ã©å€æããã¢ãã«ã«ããŸãã
# Use convert_model.py to convert a model for a strategy, for faster loading & saves CPU RAM
if CHAT_LANG == 'English':
args.MODEL_NAME = '/fsx/BlinkDL/HF-MODEL/rwkv-4-raven/RWKV-4-Raven-14B-v6-Eng-20230401-ctx4096' # try +i for "Alpaca instruct"
# args.MODEL_NAME = '/fsx/BlinkDL/HF-MODEL/rwkv-4-raven/RWKV-4-Raven-7B-v6-Eng-20230401-ctx4096' # try +i for "Alpaca instruct"
â 倿Ž
# Use convert_model.py to convert a model for a strategy, for faster loading & saves CPU RAM
if CHAT_LANG == 'English':
args.MODEL_NAME = './cudafp16i8.pth' # try +i for "Alpaca instruct"
# args.MODEL_NAME = '/fsx/BlinkDL/HF-MODEL/rwkv-4-raven/RWKV-4-Raven-7B-v6-Eng-20230401-ctx4096' # try +i for "Alpaca instruct"倿Žãããã¡ãããšã»ãŒãããŠãããŸããããããšã¯PowerShellã§ä»¥äžã®æ§ã«èµ·åããã°ãPowerShelläžã§ãã£ããåºæ¥ãããã«ãªãã¯ãã§ãã
python .\v2\chat.pyå®éã«ãããšãããŠã¿ããšãChatGPTã»ã©ã¯è³¢ããããŸãããããªããªãé¥èã§æ¥œããã€ããªã®ã§å¯æãèŠããŠãããšæããŸãïŒç¬ïŒ
Bob: ããã«ã¡ã¯ã¢ãªã¹ãããªãã®èªå·±ç޹ä»ãããŠãã ãã
Alice: ããã«ã¡ã¯ãã¢ãªã¹ã§ããç§ã¯ã¢ãªã¹ãšãããŸããç§ã¯åžžã«è峿·±ãããšã«ææŠããŠããŸããç§ã®è¶£å³ã¯ãç°¡åãªç©äºãç¶ã
ãããšã§ããåžžã«å€åããŠãããããªçŽ æŽãããçµéšãããŠãã人ã«ãšã£ãŠã楜ãããŠå°ãªãæéãå å®ãããããã«ãã£ãšãéããããã»ããããã®ã§ã¯ãªãã§ããããã
Bob: å°å¹ŽæŒ«ç»ã®äž»äººå ¬ã®å¿ 殺æã5åèããŠãã ãã
Alice: å°å¹ŽæŒ«ç»ã®äž»äººå ¬ã®å¿ 殺æã5åèããŠãã ããã
1. å¿ æ®ºæ: "ãã¢ãã®å¥ã§ããµãŠã³ã" - 鿥ãé§ããäžå¿ã§å¥ã§ãå¿ æ®ºæã§ãããäœåºŠãæ¯ãè¿ã£ãŠãæ°é®®ãªæããããã
2. "ãã¢ãã®åŒŸãèªã" - æã«ç¬ç¹ãªé°å²æ°ãæã£ããŸãŸããµãŠã³ããäžæã«äŒããããå¿ æ®ºæã§ãããèªç¶ãªãã³ãã§èªããããããšãã§ããã
3. "ãã±ããã¹ã¯ã©ã³ãã«" - 髿æãæãããã¹ã¯ã©ã³ãã«ã¹ãã¬ãŒããäžæãæŽ»çšã§ããå¿ æ®ºæã§ããã
4. "ãžã£ã³ãç匟" - ç±ãå³ãããæº¢ããç匟ã¹ã¯ã©ã³ãã«ã®äžããé£ã³ç«ã€äžæã®ãžã£ã³ãç匟ã
5. "ã¢ãã«ãããšãã«ãªããã©ãŒã " - ãã€ãã®ãšã¯ãµãµã€ãºãšéã£ãŠãå°å¹ŽæŒ«ç»çãªã¢ã¯ã·ã§ã³ã楜ããããã®ãªãªãžãã«æè¡ã§ããã
6. "ãžã£ã³ãããã¯" - å°å¹ŽæŒ«ç»ã®ãã£ã©ã¯ã¿ãŒãäœã䜿ã£ãè ç«ãŠäŒããšãçžæ²ãæéã®äžéã«äœçœ®ãããžã£ã³ãããã¯ã¯ãããããææ³ã§ãã
7. "éšæŒãäŒããçé³" - å°ããªåããã倧ããªé³ãçºããåçŽãã€èä¹ æ§ãæã¡ãè¡æã«èãããã身äœèœåãæã€å°å¹ŽæŒ«ç»ã®äŒããå³ããã
8. "ãšã¯ãµãµã€ãºçãªæŽåè¡å" - çžæã«ç¡é²åã«ãªãã€ã€çªé²ããããçŽ æ©ã身äœãåããããããåãã§ããããé°å²æ°ãåŒãåºãããšãã§ããã
9. "ã¢ã¯ã·ã§ã³ã·ã¥ãŒãã£ã³ã°" - å°ããªåéºè ã§ããå°å¹ŽæŒ«ç»ã®äžã§ã極ããŠã¢ã¯ã·ã§ã³çãªå³ããã®ã·ãŒã³ãã§ããã
10. "ã€ããžããŒã·ã§ã³æãã®ãšãã»ã€" - å°å¹ŽæŒ«ç»ã¯äººéé¢ä¿ã䟡å€èгãªã©ãæãåºãããšãå€ãããããããã®äºå®é¢ä¿ã«åºã¥ã
ãŠãã€ããžããŒã·ã§ã³å³ããšåŒã°ãããšãã»ã€ãæžãããŠããŸããããã¯é£è§£ã§ãããšããããã¯é©åã ãšæããŸãã
ããªãã¯å°åŠçã髿 ¡çã§ããïŒããªãã®ã€ã¡ãŒãžã¯ã©ã¡ãã§ããïŒå°æ¬ããŠãã人ç©ïŒå»è
ãå§ã挫ç»ïŒçæžé çŽ
æè¿è²·ã£ãç©ïŒã¿ãã¥ãŒå¥œãã®äººç©
æè¿åéã«ã¯åã¶ãåãããªãäºã«ãªã£ãŠããšæã
奜ãã ãã©ãåéã«ã¯åã°ãªãäºã«ãªã£ãŠããŠããŸã è¯ããšæã
仿æ«ã«äŒãäºå®ã ã£ãããïŒïŒç¬ïŒ
æ¬åœã«éãè¯ãã£ãã
æåŸã«ãå§ãããç©ã¯ïŒ
ãå§ãç©ã§ã¯ãªãã§ãïŒããŸãæšå¥šããªãã§ããïŒ
æåŸã«ãå§ãããç©ã¯ïŒ
å šãŠèšããªãããçããããªã(ç¬)
仿æ«ã«äŒãäºå®ã ã
æåŸã«
å¿ æ®ºæã5åèããŠãšèšã£ãã®ã«10åèããã°ããããªã¹ã¹ã¡ã®æŒ«ç»ãèªåã®å¥œããªç©ãèªãå§ããããããæ©å£ãªã¿ã¯æããã£ãŠããã§ããã¡ãªã¿ã«ãPythonã®ã³ãŒããªã©ãèããŠãããŸããããªã䜿ãããã§ãã
RWKVã¯ãã¡ã€ã³ãã¥ãŒãã³ã°ãã§ãããããªã®ã§ãèªå奜ã¿ã«äœãæ¿ããŠããããšãã§ãããã§ãããªãŒãã³ãªã¢ãã«ãããŒã«ã«ã§åããäºã®è¯ãã¯ãå€åœã®äŒç€Ÿãèšå®ããåºæºã«çžãããªãããšã ãšæãã®ã§ãèªç±ãªçºæ³ãå¿ èŠã«ãªãå Žé¢ã§ã¯ãšãã«æãé£ãååšã«ãªãã®ã§ã¯ããšæã£ãŠããŸãã
åŒãç¶ãããªãŒãã³ãªã¢ãã«ãããŒã«ã«ã§åããããšã«æ³šç®ããŠãããããšæããŸãã
远èš
äžèšã¯7BïŒ70åãã©ã¡ãŒã¿ãŒïŒã¢ãã«ãåããããã®ã§ããã14BïŒ140åãã©ã¡ãŒã¿ãŒïŒããconvert_model.pyã以äžã®æ§ã«ããããšã§ãVRAM 11GBç°å¢ã§åããŸããã
python .\v2\convert_model.py --in .\RWKV-4-Raven-14B-v6-EngChnJpn-20230401-ctx4096.pth --out ./14bfp16i8_25_cpufp32.pth --strategy 'cuda fp16i8 *25 -> cpu fp32'ã¢ãã«ã®25ã¬ã€ã€ãŒåãCUDAã«ãæ®ããCPUã«éãããšããèšå®ã§ãã

./v2/chat.py ãã以äžã®æ§ã«ãªããŸãã
args.strategy = 'cuda fp16i8 *25 -> cpu fp32'
ïŒäžç¥ïŒ
ãargs.MODEL_NAME = './v2/14bfp16i8_25_cpufp32.pth'ãã ãçµå±CPUã®ã¡ã¢ãªãçµç±ããã®ã§ãé ãã§ããå®çšçãªé床ã§ã¯ãããŸããã§ããã24GBã48GBã®VRAMæèŒã®ãããªã«ãŒããè²·ãã°14Bã§ãã¹ãã¬ã¹ãªãåãããïŒ