
Llama3.1ãããŒã«ã«ã§åãããŠã¿ããå®å šç
ããŒã«ã«ã§åããããšãã§ããææ°ã®ãªãŒãã³ãœãŒã¹LLMãåãããŸããã
ã¢ãã«ã¯ä»¥äžã®
Llama-3.1-8B-Instruct-Q4_K_M.gguf
ã§ããåããããšã¯ã§ããŸããããæ®éãããªãåãã§ãã以äžã¬ããŒãã
Metaã®ãµã³ãã«ã³ãŒããåããã
ãããåããªãããªãªãžãã«ã®ã³ãŒãã¯ã¢ãã«ãèªåããŠã³ããŒãããŠãããªãããCLIã§ãèœãšããªãããªããªãã§ãããããä»ã®ã¢ãã«ã ãšç°¡åã«èœãšããã®ã§ããã©ã諊ããŠggufã詊ããŸããçè ã¯åžžã LLMã¢ãã«ã¯ggufã䜿ãã®ã§ãtransformersã§åããªããŠãllama.cppã§ggufãåããŠãããã°åé¡ã¯ãããŸããã
ggufã¢ãã«ãåãã
ã¢ãã«ã¯ãã€ãã®ããã«ãããªã«çããããggufãžå€æããŠãã ãã£ãŠãŸãã
ãã®äžãããæãäžè¬çãªQ4_K_Mã詊ããŸãã
Llama-3.1-8B-Instruct-Q4_K_M.ggufããããŠã³ããŒãããŠãllama.cppã®modelsãã©ã«ããŒã«ç§»åãããŠãããŸãã
èµ·åã³ãã³ã
./llama-server -m ./models/Llama-3.1-8B-Instruct-Q4_K_M.gguf -c 2048 -n 128 --n_gpu_layers 81 --host 192.168.5.71
llama.cppã®OpenAIäºæãµãŒãã§ãã
ã¯ã©ã€ã¢ã³ãåŽ
äžã€åã®èšäºã®ããã°ã©ã ã䜿ããŸããbaseã¢ãã¬ã¹ã¯äžèšãµãŒãã®hostãšåãããŠãã ããã
åãããšãã®GPUã®å©çšç¶æ³ã6.5GçšåºŠã§éåãšå°ãªãã§ãã

äŒè©±ã®çµæãèšæ¶ãæ©èœããŠããããã§ãã
User: ç®é»å·ã¯ãããïŒ
ãã: ç®é»å·ã£ãŠãããïŒïŒïŒ
User: ã©ãã«ããã®ããªïŒ
ãã: ç®é»å·ã¯åå·åºã®è¿ãã«ããå·ã ããç®é»å·ã¯ãåå·åºã®æ±ã«æµããææãšæ°å·ãšã€ãªãã£ãŠæµ·ã«æ³šãå·ãªãã ãåå·åºã®äžå¿ã®åå·é§
ã®è¿ãã«ç®é»å·ã®æ°Žè·¯ããããé§
ã®ååŽã«ã¯å·ã暪æããç®é»å·æ©ããããã
çæé床
52.92 tokens per secondãã§ååã«æ©ãã§ãã
70Bã¯ã©ãã
ã¢ãã«ã¯
llama-3.1-70B-Instruct-Q4_K_M.gguf
./llama-server -m ./models/Llama-3.1-70B-Instruct-Q4_K_M.gguf -c 2048 -n 128 --n_gpu_layers 81 --host 192.168.5.71VRAMã®äœ¿çšç¶æ³ããåèšã§ã43.27Gã§ãã

çµæ
8Bããããã£ããããçæãé
ãã§ãã
User: ç®é»å·ã¯ããã
ãã: ç®é»å·ã¯ãå·ã£ãŠèšãããããæ±ããâŠ
User: ã©ãã«ããã®ããªïŒ
ãã: ãïŒç®é»å·ïŒïŒïŒç®é»å·ã¯åå·åºã«ããããŒïŒ
çæãé
ãã®ã§ã¬ã€ã€ãŒã®ããŒãç¶æ³èŠãŸããããå
šå±€ä¹ã£ãŠããŸãã
llm_load_tensors: offloaded 81/81 layers to GPU
çæã¯ã7.59TPSããããé
ãã8bã ãš52.92 tokens per secon
GPUãå€ããŠèŠã
確ä¿ããŠãããA6000ã§è©ŠããŸããã

13.52 tokens per secondããŸããŸãã§ããã©ãæ©ãããã§ã¯ã¯ãã§ãã
奜ã¿çŒãã®äœãæ¹
Llama-3.1-8B-Instruct-Q4_K_M.ggufã®ãšãã§ãã
ã¡ãªã¿ã«ã奜ã¿çŒãã®äœãæ¹é
äžã®éãããªãã ãããã§ãã
70Bã ãšã ãã¶è¯ããªããã©ïŒïŒïŒãã§ããã
User: ã奜ã¿çŒãã®äœãæ¹ãã: ã¿ã³ãå»ãã§ããããéèã¯ã¿ããåãã«ããŠãã¿ã³ã¯å»ãã§ããã°ãåŸã§äœãã®ã«ãå°ãããšã¯ãªãããæ¬¡ã«ãã¿ãæ±ãæºåããŠããã¿ãæ±ã¯ããã®æ¯ãæäœãã§äœãã®ãæäŒããã ãã©ãããã®å Žåã¯æäœãã§äœãã®ãé¢åã ãããã¿ãæ±ã®ç²ã䜿ãããšã«ããããã¿ãæ±ãæºåããããæ°Žãå ¥ããŠæ··ããŠããããããã§ãããã®ç§äŒã®ã¿ã¬ã玹ä»ãããïŒããã®ã¿ã¬ã¯ãã¿ãæ±ãšé€æ²¹ãšé ãæ··ããŠäœããã ãã©ãããã¯é ãå«ãã ãããããã®å Žåã¯ã¿ãæ±ãšé€æ²¹ã ãã§äœãããšã«ããããæ¬¡ã«ãé€ ç²ãæ°Žã§æº¶ããŠæ··ããŠãå ·æãå ¥ããŠæ··ããŠãæåŸã«ããã®ç§äŒã®ã¿ã¬ãå ¥ããŠæ··ããŠã宿ïŒããã¯ã奜ã¿çŒãã倧奜ããªãã ãã©ãããã®æ¯ã¯ããã倪ãããšãå¿é ããŠããã ãããããã¯å€ªã£ãããã¡ã ãããããã¯ãã€ãã奜ã¿çŒããé£ã¹ãåã«ãäœéèšã§äœéãæž¬ã£ãŠã¿ããã ããããã¯äœéãå¢ããããã¡ã ãããããã¯ãã€ãäœéããã§ãã¯ãããã ãã
Llama-3.1-70B-Instruct-Q3_K_S.ggufã§è©Šã
ãããªã«æªããªãã§ãã
User: ã奜ã¿çŒãã®äœãæ¹
ããïŒ ãŸãææãæºåããŸããå°éºŠç²ãæ°Žãå¡©ãåµããã£ãããã«ãããããšããèïŒã奜ã¿ã§ïŒããœãŒã¹ãã奜ã¿çŒãç²ãªã©ãå¿
èŠã§ãã ææãæºåãããããŸããåµãçé ãã¹ã¿ã€ãã®ããŠã«ã«å
¥ããŠæ··ããŠãã ããããã®åŸãæ°Žãå ããŠããæ··ããŠãã ãããããã«ãå°éºŠç²ãšå¡©ãå ããŠãæ··ãåãããŠãã ãããæ··ãã£ããããã£ãããå»ãã§å ããŠãã ããããã£ãããæ··ãã£ãããã«ããããå»ãã§å ããŠãã ãããã«ããããæ··ãã£ããããšããèïŒã奜ã¿ã§ïŒãå ããŠãã ãããæ··ãã£ãããç±ãããã©ã€ãã³ã«å°éã®æ²¹ãå
¥ããååéããã©ã€ãã³ã«æµã蟌ã¿ãŸãããã®åŸããœãŒã¹ãšã奜ã¿çŒãç²ãæ£ãããŠãããäžåºŠååéæµã蟌ã¿ãŸããç«ãéã£ãããç¿ã«çã£ãŠåºæ¥äžããã§ãã
æåŸã«ãšã£ãŠããšã¢PCã§è©Šã
I5-9400F+ã3060-12Gx3å°ãšãããæ®éã§ã¯ãããªãã§ãããæ§æã§ããããŸããããã®æ§æã¯ã¹ãã³ãªã座è«äŒã®ãã¢ã§ã䜿çšããŠãã¬ã¹ãã³ã¹ã®éãã«é©ããæ¹ãããçšã§ãã
åãããã¢ãã«ã¯ãLlama-3.1-70B-Instruct-Q3_K_S.gguf
以äžVRAMã®ç¶æ
ãé©åºŠã«ããŒããããŠãŸãããã®PCã«ã€ããŠã¯70BçšåºŠã®ã¢ãã«ãã©ã®ãããåãã®ãå®åãç¥ããããšæã£ãŠããã®ã§ãã¡ããã©ããã¿ã€ãã³ã°ã§ãã

çæé床ã¯ã5.91 tokens per secondããªã®ã§ãã¯ãæ©ããªãªãã§ãããã¢ã®ãšãã¯gemma-2-8bã§40TPSãããåºãŠãã®ã§ãã¢ãã«ã倧ãããšå³ããã
User: ç®é»å·ã¯ãããïŒ
ãã: ãã£ã¡ããããïŒããã®å®¶ã®è¿ãã«ã¯ç®é»å·ãæµããŠããã ããããç®é»å·ã§éã¶ããšããåéãšå·ã®ã»ãšãã§ãè¶ããããšããããã ãå€ã¯ç®é»å·æ²¿ãã§ã¯ãç¥ããéãããŠãŠãã¡ãããŸããïŒ
User: ã©ãã«ããã®ïŒ
ãã: ç®é»å·ã¯æ±äº¬ãç®é»åºã«ãããŸãïŒããã®å®¶ã®è¿ãã«æµããŠããã§ãããã¯ç®é»å·ã«èŠªãã¿ããããŸãïŒ
ãã®PCã®å®åã«ã€ããŠã¯æ¥é±ã«ã§ãããŒã¿ãéããŠèšäºã«ããã€ããã§ãã
