
AutoGPTQ ãš transformers ã«ããLLMã®è»œéå
以äžã®èšäºãé¢çœãã£ãã®ã§ãããããŸãšããŸããã
1. ã¯ããã«
ãAutoGPTQãããtransformersãã«çµ±åããŸãããããã«ããããGPTQãã䜿çšããŠ8ã4ã3ã2bitã®ç²ŸåºŠã§ã¢ãã«ãéååããŠå®è¡ã§ããããã«ãªããŸãã4bitéååã«ãã粟床ã®äœäžã¯ç¡èŠã§ããå°ããããããµã€ãºã®æšè«é床ã¯fp16ããŒã¹ã©ã€ã³ã«å¹æµããŸãããã®çµ±åã¯ããNvidia GPUããšãRoCmæèŒAMD GPU ãã®äž¡æ¹ã§å©çšã§ããŸãã
2. AutoGPTQ
ãAutoGPTQãã䜿çšãããšãtransformersã¢ãã«ãéååã§ããŸãããGPTQ-for-LLaMaããExllamaããllama.cppããªã©ã®ã³ãã¥ããã£ã®åãçµã¿ã¯Llamaã¢ãŒããã¯ãã£å°çšã®éååææ³ãå®è£ ããŠããã®ã«å¯ŸãããAutoGPTQãã¯å¹ åºãtransformers ã¢ãŒããã¯ãã£ãã¹ã ãŒãºã«ã«ããŒããããšã§äººæ°ãåããŸããã
AutoGPTQã¯ã¢ãã«ãããåºç¯å²ã«ã«ããŒããŠãããããLLMéååã誰ã§ãç°¡åã«å©çšã§ããããã«ãçµ±åãããTransformers API ãæäŸããããšã«ããŸãããçŸæç¹ã§ã¯ãCUDAãªã©ã®æãäžè¬çãªæé©åãªãã·ã§ã³ãçµ±åããŸããããTriton ãããfused-attentionããªã©ã®ããé«åºŠãªãªãã·ã§ã³ã«ã€ããŠã¯ããAutoGPTQãããã§ãã¯ããŠãã ããã
3. transformersã§ã®GPTQã®ãã€ãã£ããµããŒã
ãAutoGPTQããšãoptimumã (pip install optimum) ãã€ã³ã¹ããŒã«ããããšã§ãtransformers ã§ GPTQã¢ãã«ã次ã®ããã«ç°¡åã«å®è¡ã§ããŸãã
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"TheBloke/Llama-2-7b-Chat-GPTQ",
torch_dtype=torch.float16,
device_map="auto"
)詳ããã¯ããã¥ã¡ã³ããåç §ããŠãã ããã
ãAutoGPTQãã®çµ±åã«ã¯å€ãã®å©ç¹ããããŸãã
ã»éååã¢ãã«ã¯ã·ãªã¢ã«åå¯èœã§ãããHubäžã§å ±æã§ããã
ã»GPTQã¯ãLLMãå®è¡ããããã®ã¡ã¢ãªèŠä»¶ãå€§å¹ ã«åæžããæšè«ã¬ã€ãã³ã·ã¯ FP16 æšè«ãšåçã
ã»AutoGPTQã¯ãå¹ åºãã¢ãŒããã¯ãã£ã®Exllamaã«ãŒãã«ããµããŒãã
ã»ãã®çµ±åã«ã¯ãAMD GPUã®ãã€ãã£ãRoCmãµããŒããä»å±ã
ã»PEFTã«ãããã¡ã€ã³ãã¥ãŒãã³ã°ãå¯èœã
ãæ°ã«å ¥ãã®ã¢ãã«ããã§ã«éååãããŠãããã©ãããHubã§ç¢ºèªã§ããŸãã HuggingFaceã®ãããã³ã³ããªãã¥ãŒã¿ãŒã®1人ã§ããTheBlokeæ°ã¯ãAutoGPTQ ã䜿çšããŠå€ãã®ã¢ãã«ãéååããHuggingFace Hubã§å ±æããŸããããããã®ãªããžããªãçµ±åãããŠæ©èœããããšã確èªããŠããŸãã
以äžã¯ããããµã€ãº=1ã®ãã³ãããŒã¯ãµã³ãã«ã§ãã ãã³ãããŒã¯ã¯ãåäžã® NVIDIA A100-SXM4-80GB GPU ã§å®è¡ããŸããã ããã³ããé·512ã䜿çšãã512åã®æ°ããããŒã¯ã³ãçæããŸãããæåã®è¡ã¯éååãããŠããªãfp16ããŒã¹ã©ã€ã³ã§ãããä»ã®è¡ã¯ããŸããŸãª AutoGPTQ ã«ãŒãã«ã䜿çšããã¡ã¢ãªæ¶è²»éãšããã©ãŒãã³ã¹ã瀺ããŠããŸãã

ããå æ¬çãªåçŸå¯èœãªãã³ãããŒã¯ã¯ãããããå ¥æã§ããŸãã
4. Optimumã䜿çšããã¢ãã«ã®éåå
ãAutoGPTQããtransformersã«ã·ãŒã ã¬ã¹ã«çµ±åããããã«ãåŠç¿ãšæšè«ã®æé©åã®ããã®HuggingFaceã®ããŒã«ãããã§ãããOptimumãã§å©çšã§ãã AutoGPTQ API ã®æå°éã®ããŒãžã§ã³ã䜿çšããŸããããã®ã¢ãããŒãã«åŸãããšã§ãtransformers ãšã®ç°¡åãªçµ±åãå®çŸããåæã«ãŠãŒã¶ãŒãç¬èªã®ã¢ãã«ãéååãããå Žåã« Optimum API ã䜿çšã§ããããã«ããŸããã
詳ããã¯ããã¥ã¡ã³ããåç
§ããŠãã ããã
GPTQã䜿çšããTransformersã¢ãã«ã®éååã¯ãæ°è¡ã§å®è¡ã§ããŸãã
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig
model_id = "facebook/opt-125m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
quantization_config = GPTQConfig(bits=4, dataset = "c4", tokenizer=tokenizer)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
quantization_config=quantization_config
)ã¢ãã«ã®éååã«ã¯é·ãæéããããå ŽåããããŸãã175Bã¢ãã«ã®å Žåãå€§èŠæš¡ãªããŒã¿ã»ãã (äŸ: `"c4"``) ã䜿çšããå Žåãå°ãªããšã 4 GPU æéãå¿ èŠã§ããããšã«æ³šæããŠãã ãããäžã§è¿°ã¹ãããã«ãå€ãã®GPTQã¢ãã«ã¯ãã§ã« HuggingFace Hub ã§å©çšã§ãããããã»ãšãã©ã®ãŠãŒã¹ã±ãŒã¹ã§ã¢ãã«ãèªåã§éååããå¿ èŠã¯ãããŸããã
5. TGIã«ããGPTQã¢ãã«ã®å®è¡
transformers ãžã®GPTQã®çµ±åãšäžŠè¡ããŠãéçšç°å¢ã§å€§èŠæš¡ãªèšèªã¢ãã«ãæäŸããããšãç®çãšããŠããText-Generation-Inferenceã (TGI) ã«GPTQã®ãµããŒãã远å ãããŸããã GPTQã¯ãå¹ åºãã¢ãŒããã¯ãã£ã§åçãããåŠçãããŒãž ã¢ãã³ã·ã§ã³ããã©ãã·ã¥ ã¢ãã³ã·ã§ã³ãªã©ã®æ©èœãšäœµçšã§ããããã«ãªããŸããã
ããšãã°ããã®çµ±åã«ãããåäžã® A100-80GB GPU ã§ 70B ã¢ãã«ãæäŸã§ããããã«ãªããŸããfp16 ãã§ãã¯ãã€ã³ãã䜿çšãããšãå©çšå¯èœãª GPU ã¡ã¢ãªãè¶ ãããããããã¯äžå¯èœã§ãã
詳ããã¯ããã¥ã¡ã³ããåç §ããŠãã ããã
6. PEFTã«ããéååã¢ãã«ã®ãã¡ã€ã³ãã¥ãŒãã³ã°
éåžžã®æ¹æ³ã§ã¯ãéååã¢ãã«ãããã«åŠç¿ããããšã¯ã§ããŸããã ãã ãããPEFTããæŽ»çšãããšãã¢ããã¿ãŒäžã§åŠç¿ã§ããŸãã ãããè¡ãã«ã¯ãéååã¢ãã«ã®ãã¹ãŠã®å±€ãããªãŒãºããåŠç¿å¯èœãªã¢ããã¿ãŒã远å ããŸãã GPTQã¢ãã«ã§PEFTã䜿çšããæ¹æ³ã®äŸãããã€ã瀺ããŸãã
ã»Colab ããŒãããã¯
ã»ãã¡ã€ã³ãã¥ãŒãã³ã°ã¹ã¯ãªãã
7. ãµããŒãã¢ãã«
ä»åã®çµ±åã§ã¯ããã³ãŒããŒãŸãã¯ãšã³ã³ãŒããŒã®ã¿ã®ã¢ãŒããã¯ãã£ãæã€LLMã®ã¿ããµããŒããããŸããããã¯å¶éçã«èããããããããŸãããããLlamaããOPTïœãGPT-NeoããGPT-NeoXããªã©ã®ã»ãšãã©ã®æå 端ã®LLMãå«ãŸããŸããããžã§ã³ããªãŒãã£ãªããã«ãã¢ãŒãã«ã¢ãã«ã¯çŸåšãµããŒããããŠããŸããã