ใƒกใ‚คใƒณใ‚ณใƒณใƒ†ใƒณใƒ„ใธใ‚นใ‚ญใƒƒใƒ—

FastChat-WebUIใจVicuna-13๏ฝ‚-๏ฝ–1.5ใ‚’ใƒญใƒผใ‚ซใƒซใงๅ‹•ใ‹ใ™

    n+m็•ช็…Žใ˜ใฎ่จ˜ไบ‹ใซใชใ‚Šใพใ™ใŒใ€ๅฐ‘ใ—่‹ฆๅŠดใ—ใŸใจใ“ใ‚ใŒใ‚ใ‚‹ใฎใงใƒกใƒขไปฃใ‚ใ‚Šใซ่จ˜ไบ‹ใ‚’ๆ›ธใใพใ—ใŸใ€‚ใƒญใƒผใ‚ซใƒซLLMใฏใƒใƒƒใƒˆใƒฏใƒผใ‚ฏใŒใชใใฆใ‚‚ๅ‹•ใใ€่ฒป็”จใ‚‚็™บ็”Ÿใ—ใชใ„ใฎใงAItuberใฎใ‚ˆใ†ใช้•ทๆ™‚้–“้€ฃ็ถšใƒฉใ‚คใƒ–้…ไฟกใ‚’ใ™ใ‚‹ใ‚ˆใ†ใชใจใใซใฏ้‡ๅฎใ—ใพใ™ใ€‚ไปŠๅ›žใฏ้žๅ•†็”จใฎ็ธ›ใ‚ŠใŒใชใใ€่ฆๆจกใฎๅคงใใชLLMใงใ‚ใ‚‹Vicuna-13b-v1.5ใ‚’ๅ‹•ใ‹ใ—ใพใ™ใ€‚8ๆœˆๅˆๆ—ฌใซ็™บ่กจใ•ใ‚ŒใŸๆฏ”่ผƒ็š„ๆ–ฐใ—ใ„LLMใงใ€ChatGPT-3.5ใซๅŒนๆ•ตใ™ใ‚‹ๆ€ง่ƒฝใŒใ‚ใ‚‹ใจ่จ€ใ‚ใ‚Œใฆใ„ใพใ™ใ€‚

    Vicunaใซใคใ„ใฆใฏใ€ใ™ใงใซnpakaใ•ใ‚“ใŒ่จ˜ไบ‹ใ‚’ๆ›ธใ‹ใ‚Œใฆใ„ใพใ™ใ—ใ€ไป–ใซใ‚‚่จ˜ไบ‹ใ‚’ๆ›ธใ„ใฆใ„ใ‚‰ใฃใ—ใ‚ƒใ‚‹ๆ–นใŒใ„ใ‚‰ใฃใ—ใ‚ƒใ‚‹ใฎใงๅ‚่€ƒใซใ•ใ›ใฆใ„ใŸใ ใใพใ—ใŸใ€‚

    weblog smart AI gadgets - Llama 2ใƒ™ใƒผใ‚นใฎLLM FastChat/Vicuna v1.5ใ‚’ใƒญใƒผใ‚ซใƒซใงๅ‹•ไฝœ (jweb.asia)

    ็’ฐๅขƒ
    CPU:ใ€€็ฌฌ3ไธ–ไปฃใ€€Coreใ€€i5
    MEM:ใ€€32G
    GPU:ใ€€Gfoece-4090
    OS:ย  ย  ย  Ubuntu20.04-jp

    ใพใšใฏใ€ใƒชใƒใ‚ธใƒˆใƒชใฎ่ชฌๆ˜Ž้€šใ‚Šใซใ‚คใƒณใ‚นใƒˆใƒผใƒซใ—ใพใ—ใŸใ€‚

    git clone https://github.com/lm-sys/FastChat.git
    cd FastChat
    pip install --upgrade pip  # enable PEP 660 support
    pip install -e ".[model_worker,webui]"

    ใƒชใƒใ‚ธใƒˆใƒชใฎ่กจใซใ‚ใ‚‹ใ‚ˆใ†ใซใƒขใƒ‡ใƒซใฏไฝ•็จฎ้กžใ‹ใ‚ใ‚Šใพใ™ใฎใงใ€่‡ชๅˆ†ใฎ็’ฐๅขƒใซๅˆใ†ใƒขใƒ‡ใƒซใ‚’้ธใ‚“ใงใ‚คใƒณใ‚นใƒˆใƒผใƒซใ‚ทใพใ™ใ€‚VRAMใŒ16Gใงใ‚ใ‚Œใฐใ€7Bใซใชใ‚Šใพใ™ใ€‚13Bใฏ24Gใงใ‚‚ๅ‹•ใ‹ใ›ใพใ›ใ‚“ใ€‚ไปŠๅ›žใฏ8bit้‡ๅญๅŒ–ใ‚’ใ™ใ‚‹ใ“ใจใซใ—ใฆ13Bใ‚’ๅ‹•ใ‹ใ™ใ“ใจใ‚’็›ฎๆŒ‡ใ—ใพใ™ใ€‚ใชใฎใงใ€33Bใƒขใƒ‡ใƒซใ‚’้™คใใ€ใฉใฎใƒขใƒ‡ใƒซใ‚‚VRAM-16Gใงๅ‹•ใ‹ใ™ใ“ใจใŒๅฏ่ƒฝใงใ™ใ€‚7Bใงใ™ใจ10Gใงๅ‹•ใใฏใšใงใ™ใ€‚

    ่กจใซใ‚ใ‚‹Chat Commanใ‚’ใ‚ณใƒ”ใƒšใ—ใฆๅ‹•ใ‹ใ—ใพใ™ใ€‚
    ๆ‰‹้ ƒใช๏ผ—b-v1.5ใฏไปฅไธ‹ใฎใ‚ณใƒžใƒณใƒ‰ใงใ™ใ€‚

    python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5

    ็งใฎ็’ฐๅขƒใงใฏ
    ModuleNotFoundError: No module named 'prompt_toolkit'
    ใจใ„ใ†ใ‚จใƒฉใƒผใŒๅ‡บใฆ้€ฒใฟใพใ›ใ‚“ใ€‚็†็”ฑใฏใ‚ˆใใ‚ใ‚‰ใชใ„ใฎใงใงใ™ใŒใ€prompt_toolkit'ใฏใ™ใงใซใ‚คใƒณใ‚นใƒˆใƒผใƒซใ•ใ‚ŒใฆใŠใ‚Šใ€ๅ•้กŒใฏ็„กใ„ใ‚ˆใ†ใซ่ฆ‹ใˆใพใ™ใ€‚ๅŽŸๅ› ่ฟฝๆฑ‚ใฏ้ขๅ€’ใชใฎใงใ€ไปฅไธ‹ใฎ่จ˜ไบ‹ใงใ‚„ใฃใŸ็’ฐๅขƒใ‚’ใ‚ฏใƒญใƒผใƒณใ—ใฆใ‚„ใ‚Š็›ดใ™ใ“ใจใซใ—ใพใ™ใ€‚๏ผˆ็‰นใซ็†็”ฑใฏใ‚ใ‚Šใพใ›ใ‚“ใ€‚MiniGPT4ใฎ็’ฐๅขƒใง่ฉฆใ—ใŸใ‚‰ใงใใŸใจใ„ใ†ใ“ใจใงใ™ใ€‚๏ผ‰

    ๆณจ๏ผ‰็Ÿฅใ‚Šๅˆใ„ใฏใ“ใ“ใงใ‚จใƒฉใƒผใฏๅ‡บใชใ‹ใฃใŸใจใ„ใฃใฆใพใ—ใŸใ€‚็งใฎ็’ฐๅขƒใŒไฝ•ใ‹็•ฐๅธธใŒใ‚ใ‚‹ใฎใ‹ใ‚‚ใ€‚

    ๅ‹•ใ‹ใ™ใซใฏใฉใ†ใ™ใ‚‹ใ‹

    ไฝ“็š„ใซใฏไปฅไธ‹ใฎ็’ฐๅขƒใ‚’ๅˆฉ็”จใ—ใพใ™ใ€‚

    git clone https://github.com/Vision-CAIR/MiniGPT-4.git
    cd MiniGPT-4
    conda env create -f environment.yml
    conda activate minigpt4

    ใ‚ฏใƒญใƒผใƒณใ—ใฆใ‚ขใ‚ฏใƒ†ใ‚ฃใƒ™ใƒผใƒˆ

    conda create -n fc4 --clone minigpt4
    conda actibate fc4

    ใใ—ใฆ FastChatใซ็งปๅ‹•ใ—ใ€ๅ†ๅบฆ็’ฐๅขƒๆง‹็ฏ‰ใ‚’ใ‚„ใ‚Š็›ดใ—ใพใ™ใ€‚

    cd ..
    cd FastChat
    pip install -e ".[model_worker,webui]"

    ๅ†ๅบฆใ‚ณใƒžใƒณใƒ‰ใ‚’ๅฎŸ่กŒ

    python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5

    ไปŠๅบฆใฏไธŠๆ‰‹ใใƒ€ใ‚ฆใƒณใƒญใƒผใƒ‰ใŒๅง‹ใพใ‚Šใ€่ตทๅ‹•ใงใใพใ—ใŸใ€‚ๆ—ฅๆœฌ่ชžใ‚’ๅ…ฅๅŠ›ใ™ใ‚‹ใจๅ›ž็ญ”ใ‚‚ๆ—ฅๆœฌ่ชžใงๅ‡บใฆใใพใ™ใ€‚

    (fc4) home:~/FastChat$ python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5
    
    ===================================BUG REPORT===================================
    Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
    ================================================================================
    Downloading (โ€ฆ)okenizer_config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 749/749 [00:00<00:00, 50.8kB/s]
    Downloading tokenizer.model: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 500k/500k [00:00<00:00, 20.1MB/s]
    Downloading (โ€ฆ)cial_tokens_map.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 438/438 [00:00<00:00, 217kB/s]
    Downloading (โ€ฆ)lve/main/config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 615/615 [00:00<00:00, 137kB/s]
    Downloading (โ€ฆ)model.bin.index.json: 100%|โ–ˆ| 26.8k/26.8k [00:00<00:00, 7.77MB/s]
    Downloading (โ€ฆ)l-00001-of-00002.bin: 100%|โ–ˆ| 9.98G/9.98G [01:48<00:00, 92.1MB/s]
    Downloading (โ€ฆ)l-00002-of-00002.bin: 100%|โ–ˆ| 3.50G/3.50G [00:42<00:00, 81.8MB/s]
    Downloading shards: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 2/2 [02:31<00:00, 75.81s/it]
    Loading checkpoint shards: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 2/2 [00:03<00:00,  1.91s/it]
    Downloading (โ€ฆ)neration_config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 162/162 [00:00<00:00, 14.7kB/s]
    USER: ใ“ใ‚“ ใซใกใฏ
    ASSISTANT: ใ“ใ‚“ใซใกใฏ๏ผไฝ•ใ‹ใŠๆ‰‹ไผใ„ใงใใ‚‹ใ“ใจใŒใ‚ใ‚ŒใฐใŠ็Ÿฅใ‚‰ใ›ใใ ใ•ใ„ใ€‚
    USER: 

    GPUใฎๅˆฉ็”จ็Šถๆณใ‚’่ฆ‹ใพใ™ใ€‚

    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  NVIDIA GeForce ...  On   | 00000000:01:00.0  On |                  Off |
    |  0%   55C    P8    34W / 450W |  14700MiB / 24564MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
                                                                                   
    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |    0   N/A  N/A      1142      G   /usr/lib/xorg/Xorg                102MiB |
    |    0   N/A  N/A      1814      G   /usr/lib/xorg/Xorg                251MiB |
    |    0   N/A  N/A      1962      G   /usr/bin/gnome-shell              100MiB |
    |    0   N/A  N/A      5757      G   /usr/lib/firefox/firefox          499MiB |
    |    0   N/A  N/A      8968      C   python3                         13724MiB |
    +-----------------------------------------------------------------------------+
    

    8bit้‡ๅญๅŒ–ใ‚ชใƒ—ใ‚ทใƒงใƒณใ‚’ไฝฟใฃใฆใ„ใชใ„ใฎใงใ€14.7Gใ‚‚ๅฐ‚ๆœ‰ใ—ใฆใ„ใพใ™ใ€‚

    fastchat.serve.clใ€€ใฏใ€€--helpใ‚ชใƒ—ใ‚ทใƒงใƒณใงใ‚ชใƒ—ใ‚ทใƒงใƒณใŒ่กจ็คบใ•ใ‚Œใพใ™ใ€‚

    python3 -m fastchat.serve.cli --help
    
    ===================================BUG REPORT===================================
    Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
    ================================================================================
    usage: cli.py [-h] [--model-path MODEL_PATH] [--revision REVISION]
                  [--device {cpu,cuda,mps,xpu}] [--gpus GPUS]
                  [--num-gpus NUM_GPUS] [--max-gpu-memory MAX_GPU_MEMORY]
                  [--load-8bit] [--cpu-offloading] [--gptq-ckpt GPTQ_CKPT]
                  [--gptq-wbits {2,3,4,8,16}] [--gptq-groupsize GPTQ_GROUPSIZE]
                  [--gptq-act-order] [--awq-ckpt AWQ_CKPT] [--awq-wbits {4,16}]
                  [--awq-groupsize AWQ_GROUPSIZE] [--conv-template CONV_TEMPLATE]
                  [--conv-system-msg CONV_SYSTEM_MSG] [--temperature TEMPERATURE]
                  [--repetition_penalty REPETITION_PENALTY]
                  [--max-new-tokens MAX_NEW_TOKENS] [--no-history]
                  [--style {simple,rich,programmatic}] [--multiline] [--mouse]
                  [--judge-sent-end] [--debug]
    
    optional arguments:
      -h, --help            show this help message and exit
      --model-path MODEL_PATH
                            The path to the weights. This can be a local folder or
                            a Hugging Face repo ID.
      --revision REVISION   Hugging Face Hub model revision identifier
      --device {cpu,cuda,mps,xpu}
                            The device type
      --gpus GPUS           A single GPU like 1 or multiple GPUs like 0,2
      --num-gpus NUM_GPUS
      --max-gpu-memory MAX_GPU_MEMORY
                            The maximum memory per GPU for storing model weights.
                            Use a string like '13Gib'
      --load-8bit           Use 8-bit quantization
      --cpu-offloading      Only when using 8-bit quantization: Offload excess
                            weights to the CPU that don't fit on the GPU
      --gptq-ckpt GPTQ_CKPT
                            Used for GPTQ. The path to the local GPTQ checkpoint.
      --gptq-wbits {2,3,4,8,16}
                            Used for GPTQ. #bits to use for quantization
      --gptq-groupsize GPTQ_GROUPSIZE
                            Used for GPTQ. Groupsize to use for quantization;
                            default uses full row.
      --gptq-act-order      Used for GPTQ. Whether to apply the activation order
                            GPTQ heuristic
      --awq-ckpt AWQ_CKPT   Used for AWQ. Load quantized model. The path to the
                            local AWQ checkpoint.
      --awq-wbits {4,16}    Used for AWQ. #bits to use for AWQ quantization
      --awq-groupsize AWQ_GROUPSIZE
                            Used for AWQ. Groupsize to use for AWQ quantization;
                            default uses full row.
      --conv-template CONV_TEMPLATE
                            Conversation prompt template.
      --conv-system-msg CONV_SYSTEM_MSG
                            Conversation system message.
      --temperature TEMPERATURE
      --repetition_penalty REPETITION_PENALTY
      --max-new-tokens MAX_NEW_TOKENS
      --no-history
      --style {simple,rich,programmatic}
                            Display style.
      --multiline           Enable multiline input. Use ESC+Enter for newline.
      --mouse               [Rich Style]: Enable mouse support for cursor
                            positioning.
      --judge-sent-end      Whether enable the correction logic that interrupts
                            the output of sentences due to EOS.
      --debug               Print useful debug information (e.g., prompts)
    

    ใ“ใ“ใซใ‚ใ‚‹ใ€€--load-8bit ใ€€ใ‚ชใƒ—ใ‚ทใƒงใƒณใ‚’ไฝฟใ†ใจ8bit้‡ๅญๅŒ–ใŒใงใใพใ™ใ€‚

    python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --load-8bit 

    ใƒขใƒ‡ใƒซใฏใƒ€ใ‚ฆใƒณใƒญใƒผใƒ‰ๆธˆใฟใชใฎใงใ€ใ™ใใซ็ซ‹ใกไธŠใŒใ‚Šใพใ™ใ€‚

    python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --load-8bit 
    
    ===================================BUG REPORT===================================
    Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
    ================================================================================
    100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 2/2 [00:24<00:00, 12.43s/it]
    USER: ใ“ใ‚“ใซใกใฏ
    ASSISTANT: ใ“ใ‚“ใซใกใฏ๏ผไฝ•ใ‹ใŠๆ‰‹ไผใ„ใงใใ‚‹ใ“ใจใŒใ‚ใ‚ŒใฐใŠ็Ÿฅใ‚‰ใ›ใใ ใ•ใ„ใ€‚
    USER: 

    GPUๅˆฉ็”จ็Šถๆณใงใ™ใ€‚

    | NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  NVIDIA GeForce ...  On   | 00000000:01:00.0  On |                  Off |
    | 30%   46C    P8    34W / 450W |   9112MiB / 24564MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
                                                                                   
    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |    0   N/A  N/A      1142      G   /usr/lib/xorg/Xorg                102MiB |
    |    0   N/A  N/A      1814      G   /usr/lib/xorg/Xorg                257MiB |
    |    0   N/A  N/A      1962      G   /usr/bin/gnome-shell              100MiB |
    |    0   N/A  N/A      5757      G   /usr/lib/firefox/firefox          444MiB |
    |    0   N/A  N/A     11220      C   python3                          8186MiB |
    +-----------------------------------------------------------------------------+

    9.1Gใซๆธ›ใ‚Šใพใ—ใŸใ€‚10GใฎVRAMใ‚’ๆŒใคGPUใงใ‚‚ๅ‹•ใใฎใงใฏใชใ„ใงใ—ใ‚‡ใ†ใ‹ใ€‚

    Vicna-13b-v1.5ใ‚’๏ผ˜bit้‡ๅญๅŒ–ใง่ฉฆใ™

    ๅŒๆง˜ใซ13b-v1.5ใ‚‚่ฉฆใ—ใพใ—ใ‚‡ใ†ใ€‚ --load-8bitใ‚ชใƒ—ใ‚ทใƒงใƒณใ‚’ใคใ‘ใพใ™ใ€‚

    python3 -m fastchat.serve.cli --model-path lmsys/vicuna-13b-v1.5  --load-8bit

    ๅคงใใชใƒขใƒ‡ใƒซใชใฎใงใƒ€ใ‚ฆใƒณใƒญใƒผใƒ‰ใซๆ™‚้–“ใŒใ‹ใ‹ใ‚Šใพใ™ใ€‚ใƒˆใƒผใ‚ฟใƒซใง50Gbyteๅผฑใงใ™ใญใ€‚ใ“ใ‚“ใชๆ„Ÿใ˜ใงใ™ใ€‚

    python3 -m fastchat.serve.cli --model-path lmsys/vicuna-13b-v1.5  --load-8bit
    
    ===================================BUG REPORT===================================
    Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
    ================================================================================
    Downloading (โ€ฆ)okenizer_config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 749/749 [00:00<00:00, 49.3kB/s]
    Downloading tokenizer.model: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 500k/500k [00:00<00:00, 14.4MB/s]
    Downloading (โ€ฆ)cial_tokens_map.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 438/438 [00:00<00:00, 1.03MB/s]
    Downloading (โ€ฆ)lve/main/config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 638/638 [00:00<00:00, 65.7kB/s]
    Downloading (โ€ฆ)df14bcd3a6/README.md: 100%|โ–ˆโ–ˆ| 1.97k/1.97k [00:00<00:00, 814kB/s]
    Downloading (โ€ฆ)model.bin.index.json: 100%|โ–ˆ| 33.4k/33.4k [00:00<00:00, 12.8MB/s]
    Downloading (โ€ฆ)cd3a6/.gitattributes: 100%|โ–ˆโ–ˆ| 1.52k/1.52k [00:00<00:00, 637kB/s]
    Downloading (โ€ฆ)neration_config.json: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 192/192 [00:00<00:00, 78.9kB/s]
    Downloading (โ€ฆ)l-00003-of-00003.bin: 100%|โ–ˆ| 6.18G/6.18G [03:15<00:00, 31.5MB/s]
    Downloading (โ€ฆ)l-00001-of-00003.bin: 100%|โ–ˆ| 9.95G/9.95G [03:39<00:00, 45.3MB/s]
    Downloading (โ€ฆ)l-00002-of-00003.bin: 100%|โ–ˆ| 9.90G/9.90G [03:50<00:00, 42.9MB/s]
    Fetching 11 files: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 11/11 [03:51<00:00, 21.04s/it]
    100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 3/3 [00:52<00:00, 17.35s/it]
    USER: ใ“ใ‚“ใฐใ‚“ใฏ-00002-of-00003.bin: 100%|โ–‰| 9.90G/9.90G [03:50<00:00, 71.8MB/s]
    ASSISTANT: ใ“ใ‚“ใซใกใฏ๏ผใฉใฎใ‚ˆใ†ใซใŠๅŠ›ใซใชใ‚Œใพใ™ใ‹๏ผŸ
    USER:

    ๏ผงPUไฝฟ็”จ้‡

    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  NVIDIA GeForce ...  On   | 00000000:01:00.0  On |                  Off |
    |  0%   54C    P8    33W / 450W |  15081MiB / 24564MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
                                                                                   
    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |    0   N/A  N/A      1142      G   /usr/lib/xorg/Xorg                102MiB |
    |    0   N/A  N/A      1814      G   /usr/lib/xorg/Xorg                257MiB |
    |    0   N/A  N/A      1962      G   /usr/bin/gnome-shell               99MiB |
    |    0   N/A  N/A      5757      G   /usr/lib/firefox/firefox          438MiB |
    |    0   N/A  N/A     11287      C   python3                         14162MiB |
    +-----------------------------------------------------------------------------+

    15Gใชใฎใง4060-16Gใ‚„4080ใงใ‚‚ๅ‹•ใใฏใšใงใ™ใ€‚

    FastChat web UI ใ‚’ๅ‹•ใ‹ใ™

    ๆœ€ๆœŸใซGUIใ‚’ๅ‹•ใ‹ใ—ใพใ™ใ€‚gradioใชใฎใงใ€APIใŒๅˆฉ็”จใงใใ‚‹ใ“ใจใ‚’ๆœŸๅพ…ใ—ใŸใ„ใจใ“ใ‚ใงใ™ใ€‚ใƒชใƒใ‚ธใƒˆใƒชใซใฏๅ‹•ใ‹ใ—ๆ–นใŒๆ›ธใ„ใฆใ‚ใ‚Šใพใ™ใ€‚
    To serve using the web UI, you need three main components: web servers that interface with users, model workers that host one or more models, and a controller to coordinate the webserver and model workers. You can learn more about the architecture here.
    ใ“ใ“ใ‚’ใ‚ˆใ่ชญใพใชใ‹ใฃใŸใฎใงๆœ€ๅˆใฏไธŠๆ‰‹ใใงใใพใ›ใ‚“ใงใ—ใŸใ€‚่ฆใ™ใ‚‹ใซไปฅไธ‹ใฎ3็จฎ้กžใฎใ‚ตใƒผใƒใŒใ„ใ‚‹ใจใ„ใ†ใ“ใจใงใ™ใ€‚
    -web servers that interface with users
    -model workers that host one or more models
    -controller to coordinate the webserver and model workers
    ใ“ใฎ3็จฎ้กžใฎใ‚ตใƒผใƒใ‚’็‹ฌ็ซ‹ใ—ใฆๅ‹•ใ‹ใ™ๅฟ…่ฆใŒใ‚ใ‚Šใพใ™ใ€‚
    ใ‚ฟใƒผใƒŸใƒŠใƒซใ‚’3ใค้–‹ใใ€ไปฅไธ‹ใฎใ‚ˆใ†ใซๅ…จใฆFastChatใซ็งปๅ‹•ใ—ใฆfc4็’ฐๅขƒใ‚’ๆœ‰ๅŠนใซใ—ใพใ™ใ€‚

    cd FastChat
    conda activate fc4

    ใพใšใ€controllerใ‚’ๅ‹•ใ‹ใ—ใพใ™ใ€‚

    python3 -m fastchat.serve.controller
    
    2023-08-22 18:21:13 | INFO | controller | args: Namespace(host='localhost', port=21001, dispatch_method='shortest_queue')
    2023-08-22 18:21:13 | ERROR | stderr | INFO:     Started server process [11479]

    ๆฌกใซๅˆฅใฎใ‚ฟใƒผใƒŸใƒŠใƒซใงใ€model worker(s)ใ‚’ๅ‹•ใ‹ใ—ใพใ™ใ€‚่ค‡ๆ•ฐๅ‹•ใ‹ใ™ใจGUIใง้ธๆŠžใงใใพใ™ใ€‚--load-8bitใ‚’ๅฟ˜ใ‚Œใ‚‹ใจ4090ใงใ‚‚13bใฏๅ‹•ใใพใ›ใ‚“ใ€‚

    python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-13b-v1.5  --load-8bit
    
    2023-08-22 18:25:20 | INFO | model_worker | args: Namespace(host='localhost', port=21002, worker_address='http://localhost:21002', controller_address='http://localhost:21001', model_path='lmsys/vicuna-13b-v1.5', revision='main', device='cuda', gpus=None, num_gpus=1, max_gpu_memory=None, load_8bit=True, cpu_offloading=False, gptq_ckpt=None, gptq_wbits=16, gptq_groupsize=-1, gptq_act_order=False, awq_ckpt=None, awq_wbits=16, awq_groupsize=-1, model_names=None, conv_template=None, embed_in_truncate=False, limit_worker_concurrency=5, stream_interval=2, no_register=False)
    2023-08-22 18:25:20 | INFO | model_worker | Loading the model ['vicuna-13b-v1.5'] on worker 2a851d14 ...
      0%|                                                             | 0/3 [00:00<?, ?it/s]
     33%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‹                                   | 1/3 [00:12<00:24, 12.29s/it]
     67%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–Ž                 | 2/3 [00:32<00:17, 17.10s/it]
    100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 3/3 [00:49<00:00, 17.08s/it]
    100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 3/3 [00:49<00:00, 16.61s/it]
    2023-08-22 18:26:10 | ERROR | stderr | 
    2023-08-22 18:26:10 | INFO | model_worker | Register to controller
    2023-08-22 18:26:10 | ERROR | stderr | INFO:     Started server process [11543]
    2023-08-22 18:26:10 | ERROR | stderr | INFO:     Waiting for application startup.
    2023-08-22 18:26:10 | ERROR | stderr | INFO:     Application startup complete.
    2023-08-22 18:26:10 | ERROR | stderr | INFO:     Uvicorn running on http://localhost:21002 (Press CTRL+C to quit)

    ๆœ€ๆœŸใซGradio web serverใ‚’ๅ‹•ใ‹ใ—ใพใ™ใ€‚ๅˆฅใฎใ‚ฟใƒผใƒŸใƒŠใƒซใ‹ใ‚‰ไปฅไธ‹ใ‚’ๅ…ฅๅŠ›

    python3 -m fastchat.serve.gradio_web_server
    
    2023-08-22 18:30:15 | INFO | gradio_web_server | args: Namespace(host='0.0.0.0', port=None, share=False, controller_url='http://localhost:21001', concurrency_count=10, model_list_mode='once', moderate=False, add_chatgpt=False, add_claude=False, add_palm=False, gradio_auth_path=None)
    2023-08-22 18:30:15 | INFO | gradio_web_server | Models: ['vicuna-13b-v1.5']
    2023-08-22 18:30:16 | ERROR | stderr | /home/animede/anaconda3/envs/fc4/lib/python3.9/site-packages/gradio/deprecation.py:43: UserWarning: You have unused kwarg parameters in Dropdown, please remove them: {'container': False}
    2023-08-22 18:30:16 | ERROR | stderr |   warnings.warn(
    2023-08-22 18:30:16 | ERROR | stderr | /home/animede/anaconda3/envs/fc4/lib/python3.9/site-packages/gradio/deprecation.py:40: UserWarning: `height` is deprecated in `Interface()`, please use it within `launch()` instead.
    2023-08-22 18:30:16 | ERROR | stderr |   warnings.warn(value)
    2023-08-22 18:30:16 | ERROR | stderr | /home/animede/anaconda3/envs/fc4/lib/python3.9/site-packages/gradio/deprecation.py:43: UserWarning: You have unused kwarg parameters in Textbox, please remove them: {'container': False}
    2023-08-22 18:30:16 | ERROR | stderr |   warnings.warn(
    2023-08-22 18:30:17 | INFO | stdout | Running on local URL:  http://0.0.0.0:7860
    2023-08-22 18:30:17 | INFO | stdout | 
    2023-08-22 18:30:17 | INFO | stdout | To create a public link, set `share=True` 

    Gradioใฎใ„ใคใ‚‚ใฎใจใŠใ‚Šใซใ€ไปฅไธ‹ใ‚’ใ‚ขใ‚ฏใ‚ปใ‚นใ—ใพใ™ใ€‚
    http://0.0.0.0:7860
    GPUใฎๅˆฉ็”จ็Šถๆณใฏใ‚ณใƒžใƒณใƒ‰ใƒฉใ‚คใƒณใฎๆ™‚ใจๅŒใ˜ใงใ™ใ€‚

    ็”ปๅƒ
    FastChatใ€€WEB UIใฎ็”ป้ข

    ๅ…ฅๅŠ›ใซๅฏพใ—ใฆๅ‡บๅŠ›ใฏๆ—ฉใใ€ใปใจใ‚“ใฉใฎๅ…ฅๅŠ›ใซๅฏพใ—ใฆ็žฌๆ™‚ใงๅ‡บใฏใ˜ใ‚ใพใ™ใ€‚

    ไพ‹๏ผ‰
    ไป•ไบ‹ใงๅคงไบ‹ใชใ“ใจใฏใชใซใงใ™ใ‹๏ผŸ็ฎ‡ๆกๆ›ธใใงใŠ้ก˜ใ„ใ—ใพใ™ใ€‚

    ไป•ไบ‹ใงๅคงไบ‹ใชใ“ใจใฏไปฅไธ‹ใฎ้€šใ‚Šใงใ™ใ€‚
    1. ใ‚ฟใ‚นใ‚ฏใฎๅ„ชๅ…ˆ้ †ไฝไป˜ใ‘: ใ‚ฟใ‚นใ‚ฏใฎ้‡่ฆๅบฆใ‚„็ท ใ‚Šๅˆ‡ใ‚Šใ‚’่€ƒๆ…ฎใ—ใฆใ€ๅ„ชๅ…ˆ้ †ไฝใ‚’ไป˜ใ‘ใ‚‹ใ“ใจใŒ้‡่ฆใงใ™ใ€‚
    2. ใ‚ณใƒŸใƒฅใƒ‹ใ‚ฑใƒผใ‚ทใƒงใƒณ: ใƒใƒผใƒ ใ‚„ไธŠๅธใจใฎใ‚ณใƒŸใƒฅใƒ‹ใ‚ฑใƒผใ‚ทใƒงใƒณใ‚’ๅ††ๆป‘ใซใ™ใ‚‹ใ“ใจใงใ€ไป•ไบ‹ใฎ้€ฒๆ—ใ‚„ๅ•้กŒ็‚นใ‚’ๆ—ฉๆœŸใซ่งฃๆฑบใ™ใ‚‹ใ“ใจใŒใงใใพใ™ใ€‚
    3. ๆ™‚้–“็ฎก็†: ่‡ชๅˆ†ใฎๆ™‚้–“ใ‚’ๅŠน็އ็š„ใซไฝฟใ„ใ€็ดๆœŸใ‚’ๅฎˆใ‚‹ใ“ใจใŒๅคงๅˆ‡ใงใ™ใ€‚
    4. ็ŠถๆณใฎๆŠŠๆก: ไป•ไบ‹ใฎ้€ฒๆ—็Šถๆณใ‚„ๅ•้กŒ็‚นใ‚’ๆŠŠๆกใ—ใ€่‡ชๅˆ†่‡ช่บซใ‚„ใƒใƒผใƒ ใฎ่ฒฌไปป็ฏ„ๅ›ฒใ‚’ๆ˜Ž็ขบใซใ™ใ‚‹ใ“ใจใŒๅฟ…่ฆใงใ™ใ€‚
    5. ็ถ™็ถš็š„ใชๅญฆ็ฟ’: ๆ–ฐใ—ใ„็Ÿฅ่ญ˜ใ‚„ใ‚นใ‚ญใƒซใ‚’็ฟ’ๅพ—ใ—ใ€่‡ชๅˆ†่‡ช่บซใ‚’็ถ™็ถš็š„ใซๅ‘ไธŠใ•ใ›ใ‚‹ใ“ใจใŒๅคงๅˆ‡ใงใ™ใ€‚
    6. ่‡ชๅทฑ็ฎก็†: ใ‚นใƒˆใƒฌใ‚นใ‚„็–ฒๅŠดใ‚’่ปฝๆธ›ใ™ใ‚‹ใŸใ‚ใซใ€่‡ชๅˆ†่‡ช่บซใ‚’็ฎก็†ใ™ใ‚‹ใ“ใจใŒ้‡่ฆใงใ™ใ€‚
    7. ็›ฎๆจ™ใฎ่จญๅฎš: ่‡ชๅˆ†่‡ช่บซใฎใ‚ญใƒฃใƒชใ‚ขใ‚„ไป•ไบ‹ใซๅฏพใ™ใ‚‹็›ฎๆจ™ใ‚’่จญๅฎšใ—ใ€ใใ‚Œใ‚’้”ๆˆใ™ใ‚‹ใŸใ‚ใฎ่จˆ็”ปใ‚’็ซ‹ใฆใ‚‹ใ“ใจใŒๅฟ…่ฆใงใ™ใ€‚
    ใ“ใ‚Œใ‚‰ใฎใƒใ‚คใƒณใƒˆใ‚’ๆŠผใ•ใˆใ‚‹ใ“ใจใงใ€ไป•ไบ‹ใซใŠใ„ใฆๆˆๆžœใ‚’ๅ‡บใ—ใ‚„ใ™ใใชใ‚Š

    ใพใจใ‚

    ็™บ่กจใ‹ใ‚‰20ๆ—ฅใ‚‚็ตŒ้Žใ—ใŸใƒขใƒ‡ใƒซใงใ™ใŒใ€็พๆ™‚็‚นใงใ‚ชใƒผใƒ—ใƒณใ‚ฝใƒผใ‚นใฎไธญใงใฏๆœ€ใ‚‚ๆ€ง่ƒฝใŒ้ซ˜ใ„ใฎใงใฏใชใ„ใ‹ใจๆ€ใ„ใพใ™ใ€‚FastChatใฎ้ซ˜้€Ÿๅฟœ็ญ”ใ‚‚ใชใ‹ใชใ‹ใฎใ‚‚ใฎใงใ™ใ€‚

    ย 
    ย 
    AItuberใ‚ใใจใใฎใŠๅ‹้”ใ‚„ใ€้–ข้€ฃใ™ใ‚‹AIๆŠ€่ก“ใฎ่จ˜ไบ‹ใ‚’ไธญๅฟƒใซๆ›ธใ„ใฆใ„ใพใ™ใ€‚

    ใ‚ใชใŸใธใฎใŠใ™ใ™ใ‚