
LlamaIndex v0.10 ã¯ã€ãã¯ã¹ã¿ãŒãã¬ã€ã - Pythonç
Pythonçã®ãLlamaIndexãã®ã¯ã€ãã¯ã¹ã¿ãŒãã¬ã€ãããŸãšããŸããã
ã»llama-index 0.10.4
1. LlamaIndex
ãLlamaIndexãã¯ããã©ã€ããŒãããã¡ã€ã³åºæã®ç¥èãå¿ èŠãšããå°éç¥èãå¿ èŠãšãã質åå¿çãã£ããããããç°¡åã«äœæã§ããã©ã€ãã©ãªã§ãã
v0.10 ã§ã¯å€§èŠæš¡ã¢ããããŒãããããŸããã
ã»llama-index-coreãå°å ¥ããIntegrationãåå¥ããã±ãŒãžã«åé¢
ã»ServiceContextã¯éæšå¥š
ã»å šIntegrationãLlamaHubã§ç®¡ç
2. LlamaIndexã®5ã€ã®ã¹ããŒãž
ãLlamaIndexãã«ã¯ã5ã€ã®ã¹ããŒãžããããŸãã

2-1. Loading
ããŒã¿ãœãŒã¹ (ããã¹ããã¡ã€ã«ãPDFãWebãµã€ããããŒã¿ããŒã¹ãAPIãªã©) ããããŒã¿ãèªã¿èŸŒã¿ãŸãã
ãLoadingãã®äž»èŠã³ã³ããŒãã³ãã¯ã次ã®ãšããã§ãã
ã»Document : ããŒã¿ãœãŒã¹ã®ã³ã³ãã
ã»Node : Documentãåå²ãããã®ããã£ã³ã¯ãšã¡ã¿ããŒã¿ãå«ãŸãã
ã»Connector : ããŒã¿ãœãŒã¹ããDocumentããã³Nodeãåã蟌ãã¢ãžã¥ãŒã«
2-2. Indexing
ããŒã¿ã®ã¯ãšãªãå¯èœã«ããããŒã¿æ§é ãã€ã³ããã¯ã¹ããäœæããŸãã
ãIndexingãã®äž»èŠã³ã³ããŒãã³ãã¯ã次ã®ãšããã§ãã
ã»Index : ããŒã¿ã®ã¯ãšãªãå¯èœã«ããããŒã¿æ§é
ã»Embedding : é¢é£ããŒã¿ãèŠã€ããããã®ãã¯ãã«è¡šçŸã«å€æããã¢ãã«
2-3. Storing
ã€ã³ããã¯ã¹ãäœæããåŸãã€ã³ããã¯ã¹ãšä»ã®ã¡ã¿ããŒã¿ãä¿åããããšã§ãåäœæããå¿ èŠããªããªããŸãã
2-4. Querying
ã€ã³ããã¯ã¹ããé¢é£ããŒã¿ãã¯ãšãªããŸãã
ãQueryingãã®äž»èŠã³ã³ããŒãã³ãã¯ã次ã®ãšããã§ãã
ã»Retriever : ã¯ãšãªæã«Indexããé¢é£ããŒã¿ãå¹ççã«ååŸããæ¹æ³ãå®çŸ©
ã»Node Postprocessor : ååŸããããŒãã»ãããåãåãããããã«å€æããã£ã«ã¿ãªã³ã°ããªã©ã³ãã³ã°ãé©çš
ã»Response Synthesizer : ãŠãŒã¶ãŒã¯ãšãªãšååŸããããã¹ããã£ã³ã¯ã䜿çšããŠã¬ã¹ãã³ã¹ãçæ
2-3. Evaluation
ã¯ãšãªã«å¯Ÿããã¬ã¹ãã³ã¹ãã©ã®çšåºŠæ£ç¢ºãå¿ å®ãè¿ éã§ãããã客芳çã«æž¬å®ããŸãã
3. ããã¥ã¡ã³ãã®æºå
ä»åã¯ããã³ã¬ããã£ã¢ã®ããŒã£ã¡ã»ãã»ãã£ãïŒãã®ããã¥ã¡ã³ããçšæããŸããã
ã»bocchi.txt

4. 質åå¿ç
Google Colabã§ã®è³ªåå¿çã®å®è¡æé ã¯ã次ã®ãšããã§ãã
(1) ããã±ãŒãžã®ã€ã³ã¹ããŒã«ã
# ããã±ãŒãžã®ã€ã³ã¹ããŒã«
!pip install llama-index==0.10.4ãllama-indexãã¯ã¹ã¿ãŒããã³ãã«ã§ã以äžã®ããã±ãŒãžãã€ã³ã¹ããŒã«ããŠããŸãã
ã»llama-index-core
ã»llama-index-legacy # temporarily included
ã»llama-index-llms-openai
ã»llama-index-embeddings-openai
ã»llama-index-program-openai
ã»llama-index-question-gen-openai
ã»llama-index-agent-openai
ã»llama-index-readers-file
ã»llama-index-multi-modal-llms-openai
(2) ç°å¢å€æ°ã®æºåã
巊端ã®éµã¢ã€ã³ã³ã§ãOPENAI_API_KEYããèšå®ããŠããã»ã«ãå®è¡ããŠãã ããã
# ç°å¢å€æ°ã®æºå (巊端ã®éµã¢ã€ã³ã³ã§OPENAI_API_KEYãèšå®)
import os
from google.colab import userdata
os.environ["OPENAI_API_KEY"] = userdata.get("OPENAI_API_KEY")(3) ãã°ã¬ãã«ã®èšå®ã
import logging
import sys
# ãã°ã¬ãã«ã®èšå®
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, force=True)ã»DEBUG : ãããã°æ å ±ã®åºå
ã»INFO : æ å ±ã®åºå (æ³å®éãã®äºè±¡ã®çºç)
ã»WARNING : èŠåã®åºå (æ³å®å€ã®äºè±¡ã®çºç)
ã»ERROR : ãšã©ãŒã®åºå (å®è¡ã®ç¶è¡äžèœ)
(4) Colabã«dataãã©ã«ããäœæããŠããã¥ã¡ã³ããé
眮ã
巊端ã®ãã©ã«ãã¢ã€ã³ã³ã§ãã¡ã€ã«äžèЧã衚瀺ããå³ã¯ãªãã¯ãæ°ãããã©ã«ããã§dataãã©ã«ããäœæããããã¥ã¡ã³ãããã©ãã°&ããããããŸãã

(5) ããã¥ã¡ã³ãã®èªã¿èŸŒã¿ã
from llama_index.core import SimpleDirectoryReader
# ããã¥ã¡ã³ãã®èªã¿èŸŒã¿
documents = SimpleDirectoryReader("data").load_data()(6) ã€ã³ããã¯ã¹ã®äœæã
ãã€ã³ããã¯ã¹ãã¯ãããŒã¿ã®ã¯ãšãªãå¯èœã«ããããŒã¿æ§é ãä¿æããã³ã³ããŒãã³ãã§ããããã¥ã¡ã³ãããã£ã³ã¯ã«åå²ãããã£ã³ã¯æ¯ã«åã蟌ã¿ã«å€æããŠä¿æããŸãã
ãã£ã³ã¯ã¯é¡äŒŒæ€çŽ¢ã®å¯Ÿè±¡ãšãªãããŒã¿åäœã«ãªããŸãã
from llama_index.core import VectorStoreIndex
# ã€ã³ããã¯ã¹ã®äœæ
index = VectorStoreIndex.from_documents(documents)DEBUG:llama_index.node_parser.node_utils:> Adding chunk: çµæãã³ã åŸè€ã²ãšãã¯åéãäœããªãé°ãã£ã§ãã€ãäžäººã§éãããŠããããäžåп代ã«ãã¬ãã®ã€...
DEBUG:llama_index.node_parser.node_utils:> Adding chunk: æåç¥ã©ã€ã å€äŒã¿ã«å
¥ããåŸè€ã²ãšãã¯ç¥ãåããå¢ããŠããããèªåããéã³ã«èªãããšãã§ãã...
DEBUG:llama_index.node_parser.node_utils:> Adding chunk: ã²ãããè
ãªãšãšã³ã¯ãçµæãã³ããã©ã€ãã«èŠããªãããã圌女ãã¡ã«ã¢ããã€ã¹ãéãããã¡ã³ãã¡...
:(7) ã¯ãšãªãšã³ãžã³ã®äœæã
ãã¯ãšãªãšã³ãžã³ãã¯ããŠãŒã¶ãŒå
¥å(ã¯ãšãª)ãšé¢é£ããæ
å ±ãã€ã³ããã¯ã¹ããååŸãããããããšã«å¿çãçæããã¢ãžã¥ãŒã«ã§ãã
# ã¯ãšãªãšã³ãžã³ã®äœæ
query_engine = index.as_query_engine()(8) 質åå¿çã
# 質åå¿ç
print(query_engine.query("ãŒã£ã¡ã¡ããã®åŸæãªæ¥œåšã¯ïŒ"))åŸè€ã²ãšãã®åŸæãªæ¥œåšã¯ã®ã¿ãŒã§ãã
ãã°ã確èªãããšã以äžã®ã³ã³ããã¹ããååŸããŠããããšãããããŸãã
[Similarity score: 0.832151] èªåã«ã¯äœã®åãæããªãã®ãçæããŠãããããäžåŠã®é ã«æãæ§æ Œã®äººéããã³ãããã£ãŠäººæ°è
ã«ãªã£ãã€ã³ã¿ãã¥ãŒãèªãã§ãã®ã¿ãŒãå§ãããæ¯æ¥ç·Žç¿ãããé°ã§ã®ã¿ãŒã®è
åã¯ããçŽã«ãªã£ãããçµå±ã...
[Similarity score: 0.829139] å€ããè
ã§äžäººã§ããã®ã奜ãã ããåŸè€ã²ãšããšéã£ãŠç¹ã«ã³ãã¥éãšããããã§ã¯ãªãã鳿¥œã«é¢ããŠã¯ç¬èªã®äŸ¡å€èгãæã£ãŠãããæµè¡ïŒã¯ãïŒãã«æµãããã®ãå«ãããã³ãã®åæ§ãéèŠããŠããããã®ãã...OpenAI APIã®éä¿¡å 容ã確èªã§ããŸããUnicodeãšã¹ã±ãŒãã·ãŒã±ã³ã¹ãæ¥æ¬èªã«æ»ããŠãŸãã
Context information is below.
---------------------
èªåã«ã¯äœã®åãæããªãã®ãçæããŠãããããäžåŠã®é ã«æãæ§æ Œã®äººéããã³ãããã£ãŠäººæ°è
ã«ãªã£ãã€ã³ã¿ãã¥ãŒãèªãã§ãã®ã¿ãŒãå§ãããæ¯æ¥ç·Žç¿ãããé°ã§ã®ã¿ãŒã®è
åã¯ããçŽã«ãªã£ãããçµå±ã...
å€ããè
ã§äžäººã§ããã®ã奜ãã ããåŸè€ã²ãšããšéã£ãŠç¹ã«ã³ãã¥éãšããããã§ã¯ãªãã鳿¥œã«é¢ããŠã¯ç¬èªã®äŸ¡å€èгãæã£ãŠãããæµè¡ïŒã¯ãïŒãã«æµãããã®ãå«ãããã³ãã®åæ§ãéèŠããŠããããã®ãã...
---------------------
Given the context information and not prior knowledge, answer the question. If the answer is not in the context, inform the user that you can't answer the question.
Question: ãŒã£ã¡ã¡ããã®åŸæãªæ¥œåšã¯ïŒ
Answer: 5. ã€ã³ããã¯ã¹ã®ä¿å
äœæããã€ã³ããã¯ã¹ã¯ä¿åããŠãããšã次åã«åäœæããå¿ èŠããªããªããŸãã
(1) ã€ã³ããã¯ã¹ã®ä¿åã
# ã€ã³ããã¯ã¹ã®ä¿å
index.storage_context.persist()(2) ã€ã³ããã¯ã¹ã®èªã¿èŸŒã¿ã
from llama_index.core import StorageContext, load_index_from_storage
# ã€ã³ããã¯ã¹ã®èªã¿èŸŒã¿
storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)6. ã«ã¹ã¿ãã€ãº
6-1. LLMã®èšå®
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
# LLMã®èšå®
Settings.llm = OpenAI(
model="gpt-3.5-turbo",
temperature=0.1
)LLMã®äž»ãªãã©ã¡ãŒã¿ã¯ã次ã®ãšããã§ãã
ã»model : ã¢ãã«å
ã»temperature : ã©ã³ãã ã
ã»max_tokens : çæããããŒã¯ã³ã®æå€§æ°
6-2. åã蟌ã¿ã®èšå®
from llama_index.core import Settings
from llama_index.embeddings.openai import OpenAIEmbedding
# åã蟌ã¿ã®èšå®
Settings.embed_model = OpenAIEmbedding(
model="text-embedding-3-small",
)Embeddingã®äž»ãªãã©ã¡ãŒã¿ã¯ã次ã®ãšããã§ãã
ã»model : ã¢ãã«å
ã»embed_batch_size : ããããµã€ãº (é床ãšã¡ã¢ãªã®ãã¬ãŒããªã)
ã»max_length : æå€§é·
6-3. ããã¹ãã¹ããªãã¿ãŒã®èšå®
from llama_index.core import Settings
from llama_index.core.node_parser import SentenceSplitter
import tiktoken
# ããã¹ãã¹ããªãã¿ãŒã®èšå®
Settings.text_splitter = SentenceSplitter(
separator=" ",
chunk_size=1024,
chunk_overlap=20,
paragraph_separator="\n\n\n",
secondary_chunking_regex="[^,.;ã]+[,.;ã]?",
tokenizer=tiktoken.encoding_for_model("gpt-3.5-turbo").encode
)SentenceSplitterã®ãã©ã¡ãŒã¿ã¯ã次ã®ãšããã§ãã
ã»chunk_size : åãã£ã³ã¯ã®ããŒã¯ã³æ° (default:1024)
ã»chunk_overlap : å岿ã®åãã£ã³ã¯ã®ããŒã¯ã³ã®éãªã (default:20)
ã»separator : åèªã®åºåãæå (default:" ")
ã»paragraph_separator : 段èœéã®åºåãæå (default:"\n\n\n")
ã»secondary_chunking_regex : æã®åºåãæåã®æ£èŠè¡šçŸ (default:"[^,.;ã]+[,.;ã]?")
ã»chunking_tokenizer_fn : ããã¹ããæç« ã«åå²ãã颿° (default:nltk.sent_tokenize)
ã»callback_manager : ã³ãŒã«ããã¯ãããŒãžã£
ã»tokenizer : ããŒã¯ãã€ã¶ãŒ
ãã£ã³ã¯ãµã€ãºãšãã£ã³ã¯ãªãŒããŒã©ããã¯ãããã¹ãã¹ããªãã¿ãŒãèšå®ãããåå¥ã«èšå®ããããšãå¯èœã§ãã
# ãã£ã³ã¯ãµã€ãºãšãã£ã³ã¯ãªãŒããŒã©ããã®èšå®
Settings.chunk_size = 512
Settings.chunk_overlap = 206-4. ããŒã¯ãã€ã¶ãŒã®èšå®
from llama_index.core import Settings
import tiktoken
# ããŒã¯ãã€ã¶ãŒã®èšå®
Settings.tokenizer = tiktoken.encoding_for_model("gpt-3.5-turbo").encode6-5. ã³ãŒã«ããã¯ã®èšå®
from llama_index.core.callbacks import TokenCountingHandler, CallbackManager
from llama_index.core import Settings
# ã³ãŒã«ããã¯ã®èšå®
token_counter = TokenCountingHandler()
Settings.callback_manager = CallbackManager([token_counter])6-6. ããã³ãããã«ããŒã®èšå®
éåžžãLLMã®å±æ§ã䜿çšããŠèªåçã«èšå®ãããŸãããç¹å¥ãªå Žåã¯äžæžãã§ããŸãã
from llama_index.core import Settings
# ããã³ãããã«ããŒã®èšå®
Settings.context_window = 4096
Settings.num_output = 2566-7. ã¯ãšãªãšã³ãžã³ã®èšå®
ãã¯ãã«æ€çŽ¢ã§ååŸãããã£ã³ã¯æ°ã¯ãã¯ãšãªãšã³ãžã³ã®similarity_top_kã§æå®ããŸãã
# ã¯ãšãªãšã³ãžã³ã®æºå
query_engine = index.as_query_engine(
similarity_top_k=10,
)6-8. Rerankerã®èšå®
ãRerankerãã¯ãã¯ãšãªãšã³ãžã³ã®node_postprocessorsã§èšå®ããŸãã
# ããã±ãŒãžã®ã€ã³ã¹ããŒã«
!pip install llama-index-postprocessor-flag-embedding-reranker
!pip install FlagEmbeddingfrom llama_index.postprocessor.flag_embedding_reranker import FlagEmbeddingReranker
# Rerankerã®æºå
rerank = FlagEmbeddingReranker(
model="BAAI/bge-reranker-v2-m3",
use_fp16=True,
top_n=5
)# ã¯ãšãªãšã³ãžã³ã®æºå
query_engine = index.as_query_engine(
similarity_top_k=10,
node_postprocessors=[rerank]
)6-9. ãã©ã³ã¹ãã©ãŒã¡ãŒã·ã§ã³ã®èšå®
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core import Settings
# ãã©ã³ã¹ãã©ãŒã¡ãŒã·ã§ã³ã®èšå®
Settings.transformations = [SentenceSplitter(chunk_size=1024)]7. LlamaHub
ãLLMããåã蟌ã¿ããããŒããŒããããŒã«ãããã¯ãã«ã¹ãã¢ããªã©ããLlamaIndexãã®ãIntegrationãã¯ãã¹ãŠãLlamaHubãã§ç®¡çããŸãã
ãã ããLlamaHubããžã®ç§»è¡ã¯ãŸã äœæ¥äžã§ãã æ°é±é以å
ã«ãµã€ããæŽæ°ããŸãããããŸã§ã®éã¯ãNotionããã±ãŒãžã¬ãžã¹ããªãå©çšããŸãã
ä»åã¯ããllama-index-readers-webãã®ãSimpleWebPageReaderãã䜿ã£ãŠãWebããŒãžã®æ å ±ã«åºã¥ããŠãLlamaIndexãã«è³ªåããŠã¿ãŸãã
(1) ããã±ãŒãžã®ã€ã³ã¹ããŒã«ã
# ããã±ãŒãžã®ã€ã³ã¹ããŒã«
!pip install llama-index-readers-web(2) ãSimpleWebPageReaderãã§URLããããã¥ã¡ã³ããçæã
Wikipediaã®ãŒã£ã¡ã»ãã»ãã£ãã®ããŒãžãèªã¿èŸŒã¿ãŸããã
from llama_index.readers.web import SimpleWebPageReader
# ãªãŒããŒã®æºå
reader = SimpleWebPageReader(html_to_text=True)
documents = reader.load_data(urls=["https://ja.wikipedia.org/wiki/%E3%81%BC%E3%81%A3%E3%81%A1%E3%83%BB%E3%81%96%E3%83%BB%E3%82%8D%E3%81%A3%E3%81%8F!"])(3) 質åå¿çãÂ
# 質åå¿ç
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("çµæãã³ããšã¯ïŒ")
print(response)çµæãã³ãã¯ãã²ãšããæå±ããŠãããã³ãã®ååã§ããã²ãšãã¯é«æ ¡çã®æã«çµæãã³ãã®ã®ã¿ãªã¹ããšããŠå å ¥ããŸãããçµæãã³ãã¯ãã²ãšããå«ãã¡ã³ããŒãã¡ãäžç·ã«é³æ¥œæŽ»åãè¡ã£ãŠãããã³ãã§ããã²ãšãã¯ãã³ã掻åãéããŠæé·ããã©ã€ãããŠã¹ãåŠæ ¡ã®æåç¥ãªã©ã§æŒå¥ãæ«é²ããŠããŸãã