
LlamaIndex ã® RecursiveRetriever ã詊ã
ãLlamaIndexãã®ãRecursiveRetrieverãã詊ããã®ã§ããŸãšããŸããã
1. RecursiveRetriever
ãRecursiveRetrieverãã¯ãéå±€çãªææžè¡šçŸ (ãã¹ãããã衚/å³ãå«ãPDFãªã©) ã«å¯ŸããŠååž°çæ€çŽ¢ãå®è¡ã§ãããªããªããŒã§ããææžå ã®éå±€æ§é ãæŽ»çšããããšã§ãè€éãªææžã®RAGã·ã¹ãã ã®æ§èœãåäžãããããšãã§ããŸãã
ãŠãŒã¹ã±ãŒã¹ã¯ã次ã®ãšããã§ãã
(1) PDFå ã«åã蟌ãŸãã衚ã»ç»åãšæ®ãã®ããã¹ãã®ã¢ããªã³ã°
(2) 倿°ã®ç°è³ªãªææžã®ã¢ããªã³ã° - æåã«èŠçŽã«ãã£ãŠã€ã³ããã¯ã¹ãäœæãããã®å 容ã«ãªã³ã¯ã
(3) è€éãªããŒã¿ã«å¯Ÿããæ€çŽ¢ç²ŸåºŠã®åäž - åæã®ããã®ããã¹ããã£ã³ã¯ããæ€çŽ¢æ¹æ³ïŒããå°ããªãã£ã³ã¯ãã¡ã¿ããŒã¿ãèŠçŽãªã©ïŒããåãé¢ããããè¯ãæ€çŽ¢ãå®çŸã
(4) ã³ãŒããããã¯ã®ã¢ããªã³ã° + ASTã®ãã©ããŒã¹
ã¯ãšãªå¯Ÿè±¡ãšãªãããŒããªã¹ãã«ããããã¹ããã£ã³ã¯ãã ãã§ãªããã€ã³ããã¯ã¹ãžã®ãªã³ã¯ããå«ããããšã§ãéå±€æ§é ã衚çŸããŠããŸãã

2. ã»ããã¢ãã
Colabã§ã®ã»ããã¢ããæé ã¯ã次ã®ãšããã§ãã
(1) ããã±ãŒãžã®ã€ã³ã¹ããŒã«ã
# ããã±ãŒãžã®ã€ã³ã¹ããŒã«
!pip install llama-index(2) ç°å¢å€æ°ã®æºåã
以äžã®ã³ãŒãã® <OpenAI_APIã®ããŒã¯ã³> ã«ã¯OpenAI APIã®ããŒã¯ã³ãæå®ããŸãã(ææ)
import os
# ç°å¢å€æ°ã®èšå®
openai.api_key = "<OpenAI_APIã®ããŒã¯ã³>"(3) ãã°ã¬ãã«ã®èšå®ã
import logging
import sys
# ãã°ã¬ãã«ã®èšå®
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, force=True)3. ããã¥ã¡ã³ãã®æºå
Colabã§ã®ããã¥ã¡ã³ãã®æºåæé ã¯ã次ã®ãšããã§ãã
(1) Wikipediaããã®ããŒã¿èªã¿èŸŒã¿ã
Wikipediaãããéæ³å°å¥³ãŸã©ãâãã®ã«ãããŒã£ã¡ã»ãã»ãã£ã!ãã®æ
å ±ãååŸããããã¹ããã¡ã€ã«ã«ä¿åããŸãã
from pathlib import Path
import requests
# Wikipediaããã®ããŒã¿èªã¿èŸŒã¿
wiki_titles = ["éæ³å°å¥³ãŸã©ãâãã®ã«", "ãŒã£ã¡ã»ãã»ãã£ã!"]
for title in wiki_titles:
response = requests.get(
"https://ja.wikipedia.org/w/api.php",
params={
"action": "query",
"format": "json",
"titles": title,
"prop": "extracts",
# 'exintro': True,
"explaintext": True,
},
).json()
page = next(iter(response["query"]["pages"].values()))
wiki_text = page["extract"]
data_path = Path("data")
if not data_path.exists():
Path.mkdir(data_path)
with open(data_path / f"{title}.txt", "w") as fp:
fp.write(wiki_text)dataãã©ã«ãã«ãéæ³å°å¥³ãŸã©ãâãã®ã«.txtãããŒã£ã¡ã»ãã»ãã£ã!.txtããåºåãããŸãã

(2) ããã¥ã¡ã³ãã®èªã¿èŸŒã¿ã
from llama_index import SimpleDirectoryReader
# ããã¥ã¡ã³ãã®èªã¿èŸŒã¿
anime_docs = []
for wiki_title in wiki_titles:
docs = SimpleDirectoryReader(input_files=[f"data/{wiki_title}.txt"]).load_data()
docs[0].doc_id = wiki_title
anime_docs.extend(docs)4. åéå±€ã®ã¯ãšãªãšã³ãžã³ã®æºå
Colabã§ã®åéå±€ã®ã¯ãšãªãšã³ãžã³ã®æºåæé ã¯ã次ã®ãšããã§ãã
(1) QAãã³ãã¬ãŒãã®æºåã
LlamaIndexã®è±èªçã®QAããã³ããã®çŽèš³ã«ãªããŸãã
from llama_index.llms.base import ChatMessage, MessageRole
from llama_index.prompts.base import ChatPromptTemplate
# QAã·ã¹ãã ããã³ãã
TEXT_QA_SYSTEM_PROMPT = ChatMessage(
content=(
"ããªãã¯äžçäžã§ä¿¡é ŒãããŠããQAã·ã¹ãã ã§ãã\n"
"äºåç¥èã§ã¯ãªããåžžã«æäŸãããã³ã³ããã¹ãæ
å ±ã䜿çšããŠã¯ãšãªã«åçããŠãã ããã\n"
"åŸãã¹ãããã€ãã®ã«ãŒã«:\n"
"1. åçå
ã§æå®ãããã³ã³ããã¹ããçŽæ¥åç
§ããªãã§ãã ããã\n"
"2. ãã³ã³ããã¹ãã«åºã¥ããŠã...ãããã³ã³ããã¹ãæ
å ±ã¯...ãããŸãã¯ããã«é¡ãããããªèšè¿°ã¯é¿ããŠãã ããã"
),
role=MessageRole.SYSTEM,
)
# QAããã³ãããã³ãã¬ãŒãã¡ãã»ãŒãž
TEXT_QA_PROMPT_TMPL_MSGS = [
TEXT_QA_SYSTEM_PROMPT,
ChatMessage(
content=(
"ã³ã³ããã¹ãæ
å ±ã¯ä»¥äžã®ãšããã§ãã\n"
"---------------------\n"
"{context_str}\n"
"---------------------\n"
"äºåç¥èã§ã¯ãªãã³ã³ããã¹ãæ
å ±ãèæ
®ããŠãã¯ãšãªã«çããŸãã\n"
"Query: {query_str}\n"
"Answer: "
),
role=MessageRole.USER,
),
]
# ãã£ããQAããã³ãã
CHAT_TEXT_QA_PROMPT = ChatPromptTemplate(message_templates=TEXT_QA_PROMPT_TMPL_MSGS)(2) ã€ã³ããã¯ã¹ãšã¯ãšãªãšã³ãžã³ã®äœæã
from llama_index import VectorStoreIndex
indexes = []
query_engines = []
for document in anime_docs:
# ã€ã³ããã¯ã¹ã®äœæ
index = VectorStoreIndex.from_documents(
[anime_doc],
)
indexes.append(index)
# ã¯ãšãªãšã³ãžã³ã®äœæ
query_engine = index.as_query_engine(
similarity_top_k=3,
text_qa_template=CHAT_TEXT_QA_PROMPT,
)
query_engines.append(query_engine)(3) ãŸã©ãâãã®ã«ã®è³ªåå¿çã
# ãŸã©ãâãã®ã«ã®è³ªåå¿ç
response = query_engines[0].query("ãŸã©ãâãã®ã«ã®äž»é¡æãæã£ãŠããææã¯ïŒ")
print(response)ãŸã©ãâãã®ã«ã®äž»é¡æãæã£ãŠããææã¯ãã¯ã©ãªã¹ãšKalafinaã§ãã(4) ãŒã£ã¡ã»ãã»ãã£ã!ã®è³ªåå¿ç
# ãŒã£ã¡ã»ãã»ãã£ãïŒã®è³ªåå¿ç
response = query_engines[1].query("ãŒã£ã¡ã»ãã»ãã£ãïŒã®äœè
ã®ååã¯ïŒ")
print(response)ãŒã£ã¡ã»ãã»ãã£ãïŒã®äœè
ã®ååã¯ã¯ãŸãããã§ãã5. 芪éå±€ã®ã¯ãšãªãšã³ãžã³ã®æºå
Colabã§ã®èŠªéå±€ã®ã¯ãšãªãšã³ãžã³ã®æºåæé ã¯ã次ã®ãšããã§ãã
(1) 芪éå±€ã®ããŒããªã¹ããšã¯ãšãªãšã³ãžã³èŸæžã®æºåã
IndexNodeã«ã¯ãtextã«èŠçŽãindex_idã«ã€ã³ããã¯ã¹IDãæå®ããŸãã
from llama_index.schema import IndexNode
# èŠçŽ
summaries = [
"ãéæ³å°å¥³ãŸã©ãâãã®ã«ãã¯ãæ°ããã¢ãããŒãã§éæ³å°å¥³ãã®ã®ãžã£ã³ã«ãå·æ°ããããŒã¯ã»ãã¡ã³ã¿ãžãŒäœåã§ããç©èªã¯éæ³å°å¥³ãã¡ã®éåœãèŠæ©ãçµ¶æãæããŠããã圌女ãã¡ãéå¥³ãšæŠãå§¿ãäžå¿ã§ããäœåã¯è€æ°ã®è§£éããããŸãããæç¢ºãªæ
å ±ã¯æäŸãããŠããŸããããŸããè€æ°ã®äŒç€Ÿãé¢äžããŠãããæ°æ¿æä¹ç£ç£ãšã·ã£ããå¶äœã®ã¿ãã°ã«ãããªãªãžãã«äœåãšããŠæ³šç®ãããŠããŸããäœåã¯å€ãã®ãã¡ã³ãç²åŸããæ§ã
ãªè³ãåè³ããŸãããåå Žçãæ°ç·šãåéã®ç©èªãé«ãè©äŸ¡ãåããŠããŸããããŸããŸãªã€ãã³ããã³ã©ãã¬ãŒã·ã§ã³ãè¡ãããŸããã",
"æäŸãããããã¹ãã«ã¯ãããŒã£ã¡ã»ãã»ãã£ãïŒããšãã4ã³ã挫ç»ã®æ
å ±ãå«ãŸããŠããŸããç©èªã¯äž»äººå
¬ã®åŸè€ã²ãšãããã³ã掻åãéããŠæé·ããŠããæ§åãæããŠããŸãã圌ã¯å¯Ÿäººã³ãã¥ãã±ãŒã·ã§ã³ãèŠæã§ãããèªå·±è¯å®æã®äœããã³ã³ãã¬ãã¯ã¹ãæ±ããŠããŸããããã³ã掻åãéããŠæé·ããŠãããŸãã圌ã¯ã®ã¿ãŒã®æŒå¥æè¡ãé«ããäœè©ãè¡ã£ãŠããŸããããã³ãã§ã®å®åãååã«çºæ®ããããšãã§ããŠããŸããããŸããã²ãšã以å€ã®ã¡ã³ããŒã«ã¯è¹å€ãé代ããªã§ãŠãšãããã£ã©ã¯ã¿ãŒãããŸãã圌ãããã³ã掻åãéããŠæé·ãã倢ãè¿œãæ±ããŠããŸãã",
]
# ããŒããªã¹ã
nodes = [
IndexNode(text=summaries[0], index_id=anime_docs[0].doc_id),
IndexNode(text=summaries[1], index_id=anime_docs[1].doc_id),
]
# ã¯ãšãªãšã³ãžã³èŸæž
query_engine_dict = {
anime_docs[0].doc_id: query_engines[0],
anime_docs[1].doc_id: query_engines[1],
}(2) VectorStoreIndexãšVectorRetrieverã®æºåã
VectorStoreIndexã«ã¯ãå
çšæºåããããŒããªã¹ããèšå®ããŸãã
# VectorStoreIndexã®æºå
vector_index = VectorStoreIndex(nodes)
# VectorRetrieverã®æºå
vector_retriever = vector_index.as_retriever(
similarity_top_k=1
)(3) RecursiveRetrieverãšã¯ãšãªãšã³ãžã³ã®æºåã
RecursiveRetrieverã«ã¯ãå
çšæºåããVectorRetrieverãšã¯ãšãªãšã³ãžã³èŸæžãèšå®ããŸãã
from llama_index.retrievers import RecursiveRetriever
# RecursiveRetrieverã®æºå
recursive_retriever = RecursiveRetriever(
"vector",
retriever_dict={"vector": vector_retriever},
query_engine_dict=query_engine_dict,
)(4) ã¯ãšãªãšã³ãžã³ã®æºåã
from llama_index.query_engine import RetrieverQueryEngine
from llama_index.response_synthesizers import get_response_synthesizer
# ResponseSynthesizerã®æºå
response_synthesizer = get_response_synthesizer(
# service_context=service_context,
response_mode="compact",
text_qa_template=CHAT_TEXT_QA_PROMPT,
)
# ã¯ãšãªãšã³ãžã³ã®æºå
query_engine = RetrieverQueryEngine.from_args(
recursive_retriever,
response_synthesizer=response_synthesizer
)(5) ãŸã©ãâãã®ã«ã®è³ªåå¿çã
# ãŸã©ãâãã®ã«ã®è³ªåå¿ç
response = query_engines[0].query("ãŸã©ãâãã®ã«ã®äž»é¡æãæã£ãŠããææã¯ïŒ")
print(response)ãŸã©ãâãã®ã«ã®äž»é¡æãæã£ãŠããææã¯ãã¯ã©ãªã¹ãšKalafinaã§ãã
ãã°ã確èªãããšãæåã«èŠªéå±€ã®top-k=1ã§ããŸã©ãâãã®ã«ãã®ããŒããéžæãããŠããŸãã
Top 1 nodes:
[Similarity score: 0.856092] ãéæ³å°å¥³ãŸã©ãâãã®ã«ãã¯ãæ°ããã¢ãããŒãã§éæ³å°å¥³ãã®ã®ãžã£ã³ã«ãå·æ°ããããŒã¯ã»ãã¡ã³ã¿ãžãŒäœåã§ãã...åéå±€ã®top-k=3ã§è³ªåå¿çãåŠçããåŸãæåŸã«åéå±€ããè¿ãããçµæã§ä»¥äžã®è³ªåå¿çãè¡ããæçµçµæãåºåããŸãã
ã³ã³ããã¹ãæ
å ±ã¯ä»¥äžã®ãšããã§ãã
---------------------
Query: ãŸã©ãâãã®ã«ã®äž»é¡æãæã£ãŠããææã¯ïŒ
Response: ãŸã©ãâãã®ã«ã®äž»é¡æãæã£ãŠããææã¯ãKalafinaã§ãã
---------------------
äºåç¥èã§ã¯ãªãã³ã³ããã¹ãæ
å ±ãèæ
®ããŠãã¯ãšãªã«çããŸãã
Query: ãŸã©ãâãã®ã«ã®äž»é¡æãæã£ãŠããææã¯ïŒ
Answer:(6) ãŒã£ã¡ã»ãã»ãã£ã!ã®è³ªåå¿ç
# ãŒã£ã¡ã»ãã»ãã£ãïŒã®è³ªåå¿ç
response = query_engines[1].query("ãŒã£ã¡ã»ãã»ãã£ãïŒã®äœè
ã®ååã¯ïŒ")
print(response)ãŒã£ã¡ã»ãã»ãã£ãïŒã®äœè
ã®ååã¯ã¯ãŸãããã§ãã
ãã°ã確èªãããšãæåã«èŠªéå±€ã®top-k=1ã§ããŒã£ã¡ã»ãã»ãã£ãïŒãã®ããŒããéžæãããŠããŸãã
Top 1 nodes:
[Similarity score: 0.863588] æäŸãããããã¹ãã«ã¯ãããŒã£ã¡ã»ãã»ãã£ãïŒããšãã4ã³ã挫ç»ã®æ
å ±ãå«ãŸããŠããŸãã...
åéå±€ã®top-k=3ã§è³ªåå¿çãåŠçããåŸãæåŸã«åéå±€ããè¿ãããçµæã§ä»¥äžã®è³ªåå¿çãè¡ããæçµçµæãåºåããŸãã
ã³ã³ããã¹ãæ
å ±ã¯ä»¥äžã®ãšããã§ãã
---------------------
Query: ãŒã£ã¡ã»ãã»ãã£ãïŒã®äœè
ã®ååã¯ïŒ
Response: ãŒã£ã¡ã»ãã»ãã£ãïŒã®äœè
ã®ååã¯ã¯ãŸãããã§ãã
---------------------
äºåç¥èã§ã¯ãªãã³ã³ããã¹ãæ
å ±ãèæ
®ããŠãã¯ãšãªã«çããŸãã
Query: ãŒã£ã¡ã»ãã»ãã£ãïŒã®äœè
ã®ååã¯ïŒ
Answer: 6. DocumentSummaryIndex
ãDocumentSummaryIndexãã䜿ããšãèŠçŽã®èªåçæãå¯èœã§ãã