
LlamaIndex ã® SentenceWindowNodeParser ã詊ã
ãLlamaIndexãã®ãSentenceWindowNodeParserãã詊ããã®ã§ããŸãšããŸããã
1. SentenceWindowNodeParser
ãSentenceWindowNodeParserãã¯ãæãåã蟌ã¿ã§æ€çŽ¢ããæ¡åŒµãŠã£ã³ããŠã«ãªã³ã¯ããããã®ããŒãµãŒã§ããé¢é£ããã³ã³ããã¹ãããããã现ããååŸã§ããããã«ãªããåæã«ååãªã³ã³ããã¹ãã確ä¿ã§ããŸãã
2. ããã¥ã¡ã³ãã®æºå
ä»åã¯ããã³ã¬ããã£ã¢ã®ããŒã£ã¡ã»ãã»ãã£ãïŒãã®ããããã®ããã¥ã¡ã³ããçšæããŸããã
ã»bocchi.txt

3. ã»ããã¢ãã
Colabã§ã®ã»ããã¢ããæé ã¯ã次ã®ãšããã§ãã
(1) ããã±ãŒãžã®ã€ã³ã¹ããŒã«ã
# ããã±ãŒãžã®ã€ã³ã¹ããŒã«
!pip install llama-index
!pip install sentence_transformers(2) ç°å¢å€æ°ã®æºåã
以äžã®ã³ãŒãã® <OpenAI_APIã®ããŒã¯ã³> ã«ã¯OpenAI APIã®ããŒã¯ã³ãæå®ããŸãã(ææ)
import os
# ç°å¢å€æ°ã®èšå®
openai.api_key = "<OpenAI_APIã®ããŒã¯ã³>"(3) ãã°ã¬ãã«ã®èšå®ã
import logging
import sys
# ãã°ã¬ãã«ã®èšå®
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, force=True)(4) Colabã«ããã¥ã¡ã³ããbocchi.txtããã¢ããããŒãã
(5) ããã¥ã¡ã³ãã®èªã¿èŸŒã¿ã
from llama_index import SimpleDirectoryReader
# ããã¥ã¡ã³ãã®èªã¿èŸŒã¿
documents = SimpleDirectoryReader(
input_files=["bocchi.txt"]
).load_data()4. SentenceWindowNodeParserã®å©çš
Colabã§ã®ãSentenceWindowNodeParserãã®å©çšæé ã¯ã次ã®ãšããã§ãã
(1) æ¥æ¬èªçšã®æããŒã¯ãã€ã¶ãŒã®æºåã
ããã©ã«ãã®æããŒã¯ãã€ã¶ãŒã¯æ¥æ¬èªæç« ãåå²ããŠãããªãã£ãã®ã§ãæºåããŸãã
from typing import Callable, List
# æ¥æ¬èªçšã®æããŒã¯ãã€ã¶ãŒ
def split_by_sentence_tokenizer() -> Callable[[str], List[str]]:
import re
import nltk
import os
from llama_index.utils import get_cache_dir
cache_dir = get_cache_dir()
nltk_data_dir = os.environ.get("NLTK_DATA", cache_dir)
# update nltk path for nltk so that it finds the data
if nltk_data_dir not in nltk.data.path:
nltk.data.path.append(nltk_data_dir)
sent_detector = nltk.RegexpTokenizer(u'[^ãïŒïŒã]*[ïŒïŒã.\n]')
return sent_detector.tokenize (2) SentenceWindowNodeParserã®æºåã
from langchain.embeddings import HuggingFaceEmbeddings
from llama_index import ServiceContext
from llama_index.llms import OpenAI
from llama_index.node_parser import SentenceWindowNodeParser
# SentenceWindowNodeParserã®æºå
node_parser = SentenceWindowNodeParser.from_defaults(
sentence_splitter=split_by_sentence_tokenizer(),
window_size=3,
window_metadata_key="window",
original_text_metadata_key="original_text",
)
# ãµãŒãã¹ã³ã³ããã¹ãã®æºå
service_context = ServiceContext.from_defaults(
llm=OpenAI(model="gpt-3.5-turbo", temperature=0.1),
embed_model = HuggingFaceEmbeddings(model_name="intfloat/multilingual-e5-large"),
node_parser=node_parser
)(3) ã€ã³ããã¯ã¹ã®æºåã
æããšã®å€§éã®ãã£ã³ã¯ãçæãããŸãã
from llama_index import VectorStoreIndex
# ã€ã³ããã¯ã¹ã®æºå
index = VectorStoreIndex.from_documents(
documents,
service_context=service_context,
)(4) QAããã³ããã®å®çŸ©ã
LlamaIndexã®è±èªçã®QAããã³ããã®çŽèš³ã«ãªããŸãã
from llama_index.llms.base import ChatMessage, MessageRole
from llama_index.prompts.base import ChatPromptTemplate
# QAã·ã¹ãã ããã³ãã
TEXT_QA_SYSTEM_PROMPT = ChatMessage(
content=(
"ããªãã¯äžçäžã§ä¿¡é ŒãããŠããQAã·ã¹ãã ã§ãã\n"
"äºåç¥èã§ã¯ãªããåžžã«æäŸãããã³ã³ããã¹ãæ
å ±ã䜿çšããŠã¯ãšãªã«åçããŠãã ããã\n"
"åŸãã¹ãããã€ãã®ã«ãŒã«:\n"
"1. åçå
ã§æå®ãããã³ã³ããã¹ããçŽæ¥åç
§ããªãã§ãã ããã\n"
"2. ãã³ã³ããã¹ãã«åºã¥ããŠã...ãããã³ã³ããã¹ãæ
å ±ã¯...ãããŸãã¯ããã«é¡ãããããªèšè¿°ã¯é¿ããŠãã ããã"
),
role=MessageRole.SYSTEM,
)
# QAããã³ãããã³ãã¬ãŒãã¡ãã»ãŒãž
TEXT_QA_PROMPT_TMPL_MSGS = [
TEXT_QA_SYSTEM_PROMPT,
ChatMessage(
content=(
"ã³ã³ããã¹ãæ
å ±ã¯ä»¥äžã®ãšããã§ãã\n"
"---------------------\n"
"{context_str}\n"
"---------------------\n"
"äºåç¥èã§ã¯ãªãã³ã³ããã¹ãæ
å ±ãèæ
®ããŠãã¯ãšãªã«çããŸãã\n"
"Query: {query_str}\n"
"Answer: "
),
role=MessageRole.USER,
),
]
# ãã£ããQAããã³ãã
CHAT_TEXT_QA_PROMPT = ChatPromptTemplate(message_templates=TEXT_QA_PROMPT_TMPL_MSGS)(5) ã¯ãšãªãšã³ãžã³ã®æºåã
ãMetadataReplacementPostProcessorããèšå®ããŸãã
from llama_index.indices.postprocessor import MetadataReplacementPostProcessor
# ã¯ãšãªãšã³ãžã³ã®æºå
query_engine = index.as_query_engine(
similarity_top_k=2,
node_postprocessors=[
MetadataReplacementPostProcessor(target_metadata_key="window")
],
text_qa_template=CHAT_TEXT_QA_PROMPT,
)(6) 質åå¿çã
# 質åå¿ç
response = query_engine.query("ã©ã€ãããŠã¹ãSTARRYãã®åºé·ã®ååã¯ïŒ")
print(response)
以äžã®2æããããããŸããã(ããããªãã®æ¹è¡ãç¹ãã£ãŠãã®ã§ãåŸã§èª¿æŽããæ¹ãè¯ããã)
[Similarity score: 0.896519]
äŒå°ç¥ææ(ããã¡ããã)
äžåæ²¢ã®ã©ã€ãããŠã¹ãSTARRY(ã¹ã¿ãŒãªãŒ)ãã®åºé·ãåãã女æ§ã
[Similarity score: 0.852028]
PAãã(ãŽãŒããŒãã)
ã©ã€ãããŠã¹ãSTARRYãã®PA(PublicAddress)ãšã³ãžãã¢ãåãã女æ§ã
ãã°ã確èªãããšããããããæã®å2æãšåŸ2æãç¹ããã³ã³ããã¹ãã§QAããã³ãããæããŠããããšãããããŸãã
ã³ã³ããã¹ãæ
å ±ã¯ä»¥äžã®ãšããã§ãã
---------------------
äŒå°ç¥è¹å€ãã¡ãšä»²ãããã圌女ãã¡ã«ã¯æ¬åãåä¹ã£ãŠããããèŠããããŠããããPAããããšåŒã°ããŠããã ã±ã£ãšèŠã¯ãŸãããªæž
æ¥ç³»ã«èŠããããããªãããå æžãªæ§æ Œã§ã髿 ¡ãæèµ·ããã1幎ã§äžéããŠããã äŒå°ç¥ææãšã¯å幎代ã®ããã§ãããŸã«åœŒå¥³ãšç¬ã身å士ã®éãåãåºããŠããã
äŒå°ç¥ææ(ããã¡ããã)
äžåæ²¢ã®ã©ã€ãããŠã¹ãSTARRY(ã¹ã¿ãŒãªãŒ)ãã®åºé·ãåãã女æ§ã èªçæ¥ãè¿ããŠ30æ³ã«ãªã£ãã äŒå°ç¥è¹å€ã®å§ã
æ åå¶äœã®å匷ãããŠãããããçµæãã³ããMVãå¶äœããéã«äŒå°ç¥è¹å€ããäŸé Œããã2å·ãšå
±ã«MVå¶äœãæäŒã£ãã ããªãã®é¢é£ãã§ããããã女ã®åã倧奜ãã ã²ãšãã®ããžã¥ã¢ã«ããæ°ã«å
¥ãã§ã圌女ããã¡é¡ã«ãªã£ãããã³ã·ã§ã³ãäžãããããã®ããžã¥ã¢ã«ã10ç§ããæããªãã£ãããã«MVã«äœ¿ãã®ã¯ãããããã²ãšãã®åºçªãã»ãšãã©ã«ããããã
PAãã(ãŽãŒããŒãã)
ã©ã€ãããŠã¹ãSTARRYãã®PA(PublicAddress)ãšã³ãžãã¢ãåãã女æ§ã å£ãã¢ã¹ãã€ããé»ã髪ãã¹ãã¬ãŒããã³ã°ã«ããŠããã äŒå°ç¥è¹å€ãã¡ãšä»²ãããã圌女ãã¡ã«ã¯æ¬åãåä¹ã£ãŠããããèŠããããŠããããPAããããšåŒã°ããŠããã
---------------------
äºåç¥èã§ã¯ãªãã³ã³ããã¹ãæ
å ±ãèæ
®ããŠãã¯ãšãªã«çããŸãã
Query: ã©ã€ãããŠã¹ãSTARRYãã®åºé·ã®ååã¯ïŒ
Answer: