æ§é ãå£ããªããã£ã³ã¯åïœRAGæ€çŽ¢ç²ŸåºŠãé«ããèšèšãã¿ãŒã³ãšå®è·µç¥
ã¯ããã«
RAGã®ç²ŸåºŠã¯ãã£ã³ã¯ã®èšèšã§ã»ãŒæ±ºãŸã
RAGïŒRetrieval-Augmented GenerationïŒãå°å ¥ããéãæåã®å€§ããªå£ã«ãªãã®ãäŒæ¥ããã¥ã¡ã³ãã®å·šå€§ãã§ãã
PDFã»Wordãªã©ã®ããã¥ã¡ã³ãã¯ãæ°äžãæ°åäžæåèŠæš¡ã«ãªãããšãçãããããŸããã
ãã®å·šå€§ããã¥ã¡ã³ãããã®ãŸãŸåã蟌ããšããéçšãåã£ãŠããŸããšã
- æå³ãæ··ãã£ãç¶æ ã®ãã¯ãã«ãçæããã
- ãã€ãºãå¢ããé¡äŒŒåºŠæ€çŽ¢ãå®å®ããªã
- ãäžèŠé¢ä¿ããããã ããã©æ¬è³ªçã«ã¯å€ãããæèãè¿ã£ãŠãã
ãšãã£ãåé¡ã«ã€ãªãããæçµçã«ã¯ã質åã«å¯ŸããŠæ£ããããã¥ã¡ã³ããåŒããªãããšããRAGã®å žåçãªå€±æãåŒãèµ·ãããŸãã
éèŠãªã®ã¯ãããããåé¡ã®å€ãã¯åã蟌ã¿ã¢ãã«ã®æ§èœã§ã¯ãªãããã£ã³ã¯åã®èšèšã«åé¡ãããå¯èœæ§ããããšããç¹ã§ãã
ãããã倱æäŸïŒåã蟌ã¿ã¯æ£ããè¡ã£ãã®ã«ç²ŸåºŠãåºãªã
å€ãã®ãããžã§ã¯ãã§ã¯ãããªç¶æ³ãèµ·ããŸãã
ã»åã蟌ã¿ã¢ãã«ã®ç²ŸåºŠã¯åå
ã»ãã¯ãã«DBã«ãæ£ããæ ŒçŽ
ã»æ€çŽ¢ã¯æåããŠãã
...ã«ãé¢ããããåçããããŠããŸãã
ãã®åå ã®äžã€ãã
ææžæ§é ãå£ããŠããŸããã£ã³ã¯åã§ãã
äŸãã°ã
- ã¿ã€ãã«ã𿬿ãé¢ãã
- 衚ãšèª¬ææãå¥ãã£ã³ã¯ã«è¡ã£ãŠããŸã£ã
- ååŸé¢ä¿ãéåããŠæå³ãèãŸã£ãŠããŸã£ã
- 1ãã£ã³ã¯ã倧ãéããŠäœã®è©±ããããããªããªã£ãŠããŸã£ã
ãããããã£ã³ã¯ã¯ãEmbeddingã¢ãã«ããèŠãŠããæå³ã®ãŸãšãŸãããšããŠçè§£ãã¥ãããçµæãšããŠæ€çŽ¢ç²ŸåºŠãèœã¡ãèŠå ã«ããªã£ãŠããŸããŸãã
æ¬èšäºã®ç®ç
ãã®èšäºã§ã¯ãäŒæ¥ããã¥ã¡ã³ããæ±ãå®åè ã®æ¹åãã«ã
- æ§é ãå£ããªããã£ã³ã¯èšèšã®èãæ¹
- ãã£ã³ã¯ã®çš®é¡ãšäœ¿ãåã
- å®åã§äœ¿ããèšèšãã¿ãŒã³
ã«ã€ããŠç޹ä»ããŠãããŸãã
ãã£ã³ã¯åãšã¯äœã
ãã£ã³ã¯åãšã¯ã倧ããªæç« ããæå³ã®ãŸãšãŸãããšãã«åå²ããŠæ±ãããã®ååŠçã§ãã
åçŽã«æåæ°ã§åãã®ã§ã¯ãªããã¿ã€ãã«ã»æ®µèœã»ç®æ¡æžããªã©ãæç« ã®æ§é ãå£ããã«å°ããªãããã¯ãžæŽçããããšãéèŠã«ãªããŸãã
ãã£ã³ã¯åãå¿ èŠãªçç±
ãã£ã³ã¯ãé©åã«èšèšãããŠãããšã
- é¡äŒŒåºŠæ€çŽ¢ã®ç²ŸåºŠãå®å®ãã
- LLMã«æž¡ãã³ã³ããã¹ããæ£ãããªã
- åçãæ ¹æ ã«å¿ å®ã«ãªãããã«ã·ããŒã·ã§ã³ãæžã
ãšãã£ãã¡ãªãããåŸãããŸãã
ãªãæç« ããã®ãŸãŸEmbeddingã«æããŠã¯ãããªãã®ãïŒ
æç« ã倧ãããŸãŸåã蟌ããšã
- å šãç°ãªã話é¡ãåããã¯ãã«ã«æ··ãã
- ã¿ã€ãã«ãšèª¬ææãé¢ãã
- 衚ãšè§£èª¬ã奿±ãã«ãªã
- 1ãã£ã³ã¯ã«è€æ°ã®ããŒããå ¥ãããå¹³ååãããããã¯ãã«ãã§ãã
ãªã©ã®åé¡ãçºçããŸãã
åã蟌ã¿ã¢ãã«ã¯ãæå³ã®ãŸãšãŸããåæã«èšèšãããŠããããã
ãŸãšãŸãã®ãªããã£ã³ã¯ â ãã€ãºãå¢ãã â æ€çŽ¢ç²ŸåºŠãäžå®å®
ãšãã倱æã«ã€ãªãããŸãã
ãã£ã³ã¯åã®çš®é¡
ãã£ã³ã¯åãšãã£ãŠãã1ã€ã®æ¹æ³ã§äžèœã«å¯Ÿå¿ã§ããããã§ã¯ãããŸããã
ææžã®çš®é¡ãç®çã«å¿ããŠè€æ°ã®ã¢ãããŒããååšããŸããããã§ã¯ãRAGã§ãã䜿ãããäž»èŠãªãã£ã³ã¯ææ³ãšããã®ç¹åŸŽããŸãšããŠæ¯èŒããŸãã
| ææ³ | ç¹åŸŽ | ã¡ãªãã | ãã¡ãªãã | åããŠããã±ãŒã¹ |
|---|---|---|---|---|
| åºå®é·ãã£ã³ã¯ (Fixed-size) |
äžå®ããŒã¯ã³æ°ã§æ©æ¢°çã«åå² | å®è£ ãæãç°¡åïŒé«éïŒäºæž¬å¯èœ | ææžæ§é ãå£ãããã | Webèšäºãªã©æ§é ã匷ããªãææž |
| ãªãŒããŒã©ãããã£ã³ã¯ (Sliding window) |
ååŸã®æèãéããŠåå² | æèãåãã«ããïŒæ€çŽ¢ã®å®å®æ§UP | åé·ã§DB容éãå¢ãã | FAQãäŒè©±ãã°ãªã©ååŸé¢ä¿ãéèŠãªææž |
| æ§é ããŒã¹ãã£ã³ã¯ (Markdown/段èœ/èŠåºãã«ãŒã«) |
段èœã»èŠåºããªã©ã®æ§é ã§åå² | æ§é ãå£ããªã ã¡ã€ã³ææ³ïŒå質å®å® | ææžã®æžãæ¹ã«äŸå | ããã¥ã¢ã«ã仿§æžãè°äºé² |
| ã»ãã³ãã£ãã¯ãã£ã³ã¯ (LLM-driven) |
æå³ã®ãŸãšãŸãã«å¿ããŠèªåã§åå² | 衚ã»å³ã»è€éãªæ§é ã«åŒ·ãïŒç²ŸåºŠãé«ã | ã³ã¹ãé«ïŒã¢ãã«äŸå | å€§èŠæš¡PDFãç ä¿®è³æãå€éšã®è€éææž |
| ãã€ããªãã | æ§é à æå³ à ãªãŒããŒã©ãããçµåã | æãå®å®ã»é«ç²ŸåºŠïŒå®åã®æé©è§£ | èšèšã«å€å°ã®ç¥èãå¿ èŠ | äŒæ¥å ææžãã¹ãŠïŒæãæ±çšïŒ |
åºå®é·ãã£ã³ã¯(Fixed-size)
åºå®é·ãã£ã³ã¯ã¯ãäžå®ã®ããŒã¯ã³æ°ã§æ©æ¢°çã«æç« ãåå²ããæãã·ã³ãã«ãªæ¹æ³ã§ãã
ãã€äœ¿ãã
- ææžã®æ§é ã匱ãïŒããã°èšäºãçãæ¥åææžãªã©ïŒ
- ãŸãRAGãåããããå Žå
- ææžã®å質ã«ã°ãã€ãããããæ§é ããŒã¹ã®ã«ãŒã«ãåœãŠã«ããæ
ãã¥ãŒãã³ã°ã®ç®å®
- chunk_size = 300ã500tokenãæ¥æ¬èªææžã®å®çª
- overlapã¯0ãåºæ¬
æå°å®è£
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=400,
chunk_overlap=0
)
docs = splitter.split_text(text)
å質ãã§ãã¯
- åºå®é·ã§ç²ŸåºŠãå®å®ããªãå Žåãæ§é ããŒã¹orãªãŒããŒã©ãããžåãæ¿ã
ãããã倱æ
- chunkã倧ãããéãã1ãã£ã³ã¯ã«è€æ°ã®ããŒããæ··ãã£ãŠæ€çŽ¢ããã¬ã
â»åºå®é·ãã£ã³ã¯ã¯ãæ©æ¢°çã«åå²ãããããã»ãã¬ãŒã¿ãŒã¯å ¥ããªãã
ãªãŒããŒã©ãããã£ã³ã¯(Sliding window)
ãªãŒããŒã©ãããã£ã³ã¯ã¯ãååŸã®æèãé£ç¶ããææžã«åŒ·ãææ³ã§ãã
ãã€äœ¿ãã
- FAQ/ äŒè©±ãã°/ æé æžã®ããã«ãäžã€åã®æããæå³ãæã€ææž
- åºå®é·ã§ã¯æèãåããŠããŸãå Žå
ãã¥ãŒãã³ã°ã®ç®å®
- chunk_size = 512tokens / overlap 80ã120tokens
- 容éãå¢ããããããã¯ãã«DBã®ã³ã¹ãã«æ³šæ
æå°å®è£
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=100,
separators=["\n\n", "\n", "ã", "ã", " "]
)
docs = splitter.split_text(text)
å質ãã§ãã¯
- 容éãçå¢ããŠããªãã確èª
ãããã倱æ
- overlapãå¢ããéããŠLLMã®å ¥åãè¥å€§åãã
æ§é ããŒã¹ãã£ã³ã¯
æ§é ããŒã¹ãã£ã³ã¯ã¯ãèŠåºãã»æ®µèœã»Markdownæ§é ãæŽ»ããããã£ã³ã¯åã§ãæãå å®ã§å質ã®å®å®ããææ³ã§ãã
ãã€äœ¿ãã
- 瀟å ããã¥ã¢ã«ã仿§æžãè°äºé²ãç ç©¶è³æ
- ææžæ§é ïŒH1/H2ãæ®µèœãç®æ¡æžãïŒãæŽã£ãŠãã
ãã¥ãŒãã³ã°ã®ç®å®
- èŠåºãåäœã§ç²ãåã â æ®µèœããšã«åŸ®èª¿æŽ
- 倧ããããæ®µèœã¯ chun_size=200ã400tokensãäžéã«ååå²
- ãã¿ã€ãã«+æ¬æãã1ãã£ã³ã¯ã«ãŸãšãã
æå°å®è£
from llama_index.core import Document
from llama_index.core.node_parser import MarkdownNodeParser
parser = MarkdownNodeParser()
documents = [Document(text=textdata)]
nodes = parser.get_nodes_from_documents(documents)
å質ãã§ãã¯
- ã¿ã€ãã«ã𿬿ãå¿ ãåããã£ã³ã¯ã«å«ãŸããŠããã
- 衚ã»ç®æ¡æžããåæãããŠããªãã
ãããã倱æ
- Markdownã¿ã°ã厩ããŠããPDFãçŽæ¥äœ¿ããæå³ããªãåå²ã«ãªã
以äžã®åçã®ããã«èªåã§Markdownã®ææžãçšæãããã£ã³ã¯åããŠã¿ãŸãããããNodeãã®è¡ã¯èªåã§ããã°ã©ãã³ã°ã§åºåãããã®ã§ããããšãŠã綺éºã«ãã£ã³ã¯ãã§ããŠãããšãã£ãå°è±¡ã§ããã

ã»ãã³ãã£ãã¯ãã£ã³ã¯
ã»ãã³ãã£ãã¯ãã£ã³ã¯ã¯ãLLMãææããèªç¶ãªãŸãšãŸãã倿ããŠåå²ããææ³ã§ã衚ã»å³ã»è€éææžãžã®èæ§ãé«ãã®ãç¹åŸŽã§ãã
ãã€äœ¿ãã
- 衚ã»å³ã»ç®æ¡æžããè€éã«å ¥ãæ··ããPDF
- LlamaParse/ LlamaIndexãªã©é«åºŠãªããŒãµãŒã䜿ããç°å¢
ãã¥ãŒãã³ã°ã®ç®å®
- ã³ã¹ãèŠç©ãããå¿ ã宿œ(PDF1åã§æ°ååã)
æå°å®è£
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import AzureOpenAIEmbeddings
# OSã§ãAZURE_OPENAI_API_KEYãããAZURE_OPENAI_ENDPOINTãã
#ãAZURE_OPENAI_API_VERSIONãã¯æå®
embed = AzureOpenAIEmbeddings(
azure_deployment="text-embedding-3-small"
)
splitter = SemanticChunker(
embeddings=embed,
breakpoint_threshold_type="percentile",
breakpoint_threshold_amount=0.95, # 調æŽç®æ
)
chunks = splitter.split_text(text)
å質ãã§ãã¯
- 衚 + 説æãåããã£ã³ã¯ã«å ¥ã£ãŠããã
- æå³çã«åŸ®åŠãªåå²ãçºçããŠããªãã
ãããã倱æ
- ã»ãã³ãã£ãã¯ãå šäœã«åŒ·åŒã«é©çš â ã³ã¹ãçå¢
ãã€ããªãããã£ã³ã¯
ãã€ããªããã¯ãæ§é * æå³ * ãªãŒããŒã©ãããçµã¿åãããå®åæé©è§£ã§ããäŒæ¥ã®ããã¥ã¡ã³ãã¯è€éãªãããåäžææ³ã ãã ãšéçããããŸãã
ãã€äœ¿ãã
- 倧åã®ç€Ÿå ææžïŒããã¥ã¢ã«ãæé æžãå éšè³æïŒ
- åææžã®æ§è³ªãç°ãªãã1ã€ã®ææ³ã§çµ±äžã§ããªããããžã§ã¯ã
æå°ãµã³ãã«(Flow)
æ§é ããŒã¹åå² â ã»ãã³ãã£ãã¯è£æ£ â Sliding Windowå°é远å
å質ãã§ãã¯
- ãã€ããªããåŸã«ããã³ããç·ããŒã¯ã³ãå¢ããŠããªãã確èª
ãããã倱æ
- æåãããã€ããªããã䜿ããèšèšãè€éå
- è€æ°ææ³ã®é©çšé åºã誀ãå質ãå®å®ããªã
ææžã¿ã€ãå¥ïŒæé©ãªãã£ã³ã¯åæŠç¥
ææžã®çš®é¡ã«ãã£ãŠãæé©ãªãã£ã³ã¯åæŠç¥ã¯å€§ããç°ãªããŸãã
åããRAGãã®ä»çµã¿ã§ããããããã®ææžã«ãã£ãŠãæ§é ãäŸåé¢ä¿ãå
šãéããããææ³ãåäžåããããšã¯é£ãããšèããŠããŸãã
ãããªäžã§ãææžæ¯ã®ãã£ã³ã¯åã®æé©ãäžèЧåèŽããŸãã
| ææžã¿ã€ã | æšå¥šãã£ã³ã¯ææ³ | çç±ïŒãªãçžæ§ãè¯ããïŒ | 泚æç¹ |
|---|---|---|---|
| æ¥åããã¥ã¢ã« / æé æž | æ§é ããŒã¹ + ãªãŒããŒã©ããå°é | èŠåºãã»çªå·ã»æ®µèœæ§é ãæç¢ºã§æ©æ¢°çã«åå²ããããïŒã¹ãããéã®æèäŸåã匷ããæµããåããªãããšãéèŠ | 倧段èœãé·ãå Žåãå³è¡šãå«ãå Žåã¯ãéšåçã«ã»ãã³ãã£ãã¯ã§è£æ£ |
| FAQ | Q&Aåäœã®æ§é ããŒã¹ or å°ããã®åºå®é·ãã£ã³ã¯ | 質åãšåçãã»ããã§æå³ãæã€ãããQuestion + Answerã1ãã£ã³ã¯ã«ãã | 質åã ãæ€çŽ¢ãããå Žåã¯å¥ã€ã³ããã¯ã¹æšå¥š |
| å¥çŽæž / èŠçŽ | æ§é ããŒã¹ïŒæ¡é åäœïŒ + ãªãŒããŒã©ãã | æ¡é ãç¬ç«ãã€ã€çžäºåç §ãçºçãããã | å®çŸ©ã»ã¯ã·ã§ã³ã¯ç¹ã«äŸåé¢ä¿ã匷ã |
| æè¡è³æ / èšèšæž | æ§é ããŒã¹ + ã»ãã³ãã£ãã¯è£æ£ | å³è¡šã»ç®æ¡æžãã»æ®µèœå·®ã倧ãã | ãã£ãã·ã§ã³ãåé¢ããªãããèª¿æŽ |
| PDFïŒè¡šã»å³ã»ç»åå€ãïŒ | ã»ãã³ãã£ã㯠or LlamaParse | æ§é ã厩ããã¡ïŒå³è¡šã𿬿ã®çŽä»ããå¿ é | OCR誀èªèã衚ã®ãºã¬ã«æ³šæ |
èšèšãã¿ãŒã³é
ãã¿ãŒã³AïŒéå±€æ§é ããŒã¹
èŠåºããç« æ§é ãæç¢ºãªææžã®ãã¿ãŒã³ã§ãã
倧æ ãèŠåºãåäœã§åå²ããå¿
èŠã«å¿ããŠæ®µèœæ¯ã«ç²åºŠãæŽããŸãã
ãã¿ãŒã³BïŒæ®µèœããšã®å·®åã倧ããææžåã
段èœã®é·ããå¯åºŠã«ã°ãã€ããããææžã§ã¯ããã®ãŸãŸæ§é ããŒã¹ã§åããšç²åºŠãåããŸããã
ãã®ãã¿ãŒã³ã§ã¯ãèŠåºãã§å€§æ ãåãã€ã€ãé·ãããæ®µèœã ããã»ãã³ãã£ãã¯ãã£ã³ã¯ã§èª¿æŽããŸãã
ãã¿ãŒã³CïŒPDFã®å³è¡šãå€ãææžåã
å³è¡šã»ãã£ãã·ã§ã³ã»æ¬æãå€ãåºãŠããPDFã§ã¯ãæ§é ããŒã¹ã ãšèª€åå²ãçºçããããã§ãã
ãã®ãã¿ãŒã³ã§ã¯ãLlamaParseãã»ãã³ãã£ãã¯ã䜿ããå³ãšèª¬æãåããã£ã³ã¯ã«å
¥ãããã«è£æ£ããŸãã
ãã¿ãŒã³DïŒæ··åšææž
èŠåºãã»æ®µèœã»å³è¡šã»ç®æ¡æžããæ··ãã£ãŠãããåäžææ³ã§ã¯å質ãå®å®ããªãã±ãŒã¹ã§ãã
æ§é ããŒã¹ã§å€§æ ãäœããé£ããç®æã§ã»ãã³ãã£ãã¯ã§è£æ£ããå¿
èŠã«å¿ããŠå°éãªãŒããŒã©ããã远å ããŠãããŸãã
ãŸãšã
- RAGã®ç²ŸåºŠã¯ãåã蟌ã¿ã¢ãã«ãã®ãã®ããããã£ã³ã¯èšèšã«å€§ããäŸåããŸãã
- ææžã®çš®é¡æ¯ã«é©åãªãã£ã³ã¯åææ³ãéžæããããšã§ãæ€çŽ¢ç²ŸåºŠãå®å®ãããããšãã§ããã
- å³è¡šã»è¡šã»ç»åãªã©ãæ··åšããææžã§ã¯ãåäžã®ææ³ã§ã¯éçããããè€æ°ææ³ãçµã¿åãããã¢ãããŒããæ€èšããå¿ èŠãããã
é¢é£èšäº
ãã£ã³ã¯èšèšãRAGå šäœã®ç²ŸåºŠã«ã©ã圱é¿ããã®ãããè©äŸ¡èгç¹ããæŽçããèšäºã¯ãã¡ã
Discussion