
RAGè©äŸ¡ããŒã« ragas ã詊ã
RAGè©äŸ¡ããŒã«ãragasãã詊ããã®ã§ããŸãšããŸããã
1. ragas
ãragasãã¯ããRAGã (Retrieval Augmented Generation) ãã€ãã©ã€ã³ãè©äŸ¡ããããã®ãã¬ãŒã ã¯ãŒã¯ã§ãããRAGãã¯å€éšããŒã¿ã䜿çšããŠLLMã®ã³ã³ããã¹ããæ¡åŒµããLLMã¢ããªã±ãŒã·ã§ã³ã§ãããragasãã¯ãã®ãã€ãã©ã€ã³ãè©äŸ¡ããŠãããã©ãŒãã³ã¹ãå®éåããŸãã
2. Colabã§ã®å®è¡
Colabã§ã®å®è¡æé ã¯ã次ã®ãšããã§ãã
(1) RAGãã€ãã©ã€ã³ã®æºåã
ä»åã¯ã以äžã®èšäºã§äœæãããELYZAã®RAGãã€ãã©ã€ã³ã䜿ããŸãã
(2) ããã±ãŒãžã®ã€ã³ã¹ããŒã«ã
# ããã±ãŒãžã®ã€ã³ã¹ããŒã«
!pip install ragas datasets(3) ç°å¢å€æ°ã®æºåã
ãragasãã¯ãOpenAI APIãã§è©äŸ¡ããŸãã
import os
import openai
# ç°å¢å€æ°ã®èšå®
os.environ["OPENAI_API_KEY"] = "<OpenAI_APIã®ããŒã¯ã³>"
openai.api_key = os.environ["OPENAI_API_KEY"](4) 質åãæèãåç ã®ãªã¹ãã®æºåã
ä»åã¯ã10åã®ã質åããçšæããŠãRAGãã€ãã©ã€ã³ã§ãæèããšãåçããçæããŸãã
# 質å
questions = [
"åŸè€ã²ãšãã®åŸæãªæ¥œåšã¯ïŒ",
"åŸè€ã²ãšãã®åйã®ååã¯ïŒ",
"åŸè€ã²ãšããå å
¥ãããã³ãåã¯ïŒ",
"ã®ã¿ãŒããŒããŒã®æ£äœã¯ïŒ",
"åå€é代ã®é«ªã®è²ã¯ïŒ",
"äŒå°ç¥è¹å€ãéãåŠæ ¡ã®ååã¯ïŒ",
"å±±ç°ãªã§ãŠã®è¶£å³ã¯ïŒ",
"廣äºããããæå±ãããã³ãåã¯ïŒ",
"ã©ã€ãããŠã¹ãSTARRYãã®åºé·ã®ååã¯ïŒ",
"ãŒã£ã¡ã¡ãããæåç¥ã§æ«é²ããæŒå¥æ³ã¯ïŒ",
]
# 質åããã®æèãšå¿çã®çæ
contexts = []
answers = []
for question in questions:
response = query_engine.query(question)
contexts.append([x.node.get_content() for x in response.source_nodes])
answers.append(str(response))(5) ããŒã¿ã»ããã®æºåã
ãquestionããanswerããcontextsãã®3ã«ã©ã ã§ããŒã¿ã»ãããäœæããŸãã
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness
# ããŒã¿ã»ããã®æºå
ds = Dataset.from_dict(
{
"question": questions,
"answer": answers,
"contexts": contexts,
}
)(6) è©äŸ¡ã®å®è¡ã
from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness
# è©äŸ¡
result = evaluate(ds, [answer_relevancy, faithfulness])
print(result){'ragas_score': 0.8578, 'answer_relevancy': 0.8195, 'faithfulness': 0.9000}ä»åã¯ããanswer_relevancyã(åçã質åã®é¢é£æ§)ãšãfaithfulnessã (æèã«å¯Ÿããåçã®äºå®ã®äžè²«æ§) ã®2ã€ã®ã¡ããªãã¯ã¹ã䜿ããŸãããragas_score ã¯ãåã ã®ã¡ããªãã¯ã¹ã³ã¢ã®èª¿åå¹³åã§ãã
3. ã¡ããªãã¯ã¹
3-1. Faithfulness (å¿ å®ã)
ç¹å®ã®æèã«å¯Ÿããåçã®äºå®ã®äžè²«æ§ã枬å®ããŸãã ããã¯ãåçããããã¹ããçæãããããã®åããã¹ããæèã«ç §ãããŠæ€èšŒããããšãå«ããè€æ°ã¹ãããã䜿çšããŠè¡ãããŸãã å€ã®ç¯å²ã¯ 0 ã 1 ã§ãé«ãã»ã©è¯ãã§ãã
from ragas.metrics.factuality import Faithfulness
faithfulness = Faithfulness()
# Dataset({
# features: ['question','contexts','answer'],
# num_rows: 25
# })
dataset: Dataset
results = faithfulness.score(dataset)3-2. ContextRelevancy (æèé¢é£æ§)
æèã質å (ããã³ãã) ã«ã©ã®çšåºŠé¢é£ããŠããããæž¬å®ããŸããããã¯ãOpenAI ã¢ãã«ãšã¯ãã¹ãšã³ã³ãŒããŒã¢ãã«ã®çµã¿åããã䜿çšããŠè¡ãããŸããã¹ã³ã¢ãæ¹åããã«ã¯ãæèã®æ å ±éã調æŽããå¿ èŠããããŸãã
from ragas.metrics.context_relevancy import ContextRelevancy
context_rel = ContextRelevancy(strictness=3)
# Dataset({
# features: ['question','contexts'],
# num_rows: 25
# })
dataset: Dataset
results = context_rel.score(dataset)3-3. Context Recall (æèåŒã³åºã)
泚éä»ãã®åçãã°ã©ãŠã³ããã¥ã«ãŒã¹ãšããŠäœ¿çšããŠãæèã®åçŸçãæž¬å®ããŸãã æ³šéä»ãã®åçã¯ãã°ã©ãŠã³ããã¥ã«ãŒã¹ ã³ã³ããã¹ãã®ä»£çãšããŠäœ¿çšãããŸãã
from ragas.metrics.context_recall import ContextRecall
context_recall = ContextRecall()
# Dataset({
# features: ['contexts','ground_truths'],
# num_rows: 25
# })
dataset: Dataset
results = context_recall.score(dataset)3-4. AnswerRelevancy (åçé¢é£æ§)
åçã質å (ããã³ãã) ã«å¯ŸããŠã©ã®çšåºŠé¢é£ããŠããããæž¬å®ããŸããåçãäžå®å šã§ããããåé·ãªæ å ±ãå«ãŸããŠããå Žåãã¹ã³ã¢ã¯äœããªããŸããããã¯ãåçã䜿çšã㊠LLM ãç¹å®ã®è³ªåãçæããå¯èœæ§ãèšç®ããããšã§å®éåãããŸãã å€ã®ç¯å²ã¯ 0 ã 1 ã§ã倧ããã»ã©è¯ãã§ãã
from ragas.metrics.answer_relevancy import AnswerRelevancy
answer_relevancy = AnswerRelevancy()
# Dataset({
# features: ['question','answer'],
# num_rows: 25
# })
dataset: Dataset
results = answer_relevancy.score(dataset)3-5. AspectCritique (ã¢ã¹ãã¯ãæ¹è©)
Aspect Critiques ã¯ãæäŸãããAspectã䜿çšããŠæåºç©ãè©äŸ¡ãã LLM è©äŸ¡è
ã§ããRagas Critiques ã«ã¯ãæ£ãããæå®³ããªã©ããã€ãã®Aspectãäºåå®çŸ©ãããŠããŸãã ç¬èªã®Aspectãå®çŸ©ããããšãã§ããŸãã
strictness ãã©ã¡ãŒã¿ã¯ãäºæž¬ã«ãããèªå·±äžè²«æ§ã®ã¬ãã« 2 ïœ 4 (çæ³çãªç¯å²) ãä¿èšŒããããã«äœ¿çšãããŸãã
åºåã¯åžžã«ãã€ããªã§ãæåºç©ãç¹å®ã®Aspectå®çŸ©ã«æºæ ããŠãããã©ããã瀺ããŸãããããã®ã¹ã³ã¢ã¯éé£ç¶ã®ãããæçµç㪠ragas_score ã«ã¯åæ ãããŸããã
äºåå®çŸ©ãããŠããAspectã®ãªã¹ãã¯ã次ã®ãšããã§ãã
ã»correctness (æ£ç¢ºã)
ã»harmfulness (æå®³ã)
ã»coherence (äžè²«æ§)
ã»conciseness (ç°¡æœã)
ã»maliciousness (æªæ)
## äºåå®çŸ©ãããåŽé¢ããã§ãã¯
from ragas.metrics.critique import SUPPORTED_ASPECTS
print(SUPPORTED_ASPECTS)
from ragas.metrics.critique import conciseness
from ragas
# Dataset({
# features: ['question','answer'],
# num_rows: 25
# })
dataset: Dataset
results = conciseness.score(dataset)
## èªåã®æ¹è©ãå®çŸ©
from ragas.metrics.critique import AspectCritique
mycritique = AspectCritique(name="my-critique", definition="Is the submission safe to children?", strictness=2)4. å©çšç¶æ³ã®è¿œè·¡
ãragasãã§ã¯ãéåžžã«åºæ¬çãªå©çšç¶æ³ã远跡ããŠããŸããå人ãäŒç€Ÿãç¹å®ã§ãããããªæ å ±ã¯è¿œè·¡ããŸãããåéããããŒã¿ããã¹ãŠãªãŒãã³ãœãŒã¹ã«ããŠããŸã (詳ããã¯ãã¡ããåç §)ãäœããã©ããã³ã°ããŠãããã¯ãã³ãŒãã§ç¢ºèªã§ããŸãã
å©çšç¶æ³ã®è¿œè·¡ãç¡å¹ã«ããã«ã¯ãRAGAS_DO_NOT_TRACK ã«Â true ãæå®ããŠãã ããã
5. FAQ
Q. 調åå¹³åã¯äœãæå³ããïŒ
A. 調åå¹³åã¯ã極端ãªå€ã«ããã«ãã£ãäžããŸããããšãã°ãçæãããåçãæèãšå®å
šã«äºå®ãšäžèŽããŠãã (å¿ å®åºŠ=1) ãã質åãšã¯ç¡é¢ä¿ã§ãã (é¢é£æ§=0) å ŽåãåçŽå¹³åã§ã¯ 0.5 ã®ã¹ã³ã¢ãåŸãããŸãããHarmonic-Meanã§ã¯ 0.0 ã«ãªããŸãã
Q. ragas ã§ãã€ãã©ã€ã³ãæ¹åããã«ã¯ïŒ
A. ãæž¬å®ã¯å¶åŸ¡ã«ã€ãªãããæçµçã«ã¯æ¹åã«ã€ãªããæåã®ã¹ãããã§ãã - James Harrington
ããã§ã¯ãRAGãã€ãã©ã€ã³ã®æºåããã§ã«æŽã£ãŠããããšãåæãšããŠããŸããRAGãã€ãã©ã€ã³ã«ã¯ãäž»ã«ãRetrieverããšãGeneratorãã®2ã€ã®éšåããããŸãããããã®ããããã倿Žãããšããã€ãã©ã€ã³ã®å質ã«ãåæ ãããŸãã