
OpenAIã®ãã¡ã€ã³ãã¥ãŒãã³ã°APIã«ãã GPT-4 ãã GPT-3.5 ãžã®èžçã詊ã
OpenAIã®ãã¡ã€ã³ãã¥ãŒãã³ã°APIã«ãã GPT-4 ãã GPT-3.5 ãžã®èžçã詊ããã®ã§ãŸãšããŸããã
1. GPT-4 ãã GPT-3.5 ãžã®èžç
ãLlamaIndexãã§ãOpenAIã®ãã¡ã€ã³ãã¥ãŒãã³ã°APIã«ãã GPT-4 ãã GPT-3.5 ãžã®èžçã®ColabãæäŸãããŠãã®ã§ããããããŒã¹ã«ç¬èªããŒã¿ã§è©ŠããŠã¿ãŸããã
å ·äœçã«ã¯ããGPT-4ãã§åŠç¿ããŒã¿ãçæãããGPT-3.5ãã§ãã¡ã€ã³ãã¥ãŒãã³ã°ããããšã§ããGPT-3.5ãã«ãGPT-4ãçžåœã®ç¥èãç¿åŸãããŸãã
We successfully made gpt-3.5-turbo output GPT-4 quality responses in an e2e RAG system ð¥
â Jerry Liu (@jerryjliu0) August 23, 2023
Stack: automated training dataset creation in @llama_index + new @OpenAI finetuning + ragas (@Shahules786) eval
We did this in a day â¡ïž
Full Colab notebook here: https://t.co/GryLUrpd85 pic.twitter.com/HZY21mJ0Pg
2. ããã¥ã¡ã³ãã®æºå
ä»åã¯ããã³ã¬ããã£ã¢ã®ããŒã£ã¡ã»ãã»ãã£ãïŒãã®ããããã®ããã¥ã¡ã³ãã䜿ããŸãã
ã»bocchi.txt

3. ã»ããã¢ãã
Colabã§ã®ã»ããã¢ããæé ã¯ã次ã®ãšããã§ãã
(1) ããã±ãŒãžã®ã€ã³ã¹ããŒã«ã
# ããã±ãŒãžã®ã€ã³ã¹ããŒã«
!pip install llama-index sentence-transformers ragas(2) ç°å¢å€æ°ã®æºåã
以äžã®ã³ãŒãã® <OpenAI_APIã®ããŒã¯ã³> ã«ã¯OpenAI APIã®ããŒã¯ã³ãæå®ããŸãã(ææ)
import os
import openai
# ç°å¢å€æ°ã®èšå®
os.environ["OPENAI_API_KEY"] = "<OpenAI_APIã®ããŒã¯ã³>"
openai.api_key = os.environ["OPENAI_API_KEY"](3) Colabã«ããã¥ã¡ã³ããbocchi.txtããã¢ããããŒãã

(4) ããã¥ã¡ã³ãã®èªã¿èŸŒã¿ã
from llama_index import SimpleDirectoryReader
# ããã¥ã¡ã³ãã®èªã¿èŸŒã¿
documents = SimpleDirectoryReader(
input_files=["bocchi.txt"]
).load_data()4. 質åããŒã¿ã®äœæ
Colabã§ã®è³ªåããŒã¿ã»ããã®äœææé ã¯ã次ã®ãšããã§ãã
(1) ãµãŒãã¹ã³ã³ããã¹ãã®æºåã
ä»åã®ããã¥ã¡ã³ã圢åŒã«ããããŠãparagraph_separator 㫠"\n\n" ãæå®ããŸããã"\n\n" ã§æ®µèœå¥ãããèšå® (ããã©ã«ãã¯"\n\n\n") ã«ãªããŸãã
from llama_index import ServiceContext
from llama_index.llms import OpenAI
from llama_index.text_splitter import SentenceSplitter
from llama_index.node_parser import SimpleNodeParser
# ããŒãããŒãµãŒã®æºå
text_splitter = SentenceSplitter(
paragraph_separator="\n\n",
)
node_parser = SimpleNodeParser.from_defaults(text_splitter=text_splitter)
# ãµãŒãã¹ã³ã³ããã¹ãã®æºå
gpt_35_context = ServiceContext.from_defaults(
llm=OpenAI(model="gpt-3.5-turbo", temperature=0.3),
node_parser=node_parser,
)(2) 質åããŒã¿äœæã¯ãšãªãšããŒã¿ã»ãããžã§ãã¬ãŒã¿ã®æºåã
ä»åã®ããã¥ã¡ã³ã圢åŒã«ããããŠãã¯ãšãªã¯æ¥æ¬èªã«ããŸããã
from llama_index.evaluation import DatasetGenerator
# 質åããŒã¿äœæã¯ãšãªã®æºå
question_gen_query = (
"ããªãã¯æåž«ã§ãã ããªãã®ä»äºã¯ã詊éšåé¡ãäœæããããšã§ãã"
"æ¥æ¬ã®ã¢ãã¡äœåã«æèããéèŠãªäºå®ãæãã1ã€ã®è³ªåãã以äžã®æ¡ä»¶ã§äœæããŸãã"
"- 質åã¯å¿
ãæäŸãããæèã«éå®"
"- 質åã¯å¿
ãæ¥æ¬èªã§èšè¿°"
"- 代åè©ã¯çµ¶å¯Ÿã«äœ¿çšããªã"
)
# ããŒã¿ã»ãããžã§ãã¬ãŒã¿ã®æºå
dataset_generator = DatasetGenerator.from_documents(
documents,
question_gen_query=question_gen_query,
service_context=gpt_35_context,
)(3) 質åããŒã¿ã®äœæã
ããŒã¿ã»ãããžã§ãã¬ãŒã¿ãå®è¡ããŸãã80åã®è³ªåãäœæããŠããŸãã10åã»ã©æéããããŸããã
# 質åããŒã¿ã®äœæ
questions = dataset_generator.generate_questions_from_nodes(num=80)
print("Generated ", len(questions), " questions")(4) 質åããŒã¿ã®ã·ã£ããã«ã
import random
# ã·ã£ããã«
random.seed(42)
random.shuffle(questions)(5) 質åããŒã¿ã®ä¿åã
åŠç¿çšã®ãtrain_questions.txtããšè©äŸ¡çšã®ãeval_questions.txtãã«40質åãã€ä¿åããŸããã
# åŠç¿çšã®è³ªåããŒã¿ã®ä¿å
with open("train_questions.txt", "w") as f:
for question in questions[:40]:
f.write(question + "\n")
# è©äŸ¡çšã®è³ªåããŒã¿ã®ä¿å
with open("eval_questions.txt", "w") as f:
for question in questions[40:]:
f.write(question + "\n")(6) 質åããŒã¿ã確èªããŠã代åè©ãªã©è³ªåãšããŠæç«ããªãéšåãä¿®æ£ã

5. ãã¡ã€ã³ãã¥ãŒãã³ã°åã®GPT-3.5 ã®è©äŸ¡
ãã¡ã€ã³ãã¥ãŒãã³ã°åã®GPT-3.5ã§ã®è³ªåå¿çã®è©äŸ¡ãèšæž¬ããŸãã
(1) è©äŸ¡çšã®è³ªåããŒã¿ã®èªã¿èŸŒã¿ã
# è©äŸ¡çšã®è³ªåããŒã¿ã®èªã¿èŸŒã¿
questions = []
with open("eval_questions.txt", "r") as f:
for line in f:
questions.append(line.strip())(2) ãµãŒãã¹ã³ã³ããã¹ãã®æºåã
# ããŒãããŒãµãŒã®æºå
text_splitter = SentenceSplitter(
paragraph_separator="\n\n",
)
node_parser = SimpleNodeParser.from_defaults(text_splitter=text_splitter)
# ãµãŒãã¹ã³ã³ããã¹ãã®æºå
gpt_35_context = ServiceContext.from_defaults(
llm=OpenAI(model="gpt-3.5-turbo", temperature=0.3),
node_parser=node_parser,
context_window=2048,
)(3) ã€ã³ããã¯ã¹ãšã¯ãšãªãšã³ãžã³ã®æºåã
from llama_index import VectorStoreIndex
# ã€ã³ããã¯ã¹ãšã¯ãšãªãšã³ãžã³ã®æºå
index = VectorStoreIndex.from_documents(
documents,
service_context=gpt_35_context
)
query_engine = index.as_query_engine(
similarity_top_k=2
)(4) 質åããã®æèãšå¿çã®çæã
# 質åããã®æèãšå¿çã®çæ
contexts = []
answers = []
for question in questions:
response = query_engine.query(question)
contexts.append([x.node.get_content() for x in response.source_nodes])
answers.append(str(response))(5) çæããæèãšå¿çã確èªã
print("questions[0]:", questions[0])
print("contexts[0]:", contexts[0])
print("answers[0]:", answers[0])questions[0]: åŸè€ã²ãšãã¯ãªãåé¡è¡åãèµ·ãããŠããã®ã§ããïŒ
contexts[0]: ['åŸè€ã²ãšã(ããšãã²ãšã)\nç§è¯é«æ ¡ã«éã女åãæ¡è²ã®é«ªãç¡é äœã«äŒžã°ãããã€ããžã£ãŒãžã身ã«ã€ããŠãããèªä»å
±ã«èªããåŒããããäžæ©æåã®ãé°ãã£ãã§ãæ¿èªæ¬²æ±ã人äžå匷ãã«ãããããããèç
ãªæ§æ Œã§äººãšæ¥ããã®ã極床ã«èŠæãšããŠããããã®ãããããã«èªåã®äžçã«å
¥ã£ãŠèœã¡èŸŒããšããæ
ç·äžå®å®ããèŠãããæŒãå
¥ãããã³ããŒã«ã«æœã蟌ãçãããããå®çãã³ãŽãŒä»®é¢ããæã¡æ©ããŠãããéåãå匷ãèŠæã§ãç¹ã«å匷ã¯ãŸããã«ææ¥ãåããŠããã«ãããããããçæ¥ã®èŠé ã®æªãããèµ€ç¹ã®ãªã®ãªãèªåã«ã¯äœã®åãæããªãã®ãçæããŠãããããäžåŠã®é ã«æãæ§æ Œã®äººéããã³ãããã£ãŠäººæ°è
ã«ãªã£ãã€ã³ã¿ãã¥ãŒãèªãã§ãã®ã¿ãŒãå§ãããæ¯æ¥ç·Žç¿ãããé°ã§ã®ã¿ãŒã®è
åã¯ããçŽã«ãªã£ãããçµå±ããã®è
åãæ«é²ããå Žã«æµãŸãããåéãäœããªããŸãŸäžåŠã忥ãçŸåšã¯ãã®ã¿ãŒããŒããŒãã®åã§åç»é
ä¿¡ããŠãããã®ã¿ãªã¹ããæ¢ããŠããäŒå°ç¥è¹å€ã«èªããããçµæãã³ããã«å å
¥ãããã³ã掻åãå§ããããã³ãå
ã§ã¯ã®ã¿ãªã¹ãå
Œäœè©ãæ
åœããŠãããå±±ç°ãªã§ãŠã«ããŒã£ã¡ãã®æç§°ãä»ããããŠä»¥éã¯ããã®æç§°ã§åŒã°ãããã®ã¿ãŒã®è
åã¯ããªãã®ãã®ã ããä»äººãšåãããã»ãã·ã§ã³ã®çµéšãçç¡ãªããããã³ãã§ã¯ãŸãšãã«æŒå¥ã§ããªãã§ããã', 'çµæãã³ã\nåŸè€ã²ãšãã¯åéãäœããªãé°ãã£ã§ãã€ãäžäººã§éãããŠããããäžåп代ã«ãã¬ãã®ã€ã³ã¿ãã¥ãŒãèŠãŠãé°ãã£ã§ããã³ããçµãã§ããã°äººæ°è
ã«ãªãããšèããã®ã¿ãŒã®ç·Žç¿ãå§ãããã²ãšãã¯æ¯æ¥ã³ãã³ãç·Žç¿ãéããããšã§ã®ã¿ãŒã®è
åã¯ããªãã®ãã®ãšãªã£ãŠããããæ°ãã€ãã°äžåŠã¯åæ¥ãçµå±ãåéãäœããšããåœåã®ç®æšã¯éæããããšã¯ã§ããããŸãã ã£ãããããŠé«æ ¡ã«å
¥åŠããã²ãšãã¯ã®ã¿ãŒæŒå¥ãåç»é
ä¿¡ãããã®ã¿ãŒããŒããŒããšããŠãããã§ããããã®äººæ°ãéããŠããããçŸå®ã§ã¯å€ãããåéãäœããã«ãããã²ãšãã¯ã®ã¿ãŒãããã¿ãããã«æã¡ãã®ã¿ãªã¹ãã ãšããããšãã¢ããŒã«ãããã®ã®å¹æã¯ãªãã黿ïŒããããïŒãŠãããšãããã®ã¿ãªã¹ããæ¢ããŠããäŒå°ç¥è¹å€ã«èªããããã³ã掻åãããããšãšãªããã²ãšãã¯è¹å€ãå±±ç°ãªã§ãŠããšå
±ã«ãçµæãã³ãããšããŠæŽ»åãå§ããããåéã¯æ¬²ãããã©ã³ãã¥çç¬ããŒã£ã¡ã®ã²ãšãã¯åé¡è¡åã°ããèµ·ãããŠãããçµæãã³ããªã®ã«ãçµæåãŒããªåžå¹ãã³ã掻åã¯å¹éãããã®ã ã£ãããããŠãããªã²ãšãã«ãã¯ã©ã¹ã¡ã€ãã®åå€é代ã話ããããŠããŠã圌女ãçµæãã³ãã®å
ã¡ã³ããŒã ã£ãããšãç¥ããçŽäœïŒããïŒæ²æã®æ«ãé代ã¯ã²ãšããšã®äº€æµã§çµæãã³ãã«æ»ãããšã«ãªããã²ãšããã¡ã¯ã©ã€ãã«åããŠæºåãããããšãšãªããã²ãšãã¯ã©ã€ããã±ããã®ãã«ããããªããããæ¹ã
ãåã£ãŠãã±ããã売ãããšã«ãªããã人èŠç¥ãã®ã²ãšãã«ã¯é£é¡ã§ããã«å£ã«ã¶ã¡åœãã£ãŠããŸããçµ¶æããã²ãšãã¯ãããã§è¡ãåãã廣äºããããšééã圌女ã®èšããã§çªçºçãªè·¯äžã©ã€ããè¡ãªã£ãŠã芳客ãé
äºããããããã®ãé°ã§ã©ã€ããã±ããã®ãã«ããããªããçµæãã³ãã¯ã©ã€ãã«èšãã®ã ã£ãã']
answers[0]: åŸè€ã²ãšãã¯åé¡è¡åãèµ·ãããŠããçç±ã¯ãèªå·±è¯å®æã®äœããšäººãšã®é¢ãããèŠæãšããŠããæ§æ Œã«èµ·å ããŠããŸãã圌女ã¯åŒããããã®äžæ©æåã§ãããæ¿èªæ¬²æ±ãéåžžã«åŒ·ãäžæ¹ã§ãèç
ãªæ§æ Œã®ããäººãšæ¥ããããšãèŠæã§ãããã®ãããèªåã®äžçã«å
¥ã£ãŠèœã¡èŸŒãããšãå€ããæ
ç·äžå®å®ããèŠããããšããããŸãããŸããåœŒå¥³ã¯æŒãå
¥ãããã³ããŒã«ã«æœã蟌ãçããããèªåã®å±
å ŽæãèŠã€ããããã«ãå®çãã³ãŽãŒä»®é¢ããæã¡æ©ããŠããŸãããããã®èŠçŽ ã圌女ã®åé¡è¡åã«ã€ãªãã£ãŠãããšèããããŸãã(6) è©äŸ¡ã®å®è¡ã
è©äŸ¡ã«ã¯è©äŸ¡ã©ã€ãã©ãªãRagasãã䜿çšããŸãããRagasãã«ã¯ RAG ãã€ãã©ã€ã³ã®è©äŸ¡ã¡ããªã¯ã¹ã倧éã«ãããŸãã詳ããã¯ãããã¥ã¡ã³ããåç
§ã
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness
# è©äŸ¡ã®å®è¡
ds = Dataset.from_dict(
{
"question": questions,
"answer": answers,
"contexts": contexts,
}
)
result = evaluate(ds, [answer_relevancy, faithfulness])
print(result)evaluating with [answer_relevancy]
100%|ââââââââââ| 3/3 [01:37<00:00, 32.41s/it]
evaluating with [faithfulness]
100%|ââââââââââ| 3/3 [15:18<00:00, 306.21s/it]
{'ragas_score': 0.7634, 'answer_relevancy': 0.8399, 'faithfulness': 0.6997}ä»åã¯ã次ã®2ã€ã®ã¡ããªãã¯ã¹ã䜿ããŸããã
ã»answer_relevancy
çæãããåçãããã³ããã«å¯ŸããŠã©ã®çšåºŠé¢é£ããŠããããæž¬å®ããŸããçæãããåçãäžå®å šã§ããããåé·ãªæ å ±ãå«ãŸããŠããå Žåãã¹ã³ã¢ã¯äœããªããŸããããã¯ãçæãããåçã䜿çšã㊠LLM ãç¹å®ã®è³ªåãçæããå¯èœæ§ãèšç®ããããšã§å®éåãããŸãã
å€ã®ç¯å²ã¯ (0,1) ã§ã倧ããã»ã©åªããŠããŸãã
ã»faithfulness
æå®ãããã³ã³ããã¹ãã«å¯Ÿãããçæãããåçã®äºå®ã®äžè²«æ§ã枬å®ããŸããããã¯ãçæãããåçããã¹ããŒãã¡ã³ããäœæãããã®åŸããããã®åã¹ããŒãã¡ã³ããã³ã³ããã¹ãã«ç §ãããŠæ€èšŒããããšãå«ããè€æ°ã®ã¹ãããã®ãã©ãã€ã ã䜿çšããŠè¡ãããŸãã
å€ã®ç¯å²ã¯ (0,1) ã§ã倧ããã»ã©åªããŠããŸãã
6. ãã¡ã€ã³ãã¥ãŒãã³ã°ã®åŠç¿ããŒã¿ã®äœæ
Coladã§ã®ãã¡ã€ã³ãã¥ãŒãã³ã°ã®åŠç¿ããŒã¿ã®äœææé ã¯ã次ã®ãšããã§ãã
(1) ãµãŒãã¹ã³ã³ããã¹ãã®æºåã
ãã¡ã€ã³ãã¥ãŒãã³ã°ã®åŠç¿ããŒã¿ãåéããããã®ãã¡ã€ã³ãã¥ãŒãã³ã°ãã³ãã©ãŒãã³ãŒã«ããã¯ãããŒãžã£çµç±ã§èšå®ããŸãã
from llama_index import ServiceContext
from llama_index.llms import OpenAI
from llama_index.text_splitter import SentenceSplitter
from llama_index.node_parser import SimpleNodeParser
from llama_index.callbacks import OpenAIFineTuningHandler
from llama_index.callbacks import CallbackManager
# ã³ãŒã«ããã¯ãããŒãžã£ãšãã¡ã€ã³ãã¥ãŒãã³ã°ãã³ãã©ãŒã®æºå
finetuning_handler = OpenAIFineTuningHandler()
callback_manager = CallbackManager([finetuning_handler])
# ããŒãããŒãµãŒã®æºå
text_splitter = SentenceSplitter(
paragraph_separator="\n\n",
)
node_parser = SimpleNodeParser.from_defaults(text_splitter=text_splitter)
# ãµãŒãã¹ã³ã³ããã¹ãã®æºå
gpt_4_context = ServiceContext.from_defaults(
llm=OpenAI(model="gpt-4", temperature=0.3),
node_parser=node_parser,
context_window=2048, # ã³ã³ããã¹ãé·ãå¶éããŠRefineãä¿ã
callback_manager=callback_manager, # ã³ãŒã«ããã¯ãããŒãžã£
)(2) åŠç¿çšã®è³ªåããŒã¿ã®èªã¿èŸŒã¿ã
# åŠç¿çšã®è³ªåããŒã¿ã®èªã¿èŸŒã¿
questions = []
with open("train_questions.txt", "r") as f:
for line in f:
questions.append(line.strip())(3) ã€ã³ããã¯ã¹ãšã¯ãšãªãšã³ãžã³ã®æºåã
from llama_index import VectorStoreIndex
# ã€ã³ããã¯ã¹ãšã¯ãšãªãšã³ãžã³ã®æºå
index = VectorStoreIndex.from_documents(
documents,
service_context=gpt_4_context
)
query_engine = index.as_query_engine(
similarity_top_k=2
)(4) 質åããã®ãã¡ã€ã³ãã¥ãŒãã³ã°ã®åŠç¿ããŒã¿ã®äœæã
ãã¡ã€ã³ãã¥ãŒãã³ã°çšã®åŠç¿ããŒã¿ãfinetuning_events.jsonlããçæãããŸãã
# 質åããã®ãã¡ã€ã³ãã¥ãŒãã³ã°ã®åŠç¿ããŒã¿ã®äœæ
for question in questions:
response = query_engine.query(question)
# ãã¡ã€ã³ãã¥ãŒãã³ã°ã®åŠç¿ããŒã¿ã®ä¿å
finetuning_handler.save_finetuning_events("finetuning_events.jsonl")(5) æ¥æ¬èªãèªããããã«å€æã
ãã¡ã€ã³ãã¥ãŒãã³ã°çšã®åŠç¿ããŒã¿ãfinetuning_events_ja.jsonlããçæãããŸãã倿ããªããšæåã³ãŒããªã®ã§äººéãèªããŸããã
import json
# æ¥æ¬èªãèªããããã«å€æ
with open("finetuning_events.jsonl", "r", encoding="utf-8") as f:
lines = f.readlines()
data = [json.loads(line) for line in lines]
with open("finetuning_events_ja.jsonl", "w", encoding="utf-8") as f:
for item in data:
json.dump(item, f, ensure_ascii=False)
f.write("\n")
7. ãã¡ã€ã³ãã¥ãŒãã³ã°ã®å®è¡
Colabã§ã®ãã¡ã€ã³ãã¥ãŒãã³ã°ã®å®è¡æé ã¯ã次ã®ãšããã§ãã
(1) åŠç¿ããŒã¿ã®æ€èšŒã®å®è¡ã
OpenAI APIã®ããã¥ã¡ã³ãã§ç޹ä»ãããŠããã¹ã¯ãªãã (Data formatting script) ãã³ããŒ&ããŒã¹ãããŠãdata_path ã« "finetuning_events_ja.jsonl" ãæå®ããŸãã
# We start by importing the required packages
import json
import os
import tiktoken
import numpy as np
from collections import defaultdict
# Next, we specify the data path and open the JSONL file
data_path = "finetuning_events_ja.jsonl"
# Load dataset
with open(data_path) as f:
dataset = [json.loads(line) for line in f]
# We can inspect the data quickly by checking the number of examples and the first item
# Initial dataset stats
print("Num examples:", len(dataset))
print("First example:")
for message in dataset[0]["messages"]:
print(message)
# Now that we have a sense of the data, we need to go through all the different examples and check to make sure the formatting is correct and matches the Chat completions message structure
# Format error checks
format_errors = defaultdict(int)
for ex in dataset:
if not isinstance(ex, dict):
format_errors["data_type"] += 1
continue
messages = ex.get("messages", None)
if not messages:
format_errors["missing_messages_list"] += 1
continue
for message in messages:
if "role" not in message or "content" not in message:
format_errors["message_missing_key"] += 1
if any(k not in ("role", "content", "name") for k in message):
format_errors["message_unrecognized_key"] += 1
if message.get("role", None) not in ("system", "user", "assistant"):
format_errors["unrecognized_role"] += 1
content = message.get("content", None)
if not content or not isinstance(content, str):
format_errors["missing_content"] += 1
if not any(message.get("role", None) == "assistant" for message in messages):
format_errors["example_missing_assistant_message"] += 1
if format_errors:
print("Found errors:")
for k, v in format_errors.items():
print(f"{k}: {v}")
else:
print("No errors found")
# Beyond the structure of the message, we also need to ensure that the length does not exceed the 4096 token limit.
# Token counting functions
encoding = tiktoken.get_encoding("cl100k_base")
# not exact!
# simplified from https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
def num_tokens_from_messages(messages, tokens_per_message=3, tokens_per_name=1):
num_tokens = 0
for message in messages:
num_tokens += tokens_per_message
for key, value in message.items():
num_tokens += len(encoding.encode(value))
if key == "name":
num_tokens += tokens_per_name
num_tokens += 3
return num_tokens
def num_assistant_tokens_from_messages(messages):
num_tokens = 0
for message in messages:
if message["role"] == "assistant":
num_tokens += len(encoding.encode(message["content"]))
return num_tokens
def print_distribution(values, name):
print(f"\n#### Distribution of {name}:")
print(f"min / max: {min(values)}, {max(values)}")
print(f"mean / median: {np.mean(values)}, {np.median(values)}")
print(f"p5 / p95: {np.quantile(values, 0.1)}, {np.quantile(values, 0.9)}")
# Last, we can look at the results of the different formatting operations before proceeding with creating a fine-tuning job:
# Warnings and tokens counts
n_missing_system = 0
n_missing_user = 0
n_messages = []
convo_lens = []
assistant_message_lens = []
for ex in dataset:
messages = ex["messages"]
if not any(message["role"] == "system" for message in messages):
n_missing_system += 1
if not any(message["role"] == "user" for message in messages):
n_missing_user += 1
n_messages.append(len(messages))
convo_lens.append(num_tokens_from_messages(messages))
assistant_message_lens.append(num_assistant_tokens_from_messages(messages))
print("Num examples missing system message:", n_missing_system)
print("Num examples missing user message:", n_missing_user)
print_distribution(n_messages, "num_messages_per_example")
print_distribution(convo_lens, "num_total_tokens_per_example")
print_distribution(assistant_message_lens, "num_assistant_tokens_per_example")
n_too_long = sum(l > 4096 for l in convo_lens)
print(f"\n{n_too_long} examples may be over the 4096 token limit, they will be truncated during fine-tuning")
# Pricing and default n_epochs estimate
MAX_TOKENS_PER_EXAMPLE = 4096
MIN_TARGET_EXAMPLES = 100
MAX_TARGET_EXAMPLES = 25000
TARGET_EPOCHS = 3
MIN_EPOCHS = 1
MAX_EPOCHS = 25
n_epochs = TARGET_EPOCHS
n_train_examples = len(dataset)
if n_train_examples * TARGET_EPOCHS < MIN_TARGET_EXAMPLES:
n_epochs = min(MAX_EPOCHS, MIN_TARGET_EXAMPLES // n_train_examples)
elif n_train_examples * TARGET_EPOCHS > MAX_TARGET_EXAMPLES:
n_epochs = max(MIN_EPOCHS, MAX_TARGET_EXAMPLES // n_train_examples)
n_billing_tokens_in_dataset = sum(min(MAX_TOKENS_PER_EXAMPLE, length) for length in convo_lens)
print(f"Dataset has ~{n_billing_tokens_in_dataset} tokens that will be charged for during training")
print(f"By default, you'll train for {n_epochs} epochs on this dataset")
print(f"By default, you'll be charged for ~{n_epochs * n_billing_tokens_in_dataset} tokens")
print("See pricing page to estimate total costs")Num examples: 73
First example:
{'role': 'system', 'content': "You are an expert Q&A system that is trusted around the world.\nAlways answer the query using the provided context information, and not prior knowledge.\nSome rules to follow:\n1. Never directly reference the given context in your answer.\n2. Avoid statements like 'Based on the context, ...' or 'The context information ...' or anything along those lines."}
{'role': 'user', 'content': 'Context information is below.\n---------------------\nåŸè€ã²ãšã(ããšãã²ãšã)\nç§è¯é«æ ¡ã«éã女åãæ¡è²ã®é«ªãç¡é äœã«äŒžã°ãããã€ããžã£ãŒãžã身ã«ã€ããŠãããèªä»å
±ã«èªããåŒããããäžæ©æåã®ãé°ãã£ãã§ãæ¿èªæ¬²æ±ã人äžå匷ãã«ãããããããèç
ãªæ§æ Œã§äººãšæ¥ããã®ã極床ã«èŠæãšããŠããããã®ãããããã«èªåã®äžçã«å
¥ã£ãŠèœã¡èŸŒããšããæ
ç·äžå®å®ããèŠãããæŒãå
¥ãããã³ããŒã«ã«æœã蟌ãçãããããå®çãã³ãŽãŒä»®é¢ããæã¡æ©ããŠãããéåãå匷ãèŠæã§ãç¹ã«å匷ã¯ãŸããã«ææ¥ãåããŠããã«ãããããããçæ¥ã®èŠé ã®æªãããèµ€ç¹ã®ãªã®ãªãèªåã«ã¯äœã®åãæããªãã®ãçæããŠãããããäžåŠã®é ã«æãæ§æ Œã®äººéããã³ãããã£ãŠäººæ°è
ã«ãªã£ãã€ã³ã¿ãã¥ãŒãèªãã§ãã®ã¿ãŒãå§ãããæ¯æ¥ç·Žç¿ãããé°ã§ã®ã¿ãŒã®è
åã¯ããçŽã«ãªã£ãããçµå±ããã®è
åãæ«é²ããå Žã«æµãŸãããåéãäœããªããŸãŸäžåŠã忥ãçŸåšã¯ãã®ã¿ãŒããŒããŒãã®åã§åç»é
ä¿¡ããŠãããã®ã¿ãªã¹ããæ¢ããŠããäŒå°ç¥è¹å€ã«èªããããçµæãã³ããã«å å
¥ãããã³ã掻åãå§ããããã³ãå
ã§ã¯ã®ã¿ãªã¹ãå
Œäœè©ãæ
åœããŠãããå±±ç°ãªã§ãŠã«ããŒã£ã¡ãã®æç§°ãä»ããããŠä»¥éã¯ããã®æç§°ã§åŒã°ãããã®ã¿ãŒã®è
åã¯ããªãã®ãã®ã ããä»äººãšåãããã»ãã·ã§ã³ã®çµéšãçç¡ãªããããã³ãã§ã¯ãŸãšãã«æŒå¥ã§ããªãã§ããã\n\nã²ãšãã®ç¶(ã²ãšãã®ã¡ã¡)\nåŸè€ã²ãšãã®ç¶èŠªãæ¬åã¯äžæã§ãçŽ é¡ããªããæ¯åæããããŠããªãããã«äžæãè¥ãé ãã®ã¿ãŒã匟ããŠããããããã²ãšãã®äžåп代ãã®ã¿ãŒã«èå³ãèŠããåšã«èªåã®ã®ã¿ãŒã貞ãããåç»é
ä¿¡ã®ã¢ã«ãŠã³ããå®¶æã§å
±æããŠãããããã²ãšãããã®ã¿ãŒããŒããŒããšããŠäººæ°ãéããŠããããšãç¥ã£ãŠãããã²ãšãã®ããã«åºååå
¥ã貯éããŠãããã®ã¡ã«ããã®ãéãæ°ããã®ã¿ãŒã賌å
¥ããè³éãšããŠæž¡ããŠãããå€ãªç¹æãæ°å€ãæã£ãŠãããé ã身ã®è¡ã§å£ã«é ããŠããããå®çãã³ãŽãŒä»®é¢ãåã£ããã¶ã€ã³ã«æ¹é ãããããŠãããåŸè€ãµããããå®¶æããšã©ã«ããŒã®äžã§æäžäœã«çœ®ãããŠããããªããšããããæœåããããšèããŠããã\n\n2å·(ã«ãã)\n---------------------\nGiven the context information and not prior knowledge, answer the query.\nQuery: åŸè€ã²ãšãã¯ã©ã®ãããªå®¹å§¿ãããŠããã®ã§ããïŒ\nAnswer: '}
{'role': 'assistant', 'content': 'åŸè€ã²ãšãã¯æ¡è²ã®é«ªãç¡é äœã«äŒžã°ããŠããããã€ããžã£ãŒãžã身ã«ã€ããŠããŸãã'}
No errors found
Num examples missing system message: 33
Num examples missing user message: 0
#### Distribution of num_messages_per_example:
min / max: 2, 3
mean / median: 2.547945205479452, 3.0
p5 / p95: 2.0, 3.0
#### Distribution of num_total_tokens_per_example:
min / max: 403, 1397
mean / median: 1010.1506849315068, 1005.0
p5 / p95: 793.4, 1290.2
#### Distribution of num_assistant_tokens_per_example:
min / max: 19, 249
mean / median: 83.75342465753425, 76.0
p5 / p95: 26.0, 161.79999999999998
0 examples may be over the 4096 token limit, they will be truncated during fine-tuning
Dataset has ~73741 tokens that will be charged for during training
By default, you'll train for 3 epochs on this dataset
By default, you'll be charged for ~221223 tokens
See pricing page to estimate total costs(2) åŠç¿ããŒã¿ã®ã¢ããããŒãã
import openai
# åŠç¿ããŒã¿ã®ã¢ããããŒã
openai.File.create(
file=open("finetuning_events_ja.jsonl", "rb"),
purpose="fine-tune"
)<File file id=file-XXXXXXXXXXXXXXXXXXXXXXXX at 0x7ea2aeb29e40> JSON: {
"object": "file",
"id": "file-XXXXXXXXXXXXXXXXXXXXXXXX",
"purpose": "fine-tune",
"filename": "file",
"bytes": 217663,
"created_at": 1693039402,
"status": "uploaded",
"status_details": null
}ãã¡ã€ã«ID (id) ãã¡ã¢ããŸãã
(3) ãã¡ã€ã³ãã¥ãŒãã³ã°ã®å®è¡ã
training_file ã«ã¯ãã¢ããããŒããããã¡ã€ã«ã®ãã¡ã€ã«ID (id) ãæå®ããŸãã
import os
# ãã¡ã€ã³ãã¥ãŒãã³ã°ã®å®è¡
openai.FineTuningJob.create(
training_file="file-XXXXXXXXXXXXXXXXXXXXXXXX",
model="gpt-3.5-turbo"
)<FineTuningJob fine_tuning.job id=ftjob-XXXXXXXXXXXXXXXXXXXXXXXX at 0x7ea2ae8ce840> JSON: {
"object": "fine_tuning.job",
"id": "ftjob-XXXXXXXXXXXXXXXXXXXXXXXX",
"model": "gpt-3.5-turbo-0613",
"created_at": 1692777596,
"finished_at": null,
"fine_tuned_model": null,
"organization_id": "org-XXXXXXXXXXXXXXXXXXXXXXXX",
"result_files": [],
"status": "created",
"validation_file": null,
"training_file": "file-XXXXXXXXXXXXXXXXXXXXXXXX",
"hyperparameters": {
"n_epochs": 3
},
"trained_tokens": null
}ãã¡ã€ã³ãã¥ãŒãã³ã°ã®ãžã§ãID (id) ãã¡ã¢ããŸãã
(4) ã¡ãŒã«ã§ãã¡ã€ã³ãã¥ãŒãã³ã°å®äºã®éç¥ãåŸ
ã€ã
次ã®ãããªãã¡ã€ã³ãã¥ãŒãã³ã°å®äºã®ã¡ãŒã«ãéç¥ãããŸãããžã§ãIDãšã¢ãã«IDãèšèŒãããŠããŸãã

8. ãã¡ã€ã³ãã¥ãŒãã³ã°åŸã®GPT-3.5ã®è©äŸ¡
ãã¡ã€ã³ãã¥ãŒãã³ã°åŸã®GPT-3.5ã§ã®è³ªåå¿çã®è©äŸ¡ãèšæž¬ããŸãã
(1) ã¢ãã«IDã®æºåã
以äžã® <ã¢ãã«ID> ã«èªåã®ã¢ãã«IDãèšè¿°ããŸãã
# ã¢ãã«IDã®æºå
ft_model_name = "<ã¢ãã«ID>"(2) ãµãŒãã¹ã³ã³ããã¹ãã®æºåã
from llama_index import ServiceContext
from llama_index.llms import OpenAI
from llama_index.text_splitter import SentenceSplitter
from llama_index.node_parser import SimpleNodeParser
# ããŒãããŒãµãŒã®æºå
text_splitter = SentenceSplitter(
paragraph_separator="\n\n",
)
node_parser = SimpleNodeParser.from_defaults(text_splitter=text_splitter)
# ãµãŒãã¹ã³ã³ããã¹ãã®æºå
ft_context = ServiceContext.from_defaults(
llm=OpenAI(model=ft_model_name, temperature=0.3),
node_parser=node_parser,
context_window=2048, # ã³ã³ããã¹ãé·ãå¶éããŠRefineãä¿ã
)(3) è©äŸ¡çšã®è³ªåããŒã¿ã®èªã¿èŸŒã¿ã
# è©äŸ¡çšã®è³ªåããŒã¿ã®èªã¿èŸŒã¿
questions = []
with open("eval_questions.txt", "r") as f:
for line in f:
questions.append(line.strip())(4) ã€ã³ããã¯ã¹ãšã¯ãšãªãšã³ãžã³ã®æºåã
from llama_index import VectorStoreIndex
# ã€ã³ããã¯ã¹ãšã¯ãšãªãšã³ãžã³ã®æºå
index = VectorStoreIndex.from_documents(
documents,
service_context=ft_context
)
query_engine = index.as_query_engine(
similarity_top_k=2
)(5) 質åããã®æèãšå¿çã®äœæã
# 質åããã®æèãšå¿çã®äœæ
contexts = []
answers = []
for question in questions:
response = query_engine.query(question)
contexts.append([x.node.get_content() for x in response.source_nodes])
answers.append(str(response))(6) è©äŸ¡ã®å®è¡ã
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness
# è©äŸ¡ã®å®è¡
ds = Dataset.from_dict(
{
"question": questions,
"answer": answers,
"contexts": contexts,
}
)
result = evaluate(ds, [answer_relevancy, faithfulness])
print(result)evaluating with [answer_relevancy]
100%|ââââââââââ| 3/3 [01:43<00:00, 34.42s/it]
evaluating with [faithfulness]
100%|ââââââââââ| 3/3 [12:11<00:00, 243.95s/it]
{'ragas_score': 0.8152, 'answer_relevancy': 0.8274, 'faithfulness': 0.8035}9. è©äŸ¡ã®æ¯èŒ
ãã¡ã€ã³ãã¥ãŒãã³ã°åããåŸã®æ¹ããfaithfulnessãå¢ããŸãããanswer_relevancyã¯ã»ãŒå€åãªãã§ããæäœãGPT-4颚å³ã®äžå¯§ãªèšãåãã«ãªã£ãŠãæ°ãããŸãã
ã»ãã¡ã€ã³ãã¥ãŒãã³ã°å
{'ragas_score': 0.7634, 'answer_relevancy': 0.8399, 'faithfulness': 0.6997}Q: ãŒã£ã¡ã¡ããã¯ã©ããªæ§æ ŒïŒ
A: ãŒã£ã¡ã¡ããã¯åŒããããäžæ©æåã®ãé°ãã£ãã§ãèç ãªæ§æ Œã§äººãšæ¥ããã®ã極床ã«èŠæãšããŠããŸãããŸããèªåã«ã¯äœã®åãæããªãã®ãçæããŠãããæ ç·äžå®å®ããèŠããããšããããŸãã
ã»ãã¡ã€ã³ãã¥ãŒãã³ã°åŸ
{'ragas_score': 0.8152, 'answer_relevancy': 0.8274, 'faithfulness': 0.8035}Q: ãŒã£ã¡ã¡ããã¯ã©ããªæ§æ ŒïŒ
A: ãŒã£ã¡ã¡ããã¯èç ãªæ§æ Œã§ãäººãšæ¥ããã®ã極床ã«èŠæãšããŠããŸãããŸããèªåã«ã¯äœã®åãæããªããšæããŠãããèªå·±è¯å®æãäœãããã§ããããããã®ã¿ãŒã®è åã¯ããçŽã§ãããã®ã¿ãŒããŒããŒãšããŠåç»é ä¿¡ããŠããŸãã
ãããŸãã 倱æãããã¿ãŒã³
å ã®ColabãããŒã¿ã»ãã倿Žããã®ã¿ã§ã¯ãè©äŸ¡ã¯åäžããŸããã§ãããåå ã¯æ¬¡ã®3ã€ã§ãããå ã®è³ªåå¿çã®ç²ŸåºŠã調æŽããŠãããªãã£ãããããã¡ã€ã³ãã¥ãŒãã³ã°ä»¥åã«ãè©äŸ¡ãäžå®å®ã«ãªã£ãŠãããã§ãã
ã»ãã£ã³ã¯ã段èœåäœã§åå²ãããŠããªãã£ã
ä»åã®ããŒã¿ã»ããã®æ®µèœã¯ "\n\n" ã§åå²ããŠãã®ã§ãparagraph_separator="\n\n"ããæå®ããŸãã (ããã©ã«ãã¯"\n\n\n")ã
ã»è³ªåããŒã¿ãè±èªã§çæãããŠãã
èªåçæããã質åããŒã¿ãè±èªã§çæãããŠããŸããã質åããŒã¿çæã¯ãšãªãè±èªã ã£ãã®ãåå ãªã®ã§ãæ¥æ¬èªã«ããŸããã
ã»è³ªåããŒã¿ã«ä»£åè©ãå«ãŸããŠãã
èªåçæããã質åããŒã¿ã«ä»£åè©ãå«ãŸããå ŽåããããŸããã質åã«ã¯åããªãã®ã§æäœæ¥ã§ä¿®æ£ããŸããããããªãããã³ãããšã³ãžãã¢ãªã³ã°ã§èªåçæã®ç²ŸåºŠãæ¹åã§ãããããããŸããã