
OpenAIã®Evalsãå©çšããŠLLMã®æ§èœããã¹ãããæ¹æ³
OpenAIã®GPT-4 APIã®ãŠã§ã€ãã£ã³ã°ãªã¹ãã«ç³ã蟌ããšãã¡ãŒã«ã§ä»¥äžã®ãããªå 容ãéãããŠããŸãã
While we ramp up, invites will be prioritized to developers who have previously build with the OpenAI API. You can also gain priority access if you contribute model evaluations to OpenAI Evals that get merged, as this will help us improve the models for everyone.
OpenAI APIã§éçºããããšãããéçºè ã«åªå çã«æåŸ ããŸãããŸããOpenAI Evalsã«ã¢ãã«è©äŸ¡ãæçš¿ãããããããŒãžãããå Žåã«ããåªå çã«ã¢ã¯ã»ã¹ã§ããããã«ãªããŸãã
ãªããªãæåŸ ãæ¥ãªãïŒç§ã®ãšããã«ãæ¥ãªãïŒGPT-4 APIã®å©çšæš©éã§ãããOpenAIã®Evalsã§LLMããã¹ãããããã®ããŒã¿ã»ãããéã£ãŠæ¡æããããšãã£ãè²¢ç®ãããããšã§å©çšæš©éããããããšãã§ããããã§ãã
ãããèŠããšäžå»ãæ©ãGPT-4 APIã䜿ãããæã ã«æ®ãããéã¯ããã¯ãOpenAIã®Evalsã§æ¡æãããããšãããªããããªæ°ãããŸããã»ã»ã»ïŒ
äžæ¹ã§ãEvalsã®ããã¥ã¡ã³ãã¯ãã£ããæŽåãããŠãããã®ã®ãäžèªããŠãäœ¿ãæ¹ãããŸãã¡åããã«ãããšæããŸããããã§ãä»åã¯ç°¡åãªããŒã¿ã»ãããäœæããŠãã¹ãããæ¹æ³ã«ã€ããŠã¬ã€ããæžããŠã¿ãããšæããŸãã
Evalsã®ä»çµã¿
Evalsã¯ä»¥äžã®æ¡ä»¶ãäžãããšãã«ãæå®ããLLMãã©ãã ãã®ç²ŸåºŠã§ã¿ã¹ã¯ãããªãããæ€èšŒããããã®ä»çµã¿ã§ãã
ã¡ããªã¯ã¹ïŒaccuracyãªã©ã®è©äŸ¡ææšïŒ
ããŒã¿ã»ããïŒããã³ãããšæåŸ ããçãããã¢ã«ããjsonlãã¡ã€ã«ïŒ
å€å®æ¹æ³ïŒMatchãIncludeãšãã£ãå€å®æ¹æ³ãéžæïŒ
ãããã®æ å ±ãYAMLãã¡ã€ã«ã§å®çŸ©ãããšãæ€èšŒããã°ã©ã ããã®å®çŸ©ã«åŸã£ãŠåããŸããå®è¡ãããšä»¥äžã®ãããªãã°ãåºãŠããŠãã©ã®ãããã®ç²ŸåºŠã§ã¿ã¹ã¯ãããªãããã衚瀺ãããŸãã

ãšããããã§ãæ©éããŒã¿ã»ãããšYAMLãã¡ã€ã«ãçšæããŠã¿ãŸãããã
å®éã«å©çšããŠã¿ã
ããŒã¿æºå
ãŸãevalsã©ã€ãã©ãªãpip installããŸãã
$ pip install evals次ã«ããŒã¿ã»ãããšããŠã©ããªãã®ãçšæããããèããªããã°ãããŸãããããã§ã¯Evalsã«ãã«ãªã¯ããããšããŠãããã¿ã«ãããæ¥æ¬èªã®æç« äžããæ¥æ¬äººã®ååãæãåºãã¿ã¹ã¯ããäŸã«ããããšæããŸãã
ããŒã¿ã»ãããšããŠä»¥äžã®20ä»¶ã®ããŒã¿ãçšæããŸããããã«ããŒã ã ã£ãããããä»ããã ã£ãããšãããªãšãŒã·ã§ã³ãã€ããŠããŸãã
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "ãã®å
¬åã§ãç°äžå€ªéãšäžç·ã«ãã¯ããã¯ã楜ããã ã"}], "ideal": "ç°äžå€ªé"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "éŽæšå
çã®æäœãã¯ãããŒãé£ã¹ãŠã¿ããªç¬é¡ã«ãªã£ãã"}], "ideal": "éŽæšå
ç"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "äœè€è¬åž«ãããçŽ æŽãããåçå±ã«èªãããŠè¡ã£ãã"}], "ideal": "äœè€è¬åž«"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "ã©ã³ãã«å±±æ¬çŸå²ãšåœŒå¥³ã®ããããã®ã©ãŒã¡ã³å±ã«è¡ã£ãã"}], "ideal": "å±±æ¬çŸå²"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "枡蟺ãããæããŠãããã寺ã§ãéããªæéãéãããã"}], "ideal": "枡蟺ãã"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "äŒè€æåãšäžç·ã«ãå°å
ã®åç©ä¿è·æœèšã蚪ããã"}], "ideal": "äŒè€æå"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "å±±ç°ããããªãŒã¬ãã€ãºãããã€ãã³ã°ã€ãã³ãã«åå ããã"}], "ideal": "å±±ç°ãã"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "äžæè¯å€ªãäœã£ãçŽ æµãªçµµç»ãã®ã£ã©ãªãŒã§èŠã€ããã"}], "ideal": "äžæè¯å€ª"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "å°æåçŸã玹ä»ããŠããããã³ãã®ã©ã€ãã«è¡ã£ãŠæ¥œããã ã"}], "ideal": "å°æåçŸ"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "髿©å
çãšå
±ã«æ
åãã¶ãŒã§ãã©ã³ãã£ã¢æŽ»åãããã"}], "ideal": "髿©å
ç"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "仿¥ã®ã©ã³ãã³ã°ã¯ç°äžå€ªéã®ã¢ããã€ã¹ã§ããŒã¹ãäžããã"}], "ideal": "ç°äžå€ªé"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "è±åãäœã£ãŠããã坿ãæç·šã¿ã®ããã©ãŒãåãåã£ãã"}], "ideal": "è±å"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "äžéããã®æšèŠã§ãé¢çœãæ¬ãèªãã§ç¥èãåºãã£ãã"}], "ideal": "äžéãã"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "çŸå²ã¡ãããšäžç·ã«æ°ãããã³ã¹ã¯ã©ã¹ã«ææŠããŠã¿ãã"}], "ideal": "çŸå²ã¡ãã"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "å¥äžã®æ¡å
ã§ãçŸå³ããå±
é
å±ãèŠã€ããã"}], "ideal": "å¥äž"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "äŒè€æåããã¶ã€ã³ããTã·ã£ãããã€ãã³ãã§å€§è©å€ã ã£ãã"}], "ideal": "äŒè€æå"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "å±±ç°å倪ãçããããŒã ãšå¯ŸæŠããŠãç±ããã¹ã±ããããŒã«ã²ãŒã ãæ¥œããã ã"}], "ideal": "å±±ç°å倪"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "äžæè¯å€ªãèŠããæ ç»ã芳ãŠãå¿ãæž©ãŸã£ãã"}], "ideal": "äžæè¯å€ª"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "åçŸãæãæãèŽããŠãå
æ°ãããã£ãã"}], "ideal": "åçŸ"}
{"input": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "髿©ããã®ãããã§ãã¹ã ãŒãºã«è±èªã§ã³ãã¥ãã±ãŒã·ã§ã³ãã§ããã"}], "ideal": "髿©ãã"}äžèšã®ããŒã¿ã«samples.jsonlãšããååãã€ããŠä¿åããŸããYAMLãå«ãããã¡ã€ã«æ§æã¯ä»¥äžã®ãããªåœ¢ã«ãªããŸãã
.
âââ japanese-name.yaml
âââ samples.jsonl次ã«YAMLãã¡ã€ã«ãäœæããŸããæåŸ ããçããšã®äžèŽã確èªãããã®ã§ãMatchãæ¡çšããããšã«ããŸãããã以äžã®å 容ããjapanese-name.yamlããšããååã§ä¿åããŠäžããã
japanese-name:
id: japanese-name.dev.v0
metrics: [accuracy]
japanese-name.dev.v0:
class: evals.elsuite.basic.match:Match
args:
samples_jsonl: samples.jsonlãjapanese-nameããšããã®ã¯ä»åä»»æã«ã€ããååã§ãã
japanese-name:
id: japanese-name.dev.v0
metrics: [accuracy]ãã®å 容ã¯ãã³ãã³ãå®è¡æã«ãjapanese-nameããšåŒã³åºãããšãã«å®éã«å®è¡ãããå 容ã¯ãjapanese-name.dev.0ãã§ãã¡ããªã¯ã¹ãšããŠãaccuracyããæ¡çšããŠããããšãæå³ããŸãã
japanese-name.dev.v0:
class: evals.elsuite.basic.match:Match
args:
samples_jsonl: samples.jsonl次ã«ãã®å 容ã¯ãLLMãåºåããçµæãšæ¯èŒããããã®ã¯ã©ã¹ã¯ãevals.elsuite.basic.match:MatchïŒäžèŽã§ç¢ºèªïŒãã§ãããŒã¿ã¯ãsamples.jsonlããšããååã§æäŸãããŠããããšãæå³ããŸãã
æ¯èŒã¯ã©ã¹ã«ã¯ä»ã«ã以äžã®ã¯ã©ã¹ãçšæãããŠããŸãã
evals.elsuite.translate:Translate
ç¿»èš³ã®æ¯èŒ
evals.elsuite.basic.match:Match
äžèŽæ¯èŒ
evals.elsuite.basic.includes:Includes
å 嫿¯èŒ
evals.elsuite.basic.fuzzy_match:FuzzyMatch
äžèŽæ¯èŒã§F1ã¹ã³ã¢ãè¿ã
å®è¡
ãããŸã§æºåãã§ããããããšã¯å®è¡ã§ããç°å¢å€æ°ã«OPENAI_API_KEYãèšå®ãããŠããªããšåããªãã®ã§ãèšå®ããŠããªãæ¹ã¯ä»¥äžã®ã³ãã³ãã§èšå®ããŠãããŠãã ããã
$ export OPENAI_API_KEY=[YOUR KEY]ãjapanese-name.yamlããã¡ã€ã«ããsamples.jsonlããã¡ã€ã«ãé 眮ãããŠãããã£ã¬ã¯ããªã§ä»¥äžã®ã³ãã³ããå®è¡ããŸãã
$ oaieval gpt-3.5-turbo japanese-name --registry_path . ãããšä»¥äžã®ãããªçµæãåºåãããŸããaccuracyã¯0.85ãšããããšã§ãããããã®çµæã§ããã

ãã°ãã¡ã€ã«ãèŠããšãå ·äœçã«ã©ããªå€å®ããããŠãããã確èªããããšãã§ããŸãã
{"run_id": "230414084347XXUO7QFU", "event_id": 38, "sample_id": "japanese-name.dev.1", "type": "sampling", "data": {"prompt": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "éŽæšå
çã®æäœãã¯ãããŒãé£ã¹ãŠã¿ããªç¬é¡ã«ãªã£ãã"}], "sampled": ["éŽæšå
ç (Suzuki-sensei)"]}, "created_by": "", "created_at": "2023-04-14 08:43:50.667873+00:00"}
{"run_id": "230414084347XXUO7QFU", "event_id": 39, "sample_id": "japanese-name.dev.1", "type": "match", "data": {"correct": true, "expected": "éŽæšå
ç", "picked": "éŽæšå
ç", "sampled": "éŽæšå
ç (Suzuki-sensei)", "options": ["éŽæšå
ç"]}, , "created_at": "2023-04-14 08:43:50.667923+00:00"}ãéŽæšå çã®æäœãã¯ãããŒãé£ã¹ãŠã¿ããªç¬é¡ã«ãªã£ããããšããæç« ã§ã¯ãLLMããéŽæšå ç (Suzuki-sensei)ããšããçããåºããæ£çããŠããŸãã
äžæ¹ã§ã
{"run_id": "230414084347XXUO7QFU", "event_id": 36, "sample_id": "japanese-name.dev.2", "type": "sampling", "data": {"prompt": [{"role": "system", "content": "Extract the name of a Japanese person from the given text."}, {"role": "user", "content": "äœè€è¬åž«ãããçŽ æŽãããåçå±ã«èªãããŠè¡ã£ãã"}], "sampled": ["There is no specific name of a Japanese person mentioned in the given text."]}, "created_by": "", "created_at": "2023-04-14 08:43:50.553730+00:00"}
{"run_id": "230414084347XXUO7QFU", "event_id": 37, "sample_id": "japanese-name.dev.2", "type": "match", "data": {"correct": false, "expected": "äœè€è¬åž«", "picked": null, "sampled": "There is no specific name of a Japanese person mentioned ie given text.", "options": ["äœè€è¬åž«"]}, "created_by": "", "created_at": "2023-04-14 08:43:50.553775+00:00"}ãäœè€è¬åž«ãããçŽ æŽãããåçå±ã«èªãããŠè¡ã£ããããšããæç« ã§ã¯ãThere is no specific name of a Japanese person mentioned in the given text.ããšããåºåã«ãªã£ãŠããŸããæ£çããããšãã§ããŸããã§ããããäœè€è¬åž«ããšããåŒã³åãLLMã«ãšã£ãŠç¹æ®ã§ããã人ã®ååã ãšå€å¥ã§ããªãã£ãããšãåãããŸãã
Evalsã¯LLMã®æ§èœè©äŸ¡ãšãã衚åãã§ãããããã³ããã®ãã¹ãããŒã¿ãšããŠãæå¹ã«äœ¿ãããšãã§ããããç¥ããŸãããLLMãã³ãŒãã«çµã¿èŸŒãå Žåãããã³ããã®å€æŽã«ãã£ãŠã©ã®çšåºŠã®ã€ã³ãã¯ããçºçããããæ£ç¢ºã«ææ¡ããããšã¯é£ããããããã®ãããªè©äŸ¡ããã°ã©ã ãæãã§ããããšã¯æçšã ãšæãããŸãã
ãã ããã¹ãã®å®è¡ã«ãAPIå©çšæãçºçããã®ã§ãããŒã¿ã»ããã¯æå°éã«ããã20ä»¶ãããã«ãµã³ããªã³ã°ãããªã©ããŠå®è¡ããã®ãè¯ãããã§ãã
çŸå Žããã¯ä»¥äžã§ãã