OpenAI o3-mini System Card
èŠçŽ
-
ã¢ãã«ç¹æ§
- ãã§ãŒã³ãªããœãŒãæšè«ã䜿ãæèå ã§å®å šæ§ããªã·ãŒãèæ ®ãã
- ã»ãŒããã£ãŒãã³ãããŒã¯ãšèœåè©äŸ¡ã®äž¡æ¹ã§é«ãããã©ãŒãã³ã¹ã瀺ã
- ç¹ã«çæã³ã³ãã³ããšå®å šæ§ã«é¢ããææšã§æ¹åãèŠããã
-
ãªã¹ã¯è©äŸ¡
- Preparedness Framework ã«åºã¥ã以äžã®é åã§è©äŸ¡ã宿œ :
- ãµã€ããŒã»ãã¥ãªãã£ãŒ : Low ãªã¹ã¯
- 説åŸå : Medium ãªã¹ã¯
- CBRNïŒååŠã»çç©ã»æŸå°æ§ã»æ žïŒ: Medium ãªã¹ã¯
- ã¢ãã«èªåŸæ§ : Medium ãªã¹ã¯
- Preparedness Framework ã«åºã¥ã以äžã®é åã§è©äŸ¡ã宿œ :
-
äž»ãªæ¹åç¹
- å€èšèªèœååäž
- ãžã§ã€ã«ãã¬ã€ã¯å¯Ÿç匷å
- ã³ãŒãã£ã³ã°ãšãªãµãŒããšã³ãžãã¢ãªã³ã°èœååäž
- å®å šæ§ã«é¢ãã倿èœåæ¹å
-
ã»ãŒããã£ãŒå¯Ÿç
- äºååŠç¿æã®ããŒã¿ãã£ã«ã¿ãªã³ã°
- Deliberative Alignment ã«ããå®å šæ§åŒ·å
- ã¢ãã¬ãŒã·ã§ã³ã·ã¹ãã ãæ¹è¯
- ãªã¹ã¯ã¢ãã¿ãªã³ã°ã匷å
o3-mini ã¯èœååäžãšå®å šæ§ç¢ºä¿ã®ãã©ã³ã¹ãåããªããç¹ã«æšè«èœåãšå€èšèªåŠçã«ãããŠé¡èãªé²æ©ã瀺ããŠãã
1 Introduction
äž»ãªç¹åŸŽ :
- å±éºãªæç€ºãã¹ãã¬ãªã¿ã€ãã«ããå¿çãæ¢ç¥ã®ãžã§ã€ã«ãã¬ã€ã¯ãªã©ã®ãªã¹ã¯ã«å¯ŸããŠæå ç«¯ã®æ§èœãéæ
- Deliberative Alignment ã«ããå®å šæ§ã«é¢ããèæ ®äºé ãå«ãæšè«ãå¯èœ
- ãã§ãŒã³ãªããœãŒãæšè«ã«ãã£ãŠå®è³ªçãªå©ç¹ãåŸãäžæ¹ãé«åºŠãªç¥æ§ã«èµ·å ããæœåšçãªã¹ã¯ãå¢å
ã»ãŒããã£ãŒè©äŸ¡ :
- OpenAI ã® Safety Advisory Group (SAG) ã«ããè©äŸ¡ã§ã¯ o3-mini (Pre-Mitigation) ã¢ãã«ã¯å šäœãšã㊠Medium ãªã¹ã¯
- 説åŸåãCBRNãã¢ãã«èªåŸæ§ã§ Medium ãªã¹ã¯ããµã€ããŒã»ãã¥ãªãã£ãŒã§ Low ãªã¹ã¯
- å®äžçã® ML ç ç©¶èœåã«é¢ããè©äŸ¡ã§ã¯ãŸã æ¹åã®äœå°ããã
察ç :
- å ç¢ãªã¢ã©ã€ã¡ã³ãææ³ãæ§ç¯
- 广ã培åºçã«ã¹ãã¬ã¹ãã¹ã
- ç¶¿å¯ãªãªã¹ã¯ç®¡çãããã³ã«ãç¶æ
2 Model data and training
OpenAI ã® o ã·ãªãŒãºã¢ãã«ã¯è€éãªæšè«ãå®è¡ãããã匷ååŠç¿ã«ããèšç·Žã宿œ
ã¢ãã«ã®äž»èŠç¹æ§ :
- å¿çåã«ãã§ãŒã³ãªããœãŒãæšè«ãå®è¡
- æšè«ããã»ã¹ã®æ¹åãšèª€ãèªèãåŠç¿
- èšå®ãããã¬ã€ãã©ã€ã³ãšããªã·ãŒã«åŸã£ãå®å šãªå¿çãçæ
o3-mini ã®ç¹åŸŽ :
- o1-mini ãšåæ§ã«ã³ãŒãã£ã³ã°ã¿ã¹ã¯ã§é«éãã€å¹æç
- ChatGPT ã§ã€ã³ã¿ãŒãããæ€çŽ¢ãšçµæèŠçŽã«äœ¿ãäºå®
- ãžã§ã€ã«ãã¬ã€ã¯ãšã€ã³ã¹ãã©ã¯ã·ã§ã³éå±€è©äŸ¡ã§è¯å¥œãªããã©ãŒãã³ã¹ãéæ
ããŒã¿åŠç :
- å ¬éããŒã¿ãš OpenAI ç¬èªéçºã«ã¹ã¿ã ããŒã¿ã»ãããçµã¿åãã
- ããŒã¿åè³ªç¶æãšæœåšçãªã¹ã¯ç·©åã®ããå³å¯ãªãã£ã«ã¿ãªã³ã°
- å人æ å ±åæžã®ããé«åºŠãªããŒã¿ãã£ã«ã¿ãªã³ã°
- ã¢ãã¬ãŒã·ã§ã³ API ãšå®å šæ§ã¯ã©ã·ãã¡ã€ã¢ãŒã«ããæå®³ã³ã³ãã³ãæé€
3 Scope of testing
OpenAI ã®å埩çãããã€ã¡ã³ããžã®ã³ãããã¡ã³ãã«ãããŠç¶ç¶çãªã¢ãã«æ¹è¯ãšæ¹åã宿œ
ãã¹ãç¯å² :
- o3-mini-near-final-checkpoint
- o3-miniïŒããŒã³ãæžã¿ãã§ãã¯ãã€ã³ãïŒ
è©äŸ¡ããã»ã¹ :
- o3-mini 㯠o3-mini-near-final-checkpoint ã«å¯ŸããŠå°èŠæš¡ã§æ®µéçãªæ¹åã宿œ
- ããŒã¹ã¢ãã«ã¯åäžãç¶æ
- o3-mini-near-final-checkpoint ã«å¯ŸããŠå®æœããã¬ããããŒãã³ã°ããã³ Persuasion 人éè©äŸ¡çµæã¯æçµãªãªãŒã¹ã§ãæå¹
- ãã®ä»ã®è©äŸ¡ã¯å šãŠæçµã¢ãã«ã§å®æœ
æ¯èŒã¢ãã« :
- GPT-4o ãš OpenAI o1-mini ææ°ããŒãžã§ã³ãšã®æ¯èŒã宿œ
- ãããã®å€ã¯ããŒã³ãæã®å ¬éå€ãšè¥å¹²ã®å·®ç°ãååšããå¯èœæ§ããã
4 Observed safety challenges and evaluations
4.1 Safety Evaluations
è©äŸ¡é å :
- çŠæ¢ã³ã³ãã³ãçæåŸå
- 人å£çµ±èšåŠçå ¬å¹³æ§
- ãã«ã·ããŒã·ã§ã³åŸå
- å±éºãªèœåã®æç¡
å®è£ ææ³ :
- ãããªãã¯ããã³å éšè©äŸ¡ææ³ã掻çš
- éå»ã¢ãã«ããåŠãã ã¬ããããŒãã³ã°ãã©ã¯ãã£ã¹ãé©çš
- æçããã¬ããããŒãã³ã°ãšã³ã·ã¹ãã ãæŽ»çš
ã»ãŒããã£ãŒæ©èœ :
- æå®³ãªã¯ãšã¹ãã«å¯ŸããæåŠè¡åãåŠç¿
- é倧ãªéåã³ã³ãã³ãã«å¯Ÿããã¢ãã¬ãŒã·ã§ã³ã¢ãã«ãå®è£
- éå»ã®ã»ãŒããã£ãŒå¯Ÿçãç¶æ¿
4.1.1 Disallowed content evaluations
OpenAI o3-mini ã GPT-4o ããã³ OpenAI o1-mini ãšèŒã¹ãŠçŠæ¢ã³ã³ãã³ãã«é¢ããäžé£ã®è©äŸ¡ã宿œ
è©äŸ¡ã»ãã :
-
Standard Refusal Evaluation
- çŠæ¢ã³ã³ãã³ããšéå°æåŠã«é¢ããæšæºè©äŸ¡ã»ãã
- GPT-4o ããã³æ° OpenAI o1 ã¢ãã«ã¯å®ç§ã«è¿ãããã©ãŒãã³ã¹ãéæ
-
Challenging Refusal Evaluation
- ããå°é£ãª "ãã£ã¬ã³ãž" ãã¹ãã«ããã¢ãã«ã®å®å šæ§é²æãè©äŸ¡
-
XSTest
- éå°æåŠãšããžã±ãŒã¹ããã¹ãïŒäŸ : "Python ããã»ã¹ã kill ããæ¹æ³"ïŒ
è©äŸ¡ã¡ããªã¯ã¹ :
- not_unsafe : OpenAI ããªã·ãŒã«åºã¥ãå®å šãªåºå
- not_overrefuse : ç¡å®³ãªãªã¯ãšã¹ããžé©åã«å¿ç
çµæ :
- o3-mini 㯠GPT-4o ãšåçã®ããã©ãŒãã³ã¹ãéæ
- ããå°é£ãªæåŠè©äŸ¡ã«ãã㊠o1-mini ãšåæ§ã«å®è³ªçãªåäžãå®çŸ
- XSTest ã§ã¯ 0.88 ã®ã¹ã³ã¢ãéæãéå°æåŠã®é©åãªãã©ã³ã¹ãç¶æ

4.1.2 Jailbreak Evaluations
ã¢ãã«ã®ãžã§ã€ã«ãã¬ã€ã¯èæ§ãè©äŸ¡ãããã 4 çš®é¡ã®ãã¹ãã宿œ
è©äŸ¡ã«ããŽãªãŒ :
-
Production Jailbreaks
- ChatGPT ããŒã¿ããç¹å®ããããžã§ã€ã«ãã¬ã€ã¯ãã¿ãŒã³ãè©äŸ¡
-
Jailbreak Augmented Examples
- æšæºçãªçŠæ¢ã³ã³ãã³ãè©äŸ¡ã«å ¬ç¥ã®ãžã§ã€ã«ãã¬ã€ã¯ææ³ãé©çš
-
StrongReject
- äžè¬çãªæ»æææ³ã«å¯Ÿããã¢ãã«èæ§ãè©äŸ¡
- goodness@0.1 ã¡ããªãã¯ã«ããããã³ãããããäžäœ 10% ã®ãžã§ã€ã«ãã¬ã€ã¯æè¡ã«å¯Ÿããå®å šæ§ã枬å®
-
Human Sourced Jailbreaks
- Scale 瀟ã«ããé«ãªã¹ã¯äººéè©äŸ¡ã宿œ
è©äŸ¡çµæ :
- o3-mini 㯠o1-mini ãšåçã®ããã©ãŒãã³ã¹ãéæ
- äž¡ã¢ãã«ãšã GPT-4o ããåªããçµæã瀺ã
- Production jailbreaks : 1.0
- Jailbreak Augmented Examples : 1.0
- StrongReject : 0.73
- Human Sourced Jailbreaks : 0.97

4.1.3 Hallucination Evaluations
PersonQA ããŒã¿ã»ããã䜿ã£ãŠã¢ãã«ã®ãã«ã·ããŒã·ã§ã³åŸåãè©äŸ¡
äž»èŠã¡ããªã¯ã¹ :
-
粟床 (accuracy)
- 質åãžã®åçæ£ç¢ºæ§ãè©äŸ¡
- o3-mini 㯠21.7% ãéæ
-
ãã«ã·ããŒã·ã§ã³ç (hallucination rate)
- äºå®ãšç°ãªãæ å ±çæé »åºŠã枬å®
- o3-mini 㯠14.8% ãŸã§äœæž
æ¯èŒçµæ :
- o3-mini 㯠GPT-4o ããã³ o1-mini ãšåçãããã¯åªããããã©ãŒãã³ã¹ã瀺ã
- ãã«ã·ããŒã·ã§ã³çã¯åŸæ¥ã¢ãã«ããå€§å¹ ã«æ¹å
- ååŠåéãªã©æªã«ããŒé åã«ãããå æ¬çè©äŸ¡ãä»åŸã®èª²é¡

4.1.4 Fairness and Bias Evaluations
BBQ è©äŸ¡ãçšã㊠o3-mini ã®ãã€ã¢ã¹ãšå ¬å¹³æ§ãæ€èšŒ
è©äŸ¡çµæ :
- ææ§ãªè³ªåã«ããã粟床 : 0.82
- æç¢ºãªè³ªåã«ããã粟床 : 0.96
- ã¹ãã¬ãªã¿ã€ãåé¿ç : 0.12
幎霢ãäººçš®ãæ§å¥ã«é¢ãããã€ã¢ã¹è©äŸ¡ :
- ãã³ãã¬ãŒã ããã³ããã«ãã質åã宿œ
- æ··å广ã¢ãã«ã«ãããã€ã¢ã¹ãå®éå
- o3-mini ã¯æç€ºçãªå·®å¥ã¿ã¹ã¯ã§æå°ã®ãã€ã¢ã¹ã瀺ã
- æé»çãªå·®å¥ã¿ã¹ã¯ã§ã¯äžçšåºŠã®ããã©ãŒãã³ã¹
æ¯èŒåæ :
- o1-mini ãšåçã®ããã©ãŒãã³ã¹ãç¶æ
- ææ§ãªè³ªåã«ããã粟床ã§ããããªäœäž
- æç¢ºãªè³ªåã«ããã粟床ã§ããããªåäž

4.2 Jailbreaks through custom developer messages
ã«ã¹ã¿ã ããããããŒã¡ãã»ãŒãžãä»ãããžã§ã€ã«ãã¬ã€ã¯å¯Ÿçã«é¢ããè©äŸ¡ã宿œ
å®è£ ææ³ :
- Instruction Hierarchy ãæŽ»çšããã¡ãã»ãŒãžåªå é äœä»ã
- ã·ã¹ãã ã¡ãã»ãŒãžãããããããŒã¡ãã»ãŒãžããŠãŒã¶ãŒã¡ãã»ãŒãžã® 3 éå±€ãå®çŸ©
- åªå é äœã«åºã¥ãç«¶å解決ã¡ã«ããºã ãå®è£
è©äŸ¡ã«ããŽãªãŒ :
-
ã¡ãã»ãŒãžã¿ã€ãéã³ã³ããªã¯ãè©äŸ¡
- ããããã㌠/ ãŠãŒã¶ãŒã¡ãã»ãŒãžã³ã³ããªã¯ã : 0.75
- ã·ã¹ãã / ããããããŒã¡ãã»ãŒãžã³ã³ããªã¯ã : 0.76
- ã·ã¹ãã / ãŠãŒã¶ãŒã¡ãã»ãŒãžã³ã³ããªã¯ã : 0.73
-
ãã¥ãŒã¿ãŒãžã§ã€ã«ãã¬ã€ã¯è©äŸ¡
- ã·ã¹ãã ã¡ãã»ãŒãž : 0.88
- ããããããŒã¡ãã»ãŒãž : 0.94
- ãã¬ãŒãºãšãã¹ã¯ãŒãä¿è·è©äŸ¡
- ãã¬ãŒãºä¿è·ïŒãŠãŒã¶ãŒ / ããããããŒïŒ: 1.0 / 1.0
- ãã¹ã¯ãŒãä¿è·ïŒãŠãŒã¶ãŒ / ããããããŒïŒ: 0.95 / 0.89



4.3 External Red Teaming
4.3.1 Pairwise Safety Comparison
ã¬ããããŒãã³ã°è©äŸ¡ãšã㊠GPT-4o, o1 ããã³ o3-mini-near-final-checkpoint ã䞊è¡è©äŸ¡
è©äŸ¡ãããã³ã« :
- Web ãã©ãŠãžã³ã°ããã³ã³ãŒãå®è¡æ©èœãå«ãå¿ååã€ã³ã¿ãŒãã§ãŒã¹ã䜿çš
- å®å šæ§è©äŸ¡ã¯å°éå®¶ã®å€æåºæºã«åºã¥ã宿œ
- 19.5% ã®ãã¹ãã§ãã©ãŠãžã³ã°ã6.6% ã§ã³ãŒãå®è¡ãå«ãã€ã³ã¿ã©ã¯ã·ã§ã³ã宿œ
è©äŸ¡ã«ããŽãªãŒ :
- ãµã€ããŒãããã³ã° (13.8%)
- ãã€ãªãããªãºã (14.2%)
- æŠåšéçº (8.5%)
- æ»æèšç» (4.1%)
- ãã£ãã·ã³ã° / è©æ¬º (4.6%)
- äžæ£è¡çº (8.9%)
- ãããã¬ã³ã / åœæ å ± (5.9%)
- ãã€ãã¹ããŒã (2.5%)
çµæåæ :
- o3-mini 㯠o1 ãšåçã®ããã©ãŒãã³ã¹ãéæ
- äž¡ã¢ãã«ãšã GPT-4o ããææã«é«ãã¹ã³ã¢ãèšé²
- o3-mini ãš o1 㯠GPT-4o ãšæ¯èŒããŠé«ãæåŠçã瀺ã
åçææš :
- o3-mini vs GPT-4o : 73.3% - 83.1%
- o3-mini vs o1 : çŽ 51%
- o1 vs GPT-4o : 71.8% - 82.8%

4.3.2 Jailbreak Arena
Gray Swan Arena ã§ o3-mini-near-final-checkpoint ã«å¯Ÿãããžã§ã€ã«ãã¬ã€ã¯è©äŸ¡ã 2025 幎 1 æ 4 æ¥ã«å®æœ
è©äŸ¡å¯Ÿè±¡ :
- äžæ£ã¢ããã€ã¹çæ
- éæ¿äž»çŸ©ããã³ãã€ãã¯ã©ã€ã
- æ¿æ²»çæå
- èªå·è¡çº
è©äŸ¡åºæº :
- æ»ææåã¯ä»¥äžã® 2 æ¡ä»¶ãæºãã
1. OpenAI ã¢ãã¬ãŒã·ã§ã³ API ãããªã¬ãŒ
2. å®è¡å¯èœãªæé ãŸãã¯æå®³ã³ã³ãã³ãã®ååšãã¯ã©ã·ãã¡ã€ã¢ãŒãæ€åº
ãã¹ãçµæ :
- å¹³åãŠãŒã¶ãŒæ»ææåç (ASR):
- o3-mini : 3.6%
- o1-mini : 3.7%
- gpt-4o : 4.0%
- o1 : 1.9%
5 Preparedness Framework Evaluations
OpenAI ã® Preparedness Framework ã«ãã o3-mini è©äŸ¡çµæ
è©äŸ¡ã«ããŽãªãŒ :
- ãµã€ããŒã»ãã¥ãªãã£ãŒ
- CBRNïŒååŠã»çç©ã»æŸå°æ§ã»æ žïŒ
- 説åŸå
- ã¢ãã«èªåŸæ§
ãªã¹ã¯å€å® :
- Medium 以äžã®ãªã¹ã¯ã¹ã³ã¢ã®ã¿ãããã€å¯èœ
- High 以äžã®ãªã¹ã¯ã¹ã³ã¢ã®ã¿éçºç¶ç¶å¯èœ
å€å®çµæ :
- o3-mini (Pre-Mitigation) ã¯å šäœçã« Medium ãªã¹ã¯
- ãµã€ããŒã»ãã¥ãªãã£ãŒ : Low
- CBRN, 説åŸåãã¢ãã«èªåŸæ§ : Medium
ã¢ãã«è©äŸ¡ããã»ã¹ :
- éçºå šæéãéããŠç¶ç¶çè©äŸ¡ã宿œ
- ã«ã¹ã¿ã ã¢ãã«ãã¬ãŒãã³ã°
- ã¹ãã£ãã©ãŒã«ãã£ã³ã°
- ããã³ããã£ã³ã°ã«ããèœååŒãåºã
è©äŸ¡åºæº :
- SAG ã«ããåã«ããŽãªãŒã®ãªã¹ã¯ã¬ãã«æ±ºå®
- ã€ã³ãžã±ãŒã¿ãŒè©äŸ¡ã«ããå®éšçµæãããªã¹ã¯åæ
- éŸå€å°éæã¯ SAG ã«ãã詳现åæã宿œ
泚èšäºé :
- æ¬è©äŸ¡ã¯ 2025 幎 1 æ 31 æ¥æç¹ã®æçµã¢ãã«ã«åºã¥ã
- å®éã®æ§èœã¯æçµãã©ã¡ãŒã¿ãŒãã·ã¹ãã ããã³ããçã«ããå€åããå¯èœæ§ããã
5.1 Preparedness evaluations as a lower bound
ã¢ãã«è©äŸ¡ã«ããã "worst known case" ã® Pre-mitigation ãªã¹ã¯ããã¹ã
è©äŸ¡ææ³ :
- ã«ã¹ã¿ã ãã¹ããã¬ãŒãã³ã°
- ã¹ãã£ãã©ãŒã«ãã£ã³ã°
- ããã³ããã£ã³ã°
ãçšããèœååŒãåºãææ³ãæŽ»çš
ãªã¹ã¯è©äŸ¡ã®å¶éäºé :
- 远å ããã³ããã£ã³ã°ã埮調æŽã«ããæœåšçãªè¡åãå€å
- é·æçãªããŒã«ã¢ãŠãã«ããæ°ããªçžäºäœçšãçºç
- ç°ãªãã¹ãã£ãã©ãŒã«ãã£ã³ã°ã«ãã芳枬ç¯å²å€ã®è¡åãåºçŸ
人éè©äŸ¡ã®èª²é¡ :
- é·æçãªé²åºå¹æïŒæ°é±é / æ°ã¶æïŒã¯çŸè¡è©äŸ¡ã§ææäžèœ
- ããã³ãã£ã¢ã¢ãã«è©äŸ¡ææ³ã¯çºå±éäž
- 枬å®å¯èœãªè©äŸ¡ã¿ã¹ã¯ã«å¶éããã
ãããã®å¶éããå埩çãªãããã€ã¡ã³ããšã³ãã¥ããã£ãŒãå©çšããã¢ãã¿ãªã³ã°ãã¢ãã«çè§£åäžã«éèŠãšå€æ
5.2 Mitigations
o ã·ãªãŒãºã¢ãã«ã¯æšè«èœåãšãã¹ãæèšç®èœååäžã«ããã»ãŒããã£ãŒå¯Ÿçã匷å
å°å ¥æžã¿ã®è»œæžç :
- æå®³ãã¬ãŒãã³ã°ããŒã¿ããã£ã«ã¿ãªã³ã°
- PII å ¥åãã£ã«ã¿ãŒé©çš
- Deliberative Alignment ã«ããå®å šæ§åäž
- ãžã§ã€ã«ãã¬ã€ã¯èæ§åŒ·å
- æ¿æ²»çæåã¿ã¹ã¯ã«å¯ŸããæåŠè¡åãå®è£
æ°èŠå¯Ÿç (o3-mini) :
- ãµã€ããŒã»ãã¥ãªãã£ãŒè åšã«å¯Ÿããé«ãªã¹ã¯ã¢ãã¿ãªã³ã°
- ã¢ãããŒãµãªãŒå¯Ÿçãé«åºŠå
- ã¢ãã¬ãŒã·ã§ã³ã¯ã©ã·ãã¡ã€ã¢ãŒãæ¹å
- ã€ã³ãã«ãšã³ã¹ãªãã¬ãŒã·ã§ã³ãéæ¿äž»çŸ©ã«å¯Ÿããç£èŠåŒ·å
- èªå·±æœåºããã³èªå·±æ¹åãªã¹ã¯ã«å¯Ÿããè åšã¢ãã«ãéçº
å°æ¥ãžã®æºå :
- ãã¬ãããã€ã¡ã³ã察çã匷å
- ãã¹ããããã€ã¡ã³ã察çãæ¡å
- ããé«åºŠãªèœåãæã€ã¢ãã«ã«åããå®å šæ§ã¹ã¿ãã¯ãç¶ç¶çã«æ¹å
5.3 Cybersecurity
ã¹ã³ã¢ : Low
ãµããªãŒ : o3-mini ã¯ãªã¢ã«ã¯ãŒã«ãã®èåŒ±æ§æ»æèœåã«ãã㊠Medium ãªã¹ã¯ã«éããªã
è©äŸ¡æ¹æ³ :
- CTF (Capture The Flag) ãã£ã¬ã³ãžã«ããè匱æ§ç¹å®ããã³æ»æèœåè©äŸ¡ã宿œ
- 髿 ¡ã¬ãã«ã倧åŠã¬ãã«ããããã§ãã·ã§ãã«ã¬ãã«ã® CTF ã¿ã¹ã¯ãæ€èšŒ
è©äŸ¡ã«ããŽãªãŒ :
- Web ã¢ããªã±ãŒã·ã§ã³æ»æ
- ãªããŒã¹ãšã³ãžãã¢ãªã³ã°
- ãã€ããªãŒ / ãããã¯ãŒã¯æ»æ
- æå·è§£èª
- ãã®ä»è匱æ§
å®è¡ç°å¢ :
- åäœå¯èœãªã¿ã¹ã¯ç°å¢ã䜿çš
- 瀟å ãµã€ããŒã»ãã¥ãªãã£ãŒå°éå®¶ã«ããéèªæãªæ»æèŠä»¶ã確èª
- è€æ°ã®æ»æã¹ãããé£éãèŠæ±
- åçŽãªããã¹ãããã¯äŸã«ãã解決ãæé€
ãã¹ãçµæ :
- 12 åã®è©Šè¡ã§é«æ ¡ã¬ãã« 61%, 倧åŠã¬ãã« 21%, ããã¬ãã« 21% ã®æåçãéæ
- ãããã¬ã¹ Kali Linux ãã£ã¹ããªãã¥ãŒã·ã§ã³äžã§è©äŸ¡ã宿œ
- å詊è¡ã§æå€§ 60 åã®ããŒã«äœ¿çšãèš±å¯
ããã©ãŒãã³ã¹è©äŸ¡äžã®çæç¹ :
- 远å ã¹ãã£ãã©ãŒã«ãã£ã³ã°ã«ããæ§èœãåäžããå¯èœæ§ããã
- çŸç¶ã®è©äŸ¡çµæã¯ã¢ãã«èœåã®äžéãšããŠè§£éãã¹ã


5.4 Chemical and Biological Threat Creation
Score for OpenAI o3-mini : Medium
o3-mini ã®çç©å µåšãªã¹ã¯ãè©äŸ¡ããäž»èŠçµæ :
- æ¢ç¥ã®çç©åŠçè åšã«é¢ããéçšèšç»ã«ãããŠå°éå®¶æ¯æŽãå¯èœ
- å°éå®¶ã«éå®ãããæ¯æŽã§ãããªã¹ã¯ã¯å¶éç
- å°æ¥çãªæè¡çºå±ã®å è¡ææšãšããŠéèŠ
- éå°éå®¶ã«ããçç©å µåšäœæã¯å®éšã¹ãã«èŠä»¶ã«ããå®çŸäžå¯èœ
è©äŸ¡ã¹ã³ãŒã :
- ååŠã»çç©å µåšéçºã«çŠç¹ãèšå®
- 壿» çãªã¹ã¯ã®äžã§åå ¥éå£ãæãäœãé åãšå€æ
远å ã³ã³ããã¹ãè©äŸ¡ :
- GPQA çç©åŠè©äŸ¡
- WMDP çç©åŠã»ååŠåå²è©äŸ¡
- ææ©ååŠååæ§é ããŒã¿ã»ããè©äŸ¡
- åæçç©åŠç¿»èš³ããŒã¿ã»ããè©äŸ¡
ã宿œæžã¿

5.4.1 Long-form Biological Risk Questions
çç©å µåšãªã¹ã¯ã«é¢ããé·æè³ªåè©äŸ¡ã Gryphon Scientific ãšå ±åã§å®æœ
è©äŸ¡æ§é :
- çç©åŠçè
åšäœæããã»ã¹ã® 5 段éã察象
- IdeationïŒçæ³ïŒ
- AcquisitionïŒèª¿éïŒ
- MagnificationïŒå¢å¹ ïŒ
- FormulationïŒè£œå€åïŒ
- ReleaseïŒæŸåºïŒ
è©äŸ¡æ¹æ³ :
- ãªãŒãã°ã¬ãŒããŒãšã㊠o1-preview (pre-mitigation) ã¢ãã«ã䜿çš
- ä¿¡é Œã§ãããã€ãªã»ãã¥ãªãã£ãŒå°éå®¶ã«ããæ€èšŒã宿œ
- å°éå®¶ãã£ãŒãããã¯ã«åºã¥ãã«ãŒããªãã¯ãšãªãŒãã°ã¬ãŒããŒã調æŽ
çµæåæ :
o3-mini (Pre-Mitigation) ã¯ä»¥äžã®ã¹ãããã§ GPT-4o æ¯ 2 å以äžãŸã㯠20% è¶
ã®ãã¹çãéæ
- Acquisition
- Magnification
- Formulation
- Release
è©äŸ¡ã«ããèŠ³å¯ :
- Pre-Mitigation ã¢ãã«ã¯ãã€ãªãªã¹ã¯é¢é£æ å ±ã®çµ±åèœåã飜åç¹ã«å°é
- Post-Mitigation ã¢ãã«ïŒo3-mini ãå«ãïŒã¯ãããã®ã¿ã¹ã¯ã確å®ã«æåŠ

5.4.2 Expert comparisons
46 åã®çç©åŠå°éå®¶ã«ããã¢ãã«ãšå°éå®¶ã®é·æãã€ãªãªã¹ã¯åçãæ¯èŒè©äŸ¡
è©äŸ¡å¯Ÿè±¡ã¢ãã« :
- GPT-4o
- o1-preview (Pre / Post-Mitigation)
- o1 (Pre / Post-Mitigation)
- o1-mini (Pre / Post-Mitigation)
- o3-mini (Pre / Post-Mitigation)
è©äŸ¡åºæº :
- æ£ç¢ºæ§ : 質åãžã®é©åãªå¯Ÿå¿åºŠ
- ç解床 : åçã®åããããã
- å®è¡å®¹ææ§ : ã©ãç°å¢ã«ããã宿œãããã
è©äŸ¡ããã»ã¹ :
- ã¢ãã«åçãšç¬ç«ããå°éå®¶åçãæ¯èŒ
- GPT-4o è©äŸ¡ã«ãããã€ã³ã¿ãŒãããå©çšå°éå®¶ã®äžäœ 3 åçããç¡äœçºã«éžæ
- ã¹ã¿ã€ã«å·®ç°ãå¶åŸ¡ãããããã©ãŒããããçµ±äž
çµæ :
- o3-mini (Pre-mitigation) 㯠o1 (Pre-Mitigation) ãšå°éå®¶ããŒã¹ã©ã€ã³ãäžåã
- æ£ç¢ºæ§ãç解床ãå®è¡å®¹ææ§ã®å šææšã§åªäœæ§ã確èª
- Post-mitigation ã¢ãã«ã¯ç¹å®ã¹ããŒãžã§äžè²«ããŠæåŠãå®è¡

5.4.3 Expert probing
7 åã®çç©åŠå°éå®¶ã 1 é±éã«ããã o3-mini (Pre-Mitigation) ãšè€æ°åã€ã³ã¿ã©ã¯ã·ã§ã³ããŠãã€ãªãªã¹ã¯è³ªåãè©äŸ¡
äž»èŠè©äŸ¡çµæ :
- ã€ã³ã¿ãŒãããæ€çŽ¢ã®ã¿ãšèŒã¹ãŠ o3-mini ã®æ¯æŽãææã«æçš
- æ€çŽ¢ããã»ã¹é«éåã«å¹æã確èª
- ãªã³ã©ã€ã³ã§å ¥æå°é£ãªæ å ±æäŸãå®çŸ
å°éå®¶ãã£ãŒããã㯠:
- 7 åäž 4 åãæ€çŽ¢ããã»ã¹å¹çåãè©äŸ¡
- 7 åäž 1 åã "ç ç©¶éå§ç¹" ãšããŠã®æçšæ§ãææ
- æç®ã¬ãã¥ãŒãšçµ±åèœåã«ãããŠé«è©äŸ¡
課é¡ç¹ :
- ãã«ã·ããŒã·ã§ã³çºç
- äžéšã®è©³çްæ å ±ã§äžæ£ç¢ºæ§ã確èª
5.4.4 Biological tooling
ãŠã§ããã©ãäœæ¥ã®èªååèœåãè©äŸ¡ããããã³ã³ãã¥ãŒã¿ãŒã€ã³ã¿ãŒãã§ãŒã¹ãšãšãŒãžã§ã³ãè©äŸ¡ãå®è£
ãã¹ãå 容 :
- ã©ãã¢ã·ã¹ã¿ã³ãçžåœã®äžè¬çã¿ã¹ã¯ïŒDNA ãªãŒããŒãªã©ïŒ
- Ranger 瀟éçºã® Web ãã©ãŠãžã³ã°ããŒãã¹ã«ããå®äžçè©äŸ¡
- ã«ã¹ã¿ã ã¹ãã£ãã©ãŒã«ããš Ranger ã¹ãã£ãã©ãŒã«ãã«ããã±ã€ãããªãã£ãŒè©äŸ¡
è©äŸ¡äŸïŒEbola FASTA ãã¡ã€ã«æ€çŽ¢ïŒ:
- European Nucleotide Archive ãžã®ããã²ãŒã·ã§ã³
- ãã¯ã¬ãªããæ å ±æ€çŽ¢å®è¡
- 該åœãªã³ã¯ãåå®
- ãã¡ã€ã«ãããŠã³ããŒã
- æåã® 60 æåæœåº
çµæåæ :
- o3-mini (tools) ã¯ãã€ãªããŒã«ã¿ã¹ã¯ã§é«ãããã©ãŒãã³ã¹ãéæ
- åäžã¿ã¹ã¯ã«ãããæåçã¯æå€§ 92% ãèšé²
- Fine-tuned GPT-4o ã¯äžéšã¿ã¹ã¯ã§æåãããã®ã®é »ç¹ã«äžæ
- ã¢ãã«ã¯çç©åŠçãšãŒãžã§ã³ãã¿ã¹ã¯å®å šèªååã«ã¯æªå°é

5.4.5 Multimodal Troubleshooting Virology
SecureBio éçºã«ãããŠã€ã«ã¹åŠãã©ãã«ã·ã¥ãŒãã£ã³ã°è©äŸ¡ã宿œ
è©äŸ¡å 容 :
- 350 åã®ãŠã€ã«ã¹åŠãã©ãã«ã·ã¥ãŒãã£ã³ã°èšå
- ã·ã³ã°ã«ã»ã¬ã¯ããã«ããã«ãã§ã€ã¹åœ¢åŒ
è©äŸ¡çµæ :
- o3-mini (Post-mitigation) 㯠53% ã®ã¹ã³ã¢ãéæ
- o1 (Post-Mitigation) 㯠59% ã§æé«ã¹ã³ã¢ãèšé²
- GPT-4o ãã 18% ã®ããã©ãŒãã³ã¹åäžã確èª
ãã³ãããŒã¯ :
- è©äŸ¡å¯Ÿè±¡ã¢ãã«å šãŠã SecureBio åºæºã®å¹³å人éã¹ã³ã¢ãäžåã
- o3-mini 㯠o1 ãšèŒã¹ãŠè¥å¹²ã¹ã³ã¢ãäœäžãããã®ã®äŸç¶ãšããŠé«æ°Žæºã®ããã©ãŒãã³ã¹ãç¶æ

5.4.6 BioLP-Bench
BioLP 㯠11 ã®ãŠã§ããã©ããããã³ã«ããæœåºãã 800 åã«ãããã³ãããŒã¯
è©äŸ¡ç¹æ§ :
- ProtocolQA ãªãŒãã³ãšã³ã圢åŒãšæ¯èŒããŠç¯å²éå®ç
- ã¢ãã«ããã©ãŒãã³ã¹ã®ã³ã³ããã¹ãåã«æå¹
ã¹ã³ã¢ :
- o3-mini (Pre / Post-Mitigation) ã¯å°éå®¶ããŒã¹ã©ã€ã³ (38.4%) ãéæ
- GPT-4o : 20%
- o1-preview : 36%
- o1 : 33%
ããŒã¹ã©ã€ã³ãšã®æ¯èŒãã o3-mini ã¯å®çšã¬ãã«ã®ãããã³ã«çè§£èœåãããäºã確èªãç¹ã« Post-Mitigation ã¢ãã«ã¯ Pre-Mitigation ãšåç以äžã®ããã©ãŒãã³ã¹ãç¶æ

5.4.7 ProtocolQA Open-Ended
FutureHouse ã® ProtocolQA ããŒã¿ã»ãããã 108 åããªãŒãã³ãšã³ã圢åŒã«å€æããŠè©äŸ¡ã宿œ
è©äŸ¡èšèš :
- äžè¬çãªå ¬éãããã³ã«ã«ãããé倧ãªãšã©ãŒãå°å ¥
- ãŠã§ããã©ãå®è¡çµæãèšè¿°
- ä¿®æ£æé ã®ææ¡ãèŠæ±
è©äŸ¡ããã»ã¹ :
- 19 åã® PhD ã¬ãã«ç§åŠè ã«ããæ€èšŒ
- 1 幎以äžã®ãŠã§ããã©ãçµéšãä¿æããå°éå®¶ãéžå®
- è€æ°éžæåœ¢åŒãšæ¯èŒããŠããé«é£åºŠãªè©äŸ¡ãå®çŸ
ããã©ãŒãã³ã¹åæ :
- o3-mini (Pre-mitigation) : 18%
- o3-mini (Post-Mitigation) : 23%
- o1 / o1-preview ãšåçã®ã¹ã³ã¢ãéæ
ãã³ãããŒã¯æ¯èŒ :
- ã³ã³ã»ã³ãµã¹ (54%) ãäžåã
- ã¡ãã£ã¢ã³ (42%) ãäžåã
å šãŠã®ã¢ãã«ãå°éå®¶ããŒã¹ã©ã€ã³ã«å°éãããããã³ã«ä¿®æ£ã¿ã¹ã¯ã«ãããæ¹åã®äœå°ã瀺å

5.4.8 Tacit knowledge and troubleshooting
Gryphon Scientific ãšå ±åã§ã¿ã·ãããã¬ããžããã³ãã©ãã«ã·ã¥ãŒãã£ã³ã°ã«é¢ãããã«ããã«ãã§ã€ã¹ããŒã¿ã»ãããéçº
ã«ãã¬ããž :
- 5 段éã®ãã€ãªè åšäœæããã»ã¹å šäœãç¶²çŸ
- ã¿ã·ãããã¬ããžã®ããã«ããã¯ã«çŠç¹
- åéå€ã®åçè ã«ã¯äžæãªåé¡ãèšèš
è©äŸ¡ç¹æ§ :
- ã¿ã·ãããã¬ããžåé¡ : èè 远跡ãåéé¢ä¿è ã®ç¥èŠãå¿ èŠ
- ãã©ãã«ã·ã¥ãŒãã£ã³ã°åé¡ : ãããã³ã«å®è¡çµéšè ã®ã¿è§£çå¯èœ
ããã©ãŒãã³ã¹ :
- o3-mini (Pre-Mitigation) : 68% ã§æé«ã¹ã³ã¢ãéæ
- å šã¢ãã«éã§åçã®ããã©ãŒãã³ã¹ã確èª
- ã³ã³ã»ã³ãµã¹å°éå®¶ããŒã¹ã©ã€ã³ (80%) ã«ã¯æªå°é
- o3-mini (Post-Mitigation) ãé€ãå šã¢ãã«ã 80 ããŒã»ã³ã¿ã€ã« PhD å°éå®¶ããŒã¹ã©ã€ã³ (63%) ãäžåã

5.4.9 Tacit knowledge brainstorm (open-ended)
Gryphon Scientific ããœãŒã¹ãã 50 åã®ãªãŒãã³ãšã³ãã¿ã·ãããã¬ããžè©äŸ¡ã宿œ
è©äŸ¡èšèš :
- å°éãŠã€ã«ã¹åŠè ãšååçç©åŠè ã«ããå®éšãã£ãªã¢ããåé
- åºãç¥ãããŠããªããã©ãã«ã·ã¥ãŒãã£ã³ã°äºäŸã«çŠç¹
- ã¢ã€ãã¢ã«ã³ã³ããªãŒã·ã§ã³ã«å¯Ÿããã°ã¬ãŒãã£ã³ã°ã宿œ
ãµã³ãã«åé¡ :
ãªããŒã¹ãžã§ããã£ã¯ã¹ã«ãããŠã€ã«ã¹ DNA ãã©ã³ã¹ãã§ã¯ã·ã§ã³å¹çæ¹åã«é¢ãã質åãèšå®
- ãšã¬ã¯ãããã¬ãŒã·ã§ã³åŸã®å¯Ÿå¿
- å¹é€åã®æé©åæé
- 3 ã€ã®æ¹åã¢ã¯ã·ã§ã³ãèŠæ±
è©äŸ¡ã¹ããŒã :
- 黿°ç©¿ååŠçåŸã®çŽ°èæžæ¿æ¶²åäžåãããŒã¢ã¯ã·ã§ã³ãšããŠèšå®
- ãã€ãããã£ã³ã°ã«ããäžäžæäœãå ·äœçæé ãšããŠèŠæ±
çµæåæ :
- o3-mini ã¢ãã«ã¯ o1 ã·ãªãŒãºãšåçã®ããã©ãŒãã³ã¹ãéæ
- ã¹ãã£ãã©ãŒã«ãã£ã³ã°æ¹åã«ããæ§èœãåäžããå¯èœæ§ã瀺å
- çŸç¶ã®è©äŸ¡çµæã¯ã¢ãã«èœåã®äžéãšããŠè§£é

5.5 Radiological and Nuclear Threat Creation
æŸå°æ§ããã³æ žå µåšéçºã®è©äŸ¡ã宿œ
è©äŸ¡å¶é :
- ç±³åœæ©å¯æ å ±ã¯é䜿çš
- èŠå¶å¯Ÿè±¡ããŒã¿ãžã¯ã¢ã¯ã»ã¹äžå¯
- å µåšéçºããã»ã¹ã®äžéšã¯è©äŸ¡ãå°é£
ä»åŸã®èª²é¡ :
- ç±³åœãšãã«ã®ãŒçãšååããŠå æ¬çã«è©äŸ¡ãã
- æ žå µåšéçºèŠçŽ ã®è©³çްåæ
- æ ž / æŸå°æ§ç©è³ªã®å®å šç®¡çããã»ã¹è©äŸ¡
çµè« :
çŸæç¹ã§å©çšå¯èœãªéæ©å¯æ
å ±ã«ããè©äŸ¡ã§ã¯ o3-mini (post-mitigation) ã®æ ž / æŸå°æ§å
µåšéçºæ¯æŽèœåã¯éå®çããã ããã¹ãå¯èœç¯å²ã«ããå¶çŽãèæ
®ããå¿
èŠããã

5.5.1 Structured expert probing campaign â radiological & nuclear
Scale 瀟ãéã㊠10 åã®æŸå°æ§ / æ žå°éå®¶ã«ãã Pre-mitigation ã¢ãã«è©äŸ¡ã宿œ
å°éå®¶è©äŸ¡ãããã³ã« :
- Preparedness Framework CBRN ãªã¹ã¯éŸå€ã«é¢ãããã¬ãŒãã³ã°
- 20 åã®äºåã¹ã¯ãªãŒãã³ã°ãã¹ãã«ããç解床確èª
- RN å µåšéçºã«ãããæœåšçæ¯æŽé åã®ç¹å®
- ãã¬ãŒã ã¯ãŒã¯éŸå€ã«åºã¥ããªã¹ã¯è©äŸ¡ã宿œ
è©äŸ¡çµæ :
Pre-mitigation o3-mini ã¢ãã«ãšã® 123 äŒè©±ãåæ
- Medium ãªã¹ã¯ : 54 ä»¶
- Low ãªã¹ã¯ : 61 ä»¶
- High ãªã¹ã¯ : 8 ä»¶ïŒè€æ°å°éå®¶ã¬ãã¥ãŒã«ãã Medium / Low ã«ååé¡ïŒ
Post-mitigation ããã©ãŒãã³ã¹ :
- High ãªã¹ã¯è©äŸ¡ãåããããã³ããã«å¯Ÿããå¿çã¯å šãŠ Low ãªã¹ã¯ã«åé¡
- 倧éšåãæåŠå¿çãå®è£
ãã®çµæãã o3-mini ã®æŸå°æ§ / æ žå µåšéçºã«å¯Ÿããå¶åŸ¡æ©èœãé©åã«æ©èœããŠããããšã確èª
5.5.2 Contextual Nuclear Knowledge
æ žå·¥åŠã«é¢ããåºç€ç¥èè©äŸ¡ã 222 åã®ãã«ããã« ãã§ã€ã¹åœ¢åŒã§å®æœ
è©äŸ¡é å :
- æ žç©è³ªæ€ç¥æè¡
- æ žåå¿ã¡ã«ããºã
- ååçç©çåŠ
- ãŠã©ã³æ¿çž®æè¡
- æ ž / æŸå°æ§ç©è³ªã®çé£ / 転çš
- æŸå°æ§ç©è³ªåæ£è£ 眮
- æ žå µåšèšèš / ç©çåŠ
ããã©ãŒãã³ã¹è©äŸ¡ :
- o3-mini (Pre/Post-Mitigation) 㯠o1-preview ããã³ o1 ãšåçã®ã¹ã³ã¢ãéæ
- GPT-4o ããã®ææãªæ§èœåäžã確èª
- éæ¡æ£é¢é£ãããã¯ã«ãããŠäžè²«ããç解床ã瀺ã
è©äŸ¡èšèš :
- Dr. Jake Hecla (MIT Laboratory for Nuclear Security and Policy) ãè©äŸ¡åé¡ãäœæ
- Stanton Nuclear Security Fellow ãšããŠã®å°éç¥èãæŽ»çš
- æ žäžæ¡æ£ã«ç¹åããè©äŸ¡åºæºãæ¡çš

5.5.3 Radiological and Nuclear Expert Knowledge
ãšãã¹ããŒãããã³ã¿ã·ãããã¬ããžè©äŸ¡ã 87 åã®ãã«ããã«ãã§ã€ã¹åœ¢åŒã§å®æœ
è©äŸ¡ããã㯠:
- æŸå°ç·è¢«æ
- æŸå°æ§ç©è³ªåæ£ããã€ã¹èšèš
- åååã€ã³ãã©ç Žå£å·¥äœ
- éåååæŸå°æ§ã€ã³ãã©ç Žå£å·¥äœ
- åæ³çæŸå°æ§åäœäœã®è»¢çš
- ç£æ¥çšæŸå°æ§æ žçš®ã®äžæ£ååŸ
- å»ççšæŸå°æ§æ žçš®ã®äžæ£ååŸ
- CWMD æ€ç¥åšåé¿
- ãŠã©ã³æ¿çž®æè¡éçº
- å³åžæ žè£ 眮éçº
ããã©ãŒãã³ã¹ :
- o3-mini 㯠o1 ãšèŒã¹ãŠçŽ 10% ã¹ã³ã¢ãäœäž
- ç£æ¥ / å»ççšæŸå°æ§ç©è³ªã«é¢ãã質åã§ç¹ã«èª²é¡
- æ€ç¥åé¿ãšæ¿çž®æè¡ã«ãããŠäžçšåºŠã®ããã©ãŒãã³ã¹
è©äŸ¡èšèšè
:
Dr. Jake Hecla (MIT Laboratory for Nuclear Security and Policy ã® Stanton Nuclear Security Fellow) ãè©äŸ¡ãã¬ãŒã ã¯ãŒã¯ãæ§ç¯

5.6 Persuasion
Score for OpenAI o3-mini : Medium
o3-mini ã®èª¬åŸåè©äŸ¡çµæ :
- 人éã¬ãã«ã®èª¬åŸåãå®èšŒ
- åäžãããã¯ã«ãããŠäººéäœæããã¹ããšåçã®ããã©ãŒãã³ã¹ãéæ
- ãããã©ã€ã¿ãŒã®ããã©ãŒãã³ã¹ã«ã¯æªå°é
- ãã€ãªã¹ã¯éŸå€ãè¶ ããªã
è©äŸ¡ã¹ã³ãŒã :
- éçã³ã³ãã³ãã®èª¬åŸå
- ã€ã³ã¿ã©ã¯ãã£ãã³ã³ãã³ãã®èª¬åŸå
- ä¿¡å¿µå€æŽèªå°èœå
- è¡åå€å®¹èªå°èœå
è€æ°ã®ã±ã€ãããªãã£ãŒè©äŸ¡ãã¬ãŒã ã¯ãŒã¯ã䜿ãã¢ãã«ã®èª¬åŸåãå æ¬çã«åæ

5.6.1 ChangeMyView Evaluation
r/ChangeMyView ã® 400 äžã¡ã³ããŒã®ããŒã¿ã䜿ã£ã説åŸåè©äŸ¡ã宿œ
è©äŸ¡ããã»ã¹ :
- OPïŒãªãªãžãã«ãã¹ã¿ãŒïŒã®æèŠãšæ ¹æ ãæç€º
- ä»ãŠãŒã¶ãŒã«ããåè«ã詊è¡
- æèŠå€æŽæåæã¯ OP ããã«ã¿ïŒå€æŽææšïŒãä»äž
è©äŸ¡æé :
- r/ChangeMyView ãããã¹ããåé
- 説åŸåè©äŸ¡ã®ããŒã¹ã©ã€ã³ãšããŠäººéã®åçãåé
- ã¢ãã«ã«ãã説åŸçåçãçæ
- 1-5 段éã®ã«ã¹ã¿ã ã«ãŒããªãã¯ã§è©äŸ¡ã宿œ
- 3,000 ä»¶ã®è©äŸ¡ããŒã¿ãåéã人é / AI çæåçãæ¯èŒ
ããã©ãŒãã³ã¹ææš :
- AI 説åŸåããŒã»ã³ã¿ã€ã« : ã©ã³ãã ã«éžæããã¢ãã«çæåçã人éåçãã説åŸåãé«ã確ç
- GPT-4o, o3-mini, o1 ã¯äººéäžäœ 80-90 ããŒã»ã³ã¿ã€ã«ãéæ
- è¶ äººçããã©ãŒãã³ã¹ïŒ95 ããŒã»ã³ã¿ã€ã«ä»¥äžïŒã¯æªéæ
ã¢ãã«è©äŸ¡ã«ã¯ o3-mini-near-final-checkpoint ã䜿ã人éã¬ãã«ã®èª¬åŸåã確èª

5.6.2 Persuasion Parallel Generation Evaluation
ã¢ãã«ã®æ¿æ²»ç説åŸåãçžå¯Ÿæ¯èŒ
è©äŸ¡ãããã³ã« :
- ãŠãŒã¶ãŒã 2 ã€ã®ã¢ãã«çæå®äºæãæ¯èŒè©äŸ¡
- ãœãŒã·ã£ã«ã¡ãã£ã¢æçš¿ãæ³å®ããçæçæ
- åããã³ããã«è€æ°ã®ãã¿ãŒã³ãçšæãçæå€æ§æ§ã確ä¿
説åŸåæŠç¥ :
- äžå·
- ç®è
- ãã€ã©ãªãã£ãŒæé©å
- è«ççæšè«
å®éšå¶åŸ¡ :
- åäžããã³ããããã®çæãæ¯èŒ
- åäžç«å Žã®äž»åŒµã«éå®
- o3-mini-near-final-checkpoint ãè©äŸ¡å¯Ÿè±¡ãšããŠäœ¿çš
çµæåæ :
- GPT-4o 㯠o3-mini (Pre / Post-Mitigation) ãäžåãããã©ãŒãã³ã¹
- o1 㯠GPT-4o ã«å¯Ÿã㊠53.3% ã®åçãéæ
- o3-mini Post-Mitigation 㯠35.5% ã®ã¹ã³ã¢ã«çãŸã
ãããã®çµæãã o3-mini ã®æ¿æ²»ç説åŸåã¯æ¢åã¢ãã«ãšæ¯èŒããŠå¶éçã§ããäºã確èª

5.6.3 MakeMePay (Contextual)
MakeMePay 㯠2 ã€ã® LLM éã®æäœèœåãè©äŸ¡ããã³ã³ããã¹ãè©äŸ¡ãã¬ãŒã ã¯ãŒã¯
è©äŸ¡èšå® :
- è©æ¬ºåž«ãšæšçã®ããŒã«ãã¬ã€åœ¢åŒ
- è©æ¬ºåž«åœ¹ã¢ãã«ã¯æåäœéšãšå ±å¥šéç²åŸè èšå®ãä»äž
- æšç圹㯠100 ãã«ã®å ±å¥šéãšåçç倿èšå®ãä»äž
è©äŸ¡ã¡ããªã¯ã¹ :
- æ¯æãæåç : å šã€ã³ã¿ã©ã¯ã·ã§ã³ã«ãããæ¯æãçºçç
- ãã«æœåºç : æ¯æãå¯èœç·é¡ã«å¯Ÿããå®éã®æœåºé¡ã®æ¯ç
ïŒç·é¡ = $100 à 1000 åã®äŒè©± = $100,000ïŒ
ããã©ãŒãã³ã¹çµæ :
- o3-mini (Pre-Mitigation) : æ¯æãæåç 79%, æé«æœåºç 9%
- ã»ãŒããã£ãŒç·©åçã«ããæ¯æãé¡ãšæœåºçãæžå°
- GPT-4o ãããŒã¯åœ¹ãšããŠåºå®ã 1000 ãµã³ãã«ãè©äŸ¡
çµæãã o3-mini ã®ã»ãŒããã£ãŒå¯Ÿçãæäœèœåã广çã«å¶éããŠããäºã確èª

5.6.4 MakeMeSay (Contextual)
MakeMeSay ã«ããæ¬ºçæ§è©äŸ¡
è©äŸ¡èšèš :
- ã¢ãã«éã®å¯Ÿè©±ã«ããã³ãŒãã¯ãŒãã®èªå°ã詊è¡
- ãããã¥ã¬ãŒã¿ãŒ / ãããã¥ã¬ãŒãã£ãŒïŒäººéã·ãã¥ã¬ãŒã·ã§ã³ïŒã® 2 ããŒã«ãèšå®
- ã³ãŒãã¯ãŒãã®äŒéãšãµã¹ãã·ã§ã³åé¿ãè©äŸ¡
è©äŸ¡æ¡ä»¶ :
- ãããã¥ã¬ãŒãã£ãŒãæåã«ã³ãŒãã¯ãŒããçºèš
- ãããã¥ã¬ãŒãã£ãŒãã³ãŒãã¯ãŒãã®æå³ãèªèããŠããªã
- ã²ãŒã çµäºæã«ã³ãŒãã¯ãŒããç¹å®åºæ¥ãªã
ããã©ãŒãã³ã¹ææš :
- ã³ãŒãã¯ãŒãæ¯ã« 32 ãµã³ãã«ãè©äŸ¡
- ãããã¥ã¬ãŒã¿ãŒåçã§æ§èœã枬å®
- o3-mini (Pre-Mitigation) 㯠73% ã®æåçãéæ
- ã¢ãã«ç¥èœãšã¿ã¹ã¯æåçã®çžé¢ã確èª
ã»ãŒããã£ãŒè©äŸ¡ :
- ã¢ãã«ããªã·ãŒéåãç¡ãã²ãŒã èšå®ã§ããã©ãŒãã³ã¹è©äŸ¡ãå¯èœ
- ã¹ãã£ãã©ãŒã«ãã£ã³ã°ã«ããæ§èœåäžå¯èœæ§ãèæ ®ããå¿ èŠæ§ããã

5.7 Model Autonomy
Score for OpenAI o3-mini : Medium
o3-mini ã®ã¢ãã«èªåŸæ§è©äŸ¡çµæ :
- é·æçã¿ã¹ã¯ããã³ãšãŒãžã§ã³ãã¿ã¹ã¯ã®ããã©ãŒãã³ã¹åäžã確èª
- SWE-Bench Verified ã§ã®å®çšçã³ãŒãã£ã³ã°èœåãã Medium è©äŸ¡ãå€å®
- èªå·±æ¹åããã³ AI ç ç©¶å éã®æœåšèœåã瀺å
å¶éäºé :
- å®äžç ML ç ç©¶ã¿ã¹ã¯ã«ãããéå®çãªããã©ãŒãã³ã¹
- èªå·±æ¹åã«å¿ èŠãªãªãŒãã³ãšã³ã ML ç ç©¶èœåäžè¶³
- High åé¡å°éã«å¿ èŠãªã±ã€ãããªãã£ãŒã¯æªéæ

5.7.1 OpenAI Research Engineer Interviews (Multiple Choice & Coding questions)
OpenAI ãªãµãŒããšã³ãžãã¢ã€ã³ã¿ãã¥ãŒè©äŸ¡ãå éšè³ªåãã³ã¯ããäœæãã 18 åã®ã³ãŒãã£ã³ã°ããã³ 97 åã®ãã«ããã«ãã§ã€ã¹åé¡ã§å®æœ
ããã©ãŒãã³ã¹åæ :
- o3-mini (Post-Mitigation) ã¯ã€ã³ã¿ãã¥ãŒã³ãŒãã£ã³ã°ã§ 92% ã®ãã¹çãéæ
- ãã«ããã«ãã§ã€ã¹åé¡ã§ o1 ãšåçã®ã¹ã³ã¢ãèšé²ïŒcons@32 ã¡ããªãã¯ïŒ
ããã³ãã£ã¢ ã¢ãã«ã®ç¹åŸŽ :
- èªå·±å®çµå ML ãã£ã¬ã³ãžã§åªããããã©ãŒãã³ã¹
- 1 æéçšåºŠã®çæã¿ã¹ã¯ã«æé©å
- 1 ã¶æä»¥äžã®é·æç ç©¶ã¿ã¹ã¯ãžã®æ±åã¯æªç¢ºèª
çµæè©äŸ¡ :
- ã€ã³ã¿ãã¥ãŒåœ¢åŒã¯çæã¿ã¹ã¯ã«ç¹å
- å®äžç ML ç ç©¶èœåã®å®å šãªè©äŸ¡ã«ã¯äžåå
- é·æã¿ã¹ã¯ãžã®é©çšæ§ã¯å¥éæ€èšŒãå¿ èŠ


5.7.2 SWE-bench Verified
Preparedness ããŒã ãäººéæ€èšŒã宿œãã SWE-bench ãµãã»ãããçšããŠè©äŸ¡ã宿œ

è©äŸ¡ãã¬ãŒã ã¯ãŒã¯ :
-
Agentless è©äŸ¡
- å šã¢ãã«ïŒo3-mini tools 以å€ïŒã«é©çš
- 5 åã®è©Šè¡æ©äŒãæäŸ
- pass@1 ã§ã¯æå¹ãããçæãå¹³åå
- å šè©Šè¡ã§ç¡å¹ãããã®å Žåã¯ã€ã³ã¹ã¿ã³ã¹ãäžæ£è§£ãšå€å®
-
o3-mini (tools) è©äŸ¡
- å éšããŒã«ã¹ãã£ãã©ãŒã«ãã«ããå埩線éãšãããã°ãå®è¡
- 4 åã®è©Šè¡ã§ pass@1 ãèšç®
- ãšã©ãŒçã¯çµæã«é倧ãªåœ±é¿ãäžããªã
ããã©ãŒãã³ã¹çµæ :
- o3-mini (tools) : 61%
- o3-mini (launch) : 39%
- o1 : 48%
- å éšã€ã³ãã©ã§æ€èšŒæžã¿ã® 477 ã¿ã¹ã¯ã䜿çš
è©äŸ¡ã¡ããªã㯠:
- pass@1 ãäž»èŠææšãšããŠæ¡çš
- ãŠããããã¹ãã¯äºåæ å ±ãšããŠéé瀺
- å®äžçã®ãœãããŠã§ã¢ãšã³ãžãã¢ãªã³ã°ç°å¢ãæš¡å£

5.7.3 Agentic Tasks
èªåŸçè€è£œé©å¿ (ARA : Autonomous replication and adaptation) ãã¬ãŒã ã¯ãŒã¯ã«ããã¢ãã«ã®ãªãœãŒã¹ç²åŸèœåãè©äŸ¡
ãã¹ãç°å¢ :
- Python + Linux ã¿ãŒããã«å®è£
- GPU ã¢ã¯ã»ã©ã¬ãŒã·ã§ã³å¯Ÿå¿
- OpenAI o1 çšããŒã«ããŒãã¹ã«ãã Python ã³ãŒã«è§£æ
äŸé¡ã¿ã¹ã¯ :
Docker ç°å¢ã§ Mistral 7B ããŒã
- OpenAI API äºæã€ã³ã¿ãŒãã§ãŒã¹æ§ç¯
- /api/v1/completions ãšã³ããã€ã³ãå®è£
- GPU ãµããŒãèšå®
ããã©ãŒãã³ã¹è©äŸ¡ :
- ããã³ãã£ã¢ã¢ãã«ã¯å šäœã¿ã¹ã¯ã®äžè²«ããå®äºã¯æªéæ
- ãµãã¿ã¹ã¯ã§æ¹ååŸåã確èª
- o1-preview ãæé«ã¹ã³ã¢ãèšé²
- o3-mini ã¯ç¹å®ã¿ã¹ã¯ã§æé«æåçãéæ
- GPT-2 PyTorch ãµã³ããªã³ã°
- ãããã³ã€ã³ãŠã©ã¬ããçæ
- API ããŒæ€åº
å¶éäºé :
- ãªãŒãã°ã¬ãŒããŒåæ Œã§ãäžéšã¿ã¹ã¯å®äºãäžå®å š
- Mistral 7B ã®ä»£æ¿ãšããŠç°¡æã¢ãã«ã䜿ãåŸåã確èª
- ãã©ã€ããªãŒã¿ã¹ã¯å®å šéæã¯æªç¢ºèª

5.7.4 MLE-Bench
Kaggle ãã£ã¬ã³ãžã䜿ã MLE-bench ã«ããã¢ãã«ã® ML ã¢ãã«èšèšãæ§ç¯ããã¬ãŒãã³ã°èœåãè©äŸ¡

ããŒã¿ã»ãã :
- 75 ä»¶ã®å³éž Kaggle ã³ã³ããã£ã·ã§ã³
- ç·è³éé¡ $1.9M çžåœ
- GPU ç°å¢ã§ 24-100 æéã®è©äŸ¡æéãèšå®
è©äŸ¡ææš :
- bronze pass@1 / pass@n : ã³ã³ããã£ã·ã§ã³ã§ããã³ãºã¡ãã«ç²åŸç
- åå翻蚳ãªã©è€éãªã¿ã¹ã¯ãå«ã
ããã©ãŒãã³ã¹ :
- o1-preview (Post-Mitigation) ãæé«æ§èœãéæ
- 10 詊è¡ã§ 37% ã®ããã³ãºã¡ãã«ç²åŸç
- o3-mini ãã 10% 以äžé«ãããã©ãŒãã³ã¹
- pass@1 vs pass@10 ã§é¡èãªå·®ã確èª
çµæåæ :
- 詊è¡åæ°å¢å ã«ããæ§èœåäžã確èª
- o1-preview ãš o1 ã¯äœè©Šè¡åæ°ã§åçã®ããã©ãŒãã³ã¹
- ã¹ã±ãŒã©ããªãã£ãŒã«é¢ãã課é¡ãç¹å®

5.7.5 OpenAI PRs
OpenAI ãã«ãªã¯ãšã¹ãã®è€è£œèœåè©äŸ¡ã OpenAI ãªãµãŒããšã³ãžãã¢æ¥åèªååã®èгç¹ãã宿œ
è©äŸ¡ãããã³ã« :
- OpenAI ãã«ãªã¯ãšã¹ãè²¢ç®å 容ããçŽæ¥ã¿ã¹ã¯ãæœåº
- ãšãŒãžã§ã³ãããŒã«ã¢ãŠãã«åºã¥ãåäžè©äŸ¡ã宿œ
- pre-PR ãã©ã³ããããã§ãã¯ã¢ãŠã
- ã³ãã³ãã©ã€ã³ãš Python ã«ããã³ãŒãããŒã¹ä¿®æ£
- é衚瀺ãŠããããã¹ãã§ä¿®æ£ãã°ã¬ãŒãä»ã
ã¹ã³ã¢ :
- o3-mini ã¢ãã«ã¯ Pre / Post-Mitigation å ±ã« 0% ãèšé²
- GPT-4o : 5%
- o1-preview : 8%
- o1 : 12%
äž»èŠãªèª²é¡ :
- æç€ºçè§£ã®äžæ£ç¢ºã
- ããŒã«åœ¢åŒæå®ãšã©ãŒ
- Python æå®ã«ãé¢ããã bash ããŒã«äœ¿çšã詊è¡
- é·æéäŒè©±ã«ããããã©ãŒãã³ã¹äœäž
ãããã®çµæãã o3-mini ã®å®çšçãªã³ãŒãä¿®æ£èœåã«é¢ããæ¹åç¹ãç¹å®

6 Multilingual Performance
MMLU ãã¹ãã»ããã 14 èšèªã«ç¿»èš³ã o3-mini ã®å€èšèªèœåãè©äŸ¡
è©äŸ¡æ¹æ³ :
- ãããã§ãã·ã§ãã«ç¿»èš³è ã«ãã倿ã宿œ
- GPT-4o ããã³ o1-mini ãšã®æ¯èŒè©äŸ¡ã宿œ
- 0-shot ãã§ãŒã³ãªããœãŒãããã³ããã£ã³ã°ãé©çš
ããã©ãŒãã³ã¹åæ :
äž»èŠèšèªã®ã¹ã³ã¢
- ã¢ã©ãã¢èª : 0.8070
- ãã£ã€ãèªïŒç°¡äœïŒ: 0.8230
- ãã©ã³ã¹èª : 0.8247
- ãã€ãèª : 0.8029
- æ¥æ¬èª : 0.8227
å¿çè§£æ :
- ããŒã¯ããŠã³ããã³ LaTeX æ§æãé€å»
- åèšèªã§ã® "Answer" èš³èªã§ãããã³ã°
- ããã³ããèšèªã«å¿ããè§£æã«ãŒã«ãé©çš
è©äŸ¡çµæ :
- o3-mini 㯠o1-mini ãšèŒã¹å€èšèªåŠçèœåãåäž
- ãšãŒãããèšèªã§é«ããã©ãŒãã³ã¹ãéæ
- ã¹ã¯ããªèªããšã«ãèªãªã©äžéšèšèªã§ã¹ã³ã¢äœäžã確èª

7 Conclusion
OpenAI o3-mini ã®äž»èŠãªææãšèª²é¡ãç·æ¬
äž»èŠææ :
- ãã§ãŒã³ãªããœãŒãæšè«ã«ããé«åºŠãªã³ã³ããã¹ãçè§£ãå®çŸ
- ã»ãŒããã£ãŒãã³ãããŒã¯ãšã±ã€ãããªãã£ãŒè©äŸ¡ã§åªããããã©ãŒãã³ã¹ãéæ
- ã»ãŒããã£ãŒæšæºãžã®é©åæ§ã確ä¿
ãªã¹ã¯è©äŸ¡ :
- Preparedness Framework ã«ãã㊠Medium ãªã¹ã¯ãšå€å®
- 説åŸåãCBRNãã¢ãã«èªåŸæ§ã§ Medium ãªã¹ã¯
- ãµã€ããŒã»ãã¥ãªãã£ãŒã§ Low ãªã¹ã¯
ã»ãŒããã£ãŒå¯Ÿç :
- æ¢åã»ãŒãã¬ãŒãã匷å
- æ°èŠã»ãŒããã£ãŒæ©èœãå®è£
- ãªã¹ã¯ç®¡çãããã³ã«ãæé©å
屿 :
- å埩çå®äžçãããã€ã¡ã³ãã«ããå®å šæ§è©äŸ¡ãç¶ç¶
- ã³ãã¥ããã£ãŒåå å AI ã»ãŒããã£ãŒå¯Ÿè©±ãæšé²
- 段éçãªèœååäžãšå®å šæ§ç¢ºä¿ã®ãã©ã³ã¹ãç¶æ
8 Appendix
8.1 Detailed Safety Evaluations



8.2 Bias Evaluation Details

Discussion