
ãGæ€å®å¯Ÿç#10ããã£ãŒãã©ãŒãã³ã°ããã£ãšããŸãåããèŠçŽ æè¡ïŒæ£èŠåã»ããããã¢ãŠãã»åŠç¿ç
ð ãã®èšäºãèªããšãããããš
ãã£ãŒãã©ãŒãã³ã°ïŒDLïŒã¢ãã«ãå®éã«ãããŸãåãããããã«ç ç©¶è ã»ãšã³ãžãã¢ãç©ã¿äžããŠããèŠçŽ æè¡çŸ€ãäœç³»çã«çè§£ã§ããŸããéã¿ã®åæå€ã»æ£èŠåã»æé©åã¢ã«ãŽãªãºã ã»éåŠç¿å¯Ÿçã»è»¢ç§»åŠç¿ã»è»œéåã»ã¢ãã«ã®è§£éæ§ãŸã§ãGæ€å®ã®é »åºããŒãããŸãããšã«ããŒããŸãã
ã¯ããã«:ãäœã£ããã ãã§ã¯åããªã
ååã®èšäºïŒ#9ïŒã§ã¯ãCNNã»RNNã»Transformerãšãã3倧ã¢ãŒããã¯ãã£ã®æ§é ãåŠã³ãŸããã
ã§ããæ§é ãç¥ã£ãŠãããã ãã§ã¯ãå®éã®ã¢ãã«ã¯ããŸãåŠç¿ããŠãããªãããšãã»ãšãã©ã§ãã
æçã«äŸãããªããã¬ã·ãïŒã¢ãã«ã®æ§é ïŒãæã«å
¥ããŠããç«å æžã®èª¿æŽã»äžããããã»çãä»ãã®ã³ããç¥ããªããã°ãããããæçã«ã¯ãªããŸããã
ãã£ãŒãã©ãŒãã³ã°ãåãã§ãåŠç¿ãããŸãé²ããããã®ã調çæè¡ããå¿
èŠã§ãã
ãã®èšäºã§ã¯ãããããããã£ãŒãã©ãŒãã³ã°ãããŸãåããããã®æè¡éããäœç³»çã«è§£èª¬ããŸãã
Gæ€å®ã§ãé »åºã®éèŠããŒããè©°ãŸã£ãŠããŸãã®ã§ããã£ããèªã¿èŸŒãã§ãã ããã
1. éã¿ã®åæå€åé¡ââåŠç¿ã®ãã¹ã¿ãŒãã©ã€ã³ãã倧äº
ãªãåæå€ãéèŠãªã®ã
ãã£ãŒãã©ãŒãã³ã°ã®åŠç¿ã¯ããã¹ãŠã®éã¿ïŒãã©ã¡ãŒã¿ïŒã«äœããã®åæå€ãèšå®ããããšããå§ãŸããŸãã
ãã®ãã¹ã¿ãŒãã®å€ããè¯ããªããšããã®åŸã®åŠç¿ãããŸããããªãããšãããã£ãŠããŸãã

çŽæçã«èããŠã¿ãŸãããã
ç»å±±ã§ãã©ã®åºçºç¹ããç»ãããã«ãã£ãŠããã©ãçãã«ãŒãããŸã£ããå€ãããŸãã
åæå€ã¯åŠç¿ã®åºçºç¹ãã®ãã®ãªã®ã§ããããæªããã°æé©è§£ã«ãã©ãçããªãïŒãŸãã¯å°éãŸã§æéãããããããïŒã®ã§ãã
åæå€ããã¹ãŠã0ãã«ãããšã©ããªãã
æåã«æãã€ãæ¹æ³ã¯ããšããããå šéš0ã«ããããããããŸããããããããã¯å€§åé¡ãåŒãèµ·ãããŸãã

å
šãŠã®éã¿ãåãå€ïŒ0ïŒã ãšãåãã¥ãŒãã³ãåãèšç®ãããŠãåãåŸé
ãæã€ããšã«ãªããŸãã
ãã®çµæã誀差éäŒææ³ã§æŽæ°ããŠãå
šå¡ãåãããã«åããŠããŸããã倿§ãªç¹åŸŽãåŠã¶ããšããæ·±å±€åŠç¿ã®åŒ·ã¿ãå®å
šã«å€±ãããŸãã
ããã察称æ§ã®åé¡ïŒSymmetry Breaking ProblemïŒãšãããŸãã
ã©ã³ãã åæå€ã«ãèœãšã穎ããã
ããããã©ã³ãã ãªå€ãèšå®ããã°è¯ãïŒããšæããããããŸããã
ãããã«ã©ã³ãã ã¯å¯Ÿç§°æ§ã®åé¡ã解決ããŸãããå€ã®å€§ãããééãããšæ°ããªåé¡ãèµ·ããŸãã
å€ã倧ããããå ŽåïŒåå±€ã§ä¿¡å·ãå¢å¹ ããç¶ããåŸé ççºãèµ·ãããããªã
å€ãå°ããããå ŽåïŒåå±€ã§ä¿¡å·ãæžè¡°ãç¶ããåŸé æ¶å€±ãèµ·ãããããªã
Xavieråæå€ââã·ã°ã¢ã€ãã»tanhåãã®æé©è§£
ãã®åé¡ã解決ããããã«2010幎ã«Xavier Glorotãææ¡ããã®ãXavieråæå€ïŒGlorotåæå€ïŒã§ãã

ã¢ã€ãã¢ã¯ãåå±€ã®åºåã®åæ£ãæãããããšã
å
¥åã®æ°ãå€ããã°å€ãã»ã©ã1ã€1ã€ã®éã¿ã¯å°ãããã¹ãã§ãã
å
·äœçã«ã¯ãåã®å±€ã®ãã¥ãŒãã³æ°ãnãšãããšããå¹³å0ã»æšæºåå·®1÷ânã®æ£èŠååžããéã¿ãçæããŸãã
ã·ã°ã¢ã€ã颿°ã»tanh颿°ã䜿ããšãã«å¹æçã§ããããã®æŽ»æ§å颿°ã¯å ¥åã0ä»è¿ã®ãšãã«æãæåºŠãé«ããããä¿¡å·ã®åæ£ãããŸãæããããšãåŠç¿ã®å®å®ã«ã€ãªãããŸãã
Heåæå€ââReLUåãã®æ¹è¯ç
ReLU颿°ã¯å
¥åã0以äžã®ãšãåºåã0ïŒã€ãŸãæ
å ±ãåå倱ãããïŒãšããç¹æ§ããããŸãã
Xavieråæå€ã¯ãã®ç¹æ§ãèæ
®ããŠããªããããReLUãšçµã¿åããããšåæ£ãåŸã
ã«å°ãããªã£ãŠãããŸãã

ããã§Heåæå€ïŒ2015幎ãKaiming HeïŒã¯ãReLUãååã®æ
å ±ãæšãŠãããšãè£æ£ããããã«æšæºåå·®ãâ2÷ânãšå€§ããã«èšå®ããŸãã
ReLUç³»ã®æŽ»æ§å颿°ïŒLeaky ReLUãªã©ïŒã䜿ããšãã¯Heåæå€ãæšå¥šãããŸãã
å éšå ±å€éã·ããââåæå€ã ãã§ã¯è§£æ±ºã§ããªãåé¡
Xavierã»Heåæå€ã¯ãåŠç¿ã®éå§æç¹ããæŽããŸãããåŠç¿ãé²ãã«ã€ããŠæ°ããªåé¡ãèµ·ããŸãã
ãããå
éšå
±å€éã·ããïŒInternal Covariate ShiftïŒã§ãã
ãå
±å€éã·ããããšã¯å
¥åããŒã¿ã®ååžãå€åããããšãæããŸãã
ãã£ãŒãã©ãŒãã³ã°ã§ã¯ãåã®å±€ã®ãã©ã¡ãŒã¿ãæŽæ°ããããã³ã«ã次ã®å±€ãåãåãããŒã¿ã®ååžïŒå¹³åã»åæ£ïŒãå€åããŸãã
å±€ãæ·±ããªãã»ã©ãã®ããºã¬ã®é£éããç©ã¿éãªããåŠç¿ãäžå®å®ã«ãªããŸãã
ããšãã°ãæ¯æ¥éã仿§ã®ããŒããå±ãå·¥å Žããæ³åããŠãã ããã
ããæ¥ã¯å°ããããŒããå¥ã®æ¥ã¯å€§ããããŒããå±ãã°ãçµã¿ç«ãŠå·¥çšã®èšå®ãæ¯åå€ããªããã°ãªãããäœæ¥å¹çãèããèœã¡ãŸãã
å
éšå
±å€éã·ããã¯ãããšåãç¶æ
ã§ãã
ãã®åé¡ãæ ¹æ¬ãã解決ããã®ã次ã®ã»ã¯ã·ã§ã³ã®ãæ£èŠåå±€ãã§ãã
2. æ£èŠåå±€ã®çš®é¡ââåŠç¿ãå®å®ããããæšæºåãã®ä»çµã¿
æ£èŠåå±€ãšã¯ãé ãå±€ã«å
¥åããããŒã¿ãæ£èŠåïŒæšæºåïŒããããã®å±€ã§ãã
å
éšå
±å€éã·ãããæããåŠç¿ãéãã»å®å®ããã广ããããŸãã
ãããæ£èŠåïŒBatch NormalizationïŒ
2015幎ã«Ioffeã»Szegedyãææ¡ããææ³ã§ããããããå šäœã®ããŒã¿ãåãã£ãã«ããšã«æ£èŠåããŸãã

å
·äœçã«ã¯ã1åã®åŠç¿ïŒãããããïŒã§å
¥åãããè€æ°ã®ããŒã¿ã«ã€ããŠãåããã£ãã«ã®ããŒã¿ããŸãšããŠãå¹³å0ã»åæ£1ãã«ãªããã倿ããŸãã
ããã«ãåŠç¿å¯èœãªãã©ã¡ãŒã¿ïŒã¹ã±ãŒã«ä¿æ° γ ãšç§»åä¿æ° βïŒã§å€æåŸã®å€ã調æŽããŸãã
广ïŒ
å éšå ±å€éã·ãããæå¶ããåŠç¿ãå®å®ãã
æ£åå广ããããéåŠç¿ãèµ·ãã«ãããªã
åŠç¿çã倧ããèšå®ã§ããããã«ãªããåŠç¿ãéããªã
匱ç¹ïŒ ããããµã€ãºãå°ãããšçµ±èšéã®æšå®ãäžå®å®ã«ãªã
ã¬ã€ã€ãŒæ£èŠåïŒLayer NormalizationïŒ
åããŒã¿1ä»¶ããšã«ãå šãã£ãã«ããŸãšããŠæ£èŠåããææ³ã§ãã
ãããæ£èŠåãšç°ãªãããããå
ã®ä»ã®ããŒã¿ãšã¯ç¬ç«ããŠèšç®ãããããããããµã€ãºã«äŸåããŸããã
RNNã»Transformerãªã©ãç³»åããŒã¿ãæ±ãã¢ãã«ã«åããŠããŸãïŒç³»åã®é·ããå¯å€ãªããããããããããŸãæãã«ããïŒã
ã€ã³ã¹ã¿ã³ã¹æ£èŠåïŒInstance NormalizationïŒ
åããŒã¿1ä»¶ããšã»åãã£ãã«ããšã«åå¥ã«æ£èŠåããææ³ã§ãã
ç»å1æã®1ãã£ãã«ãç¬ç«ããŠæ£èŠåããŸãã
ç»åã®ã¹ã¿ã€ã«ïŒè²èª¿ã»è³ªæïŒã«é¢ããæ
å ±ãæ£èŠåãããããç»åã¹ã¿ã€ã«å€æïŒStyleGANçïŒã§ãã䜿ãããŸãã
ã°ã«ãŒãæ£èŠåïŒGroup NormalizationïŒ
åããŒã¿ããšã«ããã£ãã«ãè€æ°ã®ã°ã«ãŒãã«åããŠã°ã«ãŒãå ã§æ£èŠåããææ³ã§ãã
ãããæ£èŠåã®ãããããµã€ãºãå°ãããšäžå®å®ããšãã匱ç¹ãè£ãããã«ææ¡ãããŸããã
ããããµã€ãºã1ã®ãšããå®å®ããŠåäœããŸãã
ç©äœæ€åºã»ã»ã°ã¡ã³ããŒã·ã§ã³ãªã©ãããããµã€ãºã倧ããåããªãã¿ã¹ã¯ã§æå¹ã§ãã
ã°ããŒãã«ã¢ãã¬ãŒãžããŒãªã³ã°ïŒGAPïŒ
æ£èŠåå±€ã§ã¯ãããŸããããéåŠç¿æå¶ãšå¯æ¥ã«é¢ä¿ãããããããã§ç޹ä»ããŸãã
åŸæ¥ã®CNNã¯æåŸã«å
šçµåå±€ïŒFully Connected LayerïŒã䜿ã£ãŠäºæž¬ãè¡ã£ãŠããŸããã
ãããå
šçµåå±€ã¯ãã©ã¡ãŒã¿æ°ãèšå€§ã«ãªãããéåŠç¿ããããããŸãç»åãµã€ãºãå€ãããšäœ¿ããªããšããåé¡ããããŸããã
ã°ããŒãã«ã¢ãã¬ãŒãžããŒãªã³ã°ïŒGAPïŒã¯ãåãã£ãã«ã®ç¹åŸŽãããå
šäœã®å¹³åå€ã1ã€ã®æ°å€ã«éçŽããææ³ã§ãã
ãã©ã¡ãŒã¿ã远å ããã«æ¬¡å
ãåæžã§ããéåŠç¿ãæãã广ããããŸããGoogLeNetïŒ2014幎ïŒãæ¡çšããŠä»¥æ¥ãå€ãã®ã¢ãã«ã§å
šçµåå±€ã®ä»£æ¿ãšããŠäœ¿ãããŠããŸãã
ãããŸã§ã§ãåŠç¿ã®åå°ãæŽããæè¡ããæŒãããŸããã
ããããã¯ãåŠç¿ãããŸãé²ããæè¡ããšãåŠç¿æžã¿ã¢ãã«ã掻ããæè¡ããåŠãã§ãããŸãããã
ããªããšãªãåããŠãããããããªãããŸãåãã®ããããããããã«ãªãã®ãããã®ããŒãã§ãã
ãã²æåŸãŸã§èªã¿é²ããŠãã ããïŒ
ð ã·ãªãŒãºå š15æ¬ããŸãšããŠèªããªãããã¬ãžã³ãå§åçã«ãåŸã§ãïŒ
ååã§å šéšè³Œå ¥ãããš6,000åã§ããããã¬ãžã³ãªã 2,480åïŒçŽ58%ãªãïŒ ã§ã·ãªãŒãºå šäœãèªããŸãã
ãããããGæ€å®ã®ååŒ·ãæ¬æ Œçã«å§ãããããå¹çããåæ Œãããããšããæ¹ã¯ããã²ãã¬ãžã³ããå©çšãã ããã