
åŸé æ¶å€±ã24å±€ã®ãããã¯ãŒã¯ã§å®æž¬ããããsigmoidã ããã¹ã粟床0.49(åœãŠãã£ãœã)ãŸã§åŽ©å£ããåŸé ã15æ¡æ¶ããŠãã
ååã®éã¿åæåã®èšäºãæžããããšãnoteãå·¡åããŠããããReLUã¯sigmoidãåºåç·åœ¢ã«ç°¡ç¥åããã ãã®ãã®ããšããè¶£æšã®æçš¿ãèŠãããããããã«äž¡è ã¯ãå°ããå ¥åã§ã¯0ä»è¿ã倧ããå ¥åã§ã¯å¢å ããããšãã倧ãŸããªåœ¢ã䌌ãŠããªãããªããã§ãããã®èª¬æã¯ã©ããåŒã£ããã£ããsigmoidã¯åºåã[0,1]ã«åãŸãã飜åããã颿°ã§ãReLUã¯æ£ã®é åã§é倩äºã«äŒžã³ç¶ããã飜åããªãã颿°ã ããã®éãã¯äºçްãªè¿äŒŒèª€å·®ã§ã¯æžãŸãªãæ°ããããä»åã¯ããã®ä¿èª¬ã(1)颿°ãšããŠã®åœãŠã¯ãã(2)å®éã®ãã¥ãŒã©ã«ãããã®åŠç¿ãšãã2ã€ã®è§åºŠããæ€èšŒããã
ð å³ãšè©³ãã説æã¯ããã°æ¬äœã«ãŸãšããŠããŸãã https://shakeailab.com/blog/activation-function-myths/?utm_source=note&utm_medium=referral&utm_campaign=backlog_202609&utm_content=top
ä»åã®å®éšã§èŠããããš
ãReLUã¯sigmoidã®ç°¡ç¥åããšããä¿èª¬ãã(1)ã«ãŒããã£ããã(2)å®éã®æ·±å±€MLPåŠç¿ãã®2æ¹åã§æ€èšŒãã
å®éã«two-moonsã§åŠç¿ããããã¹ã粟床ããReLU/GELUã¯æ·±ã24ãŸã§0.89ã0.95ãç¶æããããsigmoidã¯æ·±ã12ã®0.803ããæ·±ã16ã§0.490(ãã£ã³ã¹ã¬ãã«)ãŸã§åŽç¶ã«åŽ©å£ãã
æ·±ã6(å šå¡ããŸã åŠç¿ã§ããŠããæç¹)ã§ã®æ±ºå®å¢çãæ¯èŒãããšã粟床ã¯ã»ãŒåãã§ãsigmoidã ãå¢çåšãã«å¹ åºãæ»ãããªç¢ºçåŸé åž¯ãæ®ããŠãããä»ã®3ã€ã¯å¢çã®ããå€ã§ã»ãŒç¢ºä¿¡åºŠ100%ã«åãæ¿ãã£ãŠãã

æãåãããŠåãã£ãããš
äžçªæå€ã ã£ãã®ã¯ã厩å£ãç·©ãããªåéã§ã¯ãªãåŽã ã£ãããšã ãsigmoidã®ãã¹ãç²ŸåºŠã¯æ·±ã12ã®0.803ããæ·±ã16ã®0.490ãŸã§ã1段éã§ã»ãŒå šéšå€±ãããããæ·±ãããã»ã©å°ããã€æ§èœãèœã¡ãããšããã€ã¡ãŒãžãæã£ãŠããããå®éã«ã¯ãããæ·±ããŸã§ã¯æã¡ããããŠããããè¶ ãããšäžæ°ã«å£ããããšããåŽç¶ã®åŽ©å£ã ã£ããåŸé ãææ°çã«æžã£ãŠããã°ã©ãèªäœã¯æ»ãããªçŽç·(logã¹ã±ãŒã«ã§)ãªã®ã«ããã®çµæãšããŠèµ·ããåŠç¿ã®æåŠã¯ããªãããªã ããã«ã¯å£ãããéŸå€ãå¢ã«ãªã³/ãªãã®ããã«åãæ¿ããã
ãã®æ°å€ãšæ¯èŒã¯ãæ¬æã§æ±ã£ãããŒã¿ã»å®è£ ã»èšå®ã§ã®çµæã§ãããã¹ãŠã®åé¡ã§åãé åºã«ãªããšããæå³ã§ã¯ãããŸãããæ¡ä»¶ã远å ã®å³ãã³ãŒãã¯ããã°æ¬äœã§ç¢ºèªã§ããŸãã
ð å³ãšè©³ãã説æã¯ããã°æ¬äœã«ãŸãšããŠããŸãã https://shakeailab.com/blog/activation-function-myths/?utm_source=note&utm_medium=referral&utm_campaign=backlog_202609&utm_content=bottom