
ãITãAWS ECSïŒEC2ïŒã§ã¿ã¹ã¯åæ¢åŸãEC2ã®ã¡ã¢ãªãå埩ããªãïŒ
çããŸãããã«ã¡ã¯
æ¬æ¥ã¯ãAWS ECSã§ééããéã®
ãã©ãã«ã·ã¥ãŒãã£ã³ã°ã®å¿åé²ãšãªããŸãã
0. å®éã«èµ·ãã âèªå埩æ§ã®é å»¶â ãäžå¯§ã«è§£èª¬ããïŒå®å šä¿åçïŒ
å
æ¥ãAWS ECSïŒEC2 èµ·åã¿ã€ãïŒãéçšããŠããäžã§ã
å°ãçŠãåºæ¥äºããããŸããã
ãã¿ã¹ã¯ãæå忢ããã®ã«ãEC2 ã®ã¡ã¢ãªããŸã£ããè§£æŸãããã
èªå埩æ§ã®ã¿ã¹ã¯ãé£ç¶ã§èµ·å倱æããã
æçµçã«ã¯ 3 åç®ã®èªåãªãã©ã€ã§ã¿ã¹ã¯èµ·åãæåããŸãããã
ååã§ã¯ âäœãèµ·ããŠããã®ãâ ãåãããæžæããŸããã
ä»åã®èšäºã§ã¯ããã®åºæ¥äºã æç³»åã§äžå¯§ã«æ¯ãè¿ããªããã
ãªãã¡ã¢ãªã峿ã«è§£æŸãããªãã®ã
ECS Agent / Docker / Kernel ã®äžå±€æ§é
ã©ã®ãã°ãèŠãã°åå ãåããã®ã
Amazon Linux 2 ãš 2023 ã®éã
âDockerãã°ãã©ãã«ãç¡ãââŠã¯å®ã¯æ£åžž
ECS Agent ã®ãã¡ã¢ãªäžè¶³ã倿ããžãã¯
ãããŠã次ã«åãç¶æ³ã«ãªã£ããšãã®ããã®ãã§ãã¯ãªã¹ã
ã解説ããŠãããŸãã
â»ãã®èšäºã圹ã«ç«ã€æ¹
ECSïŒEC2 èµ·åã¿ã€ãïŒã§éçšããŠããæ¹
ã¿ã¹ã¯èµ·å倱æãã¡ã¢ãªéŒè¿«ã§å°ã£ãããšãããæ¹
journalctlã»kernelãã°ã®èªã¿æ¹ã«èªä¿¡ããªãæ¹
æ¬çªã§èœã¡çããŠèª¿æ»ã§ããããåãããæ¹
1. å®éã«èµ·ããããšïŒæç³»åïŒ
ãŸãã¯ä»åã®åºæ¥äºã®æµãã§ãã
â EC2 ã®ç©ºãã¡ã¢ãªã10ïŒ ãäžåã
CloudWatch ã¢ã©ãŒã ãçºç«ãã
1å°ã® EC2 ã ãã¡ã¢ãªã極端ã«å°ãªããªã£ãŠããŸããã
⡠該åœã¿ã¹ã¯ã®ã¡ã¢ãªäœ¿çšçãç°åžžã«é«ã
ECS ã¿ã¹ã¯ã¡ããªã¯ã¹ãèŠããšããã®ã¿ã¹ã¯ã ãæ¥äžæã
⢠ã¿ã¹ã¯ãæå忢
忢ããç¬éã« ECS ãµãŒãã¹ã âDesiredCount ãæºããããâ
æ°ããã¿ã¹ã¯ãèªåã§èµ·åããããšããŸãã
⣠ããããèµ·åã§ããªã
1åç® â èµ·å倱æïŒã¡ã¢ãªäžè¶³ïŒ
2åç® â èµ·å倱æïŒãŸã ãã¹ãã¡ã¢ãªãæ»ã£ãŠããªãïŒ
†æ°ååŸã«ãããããã¡ã¢ãªãå埩
LinuxåŽã®ã¡ã¢ãªè§£æŸãé²ã¿ã
3åç®ã®ã¿ã¹ã¯èµ·åãæå
EC2 ã®ã¡ã¢ãªäœ¿çšçãæ£åžžå
ãšããæµãã§ããã
ã§ã¯ãäœæ ãã¿ã¹ã¯åæ¢ïŒå³ã¡ã¢ãªè§£æŸãã«ãªããªãã£ãã®ã§ããããïŒ
2. äœæ ã¿ã¹ã¯åæ¢åŸãã¡ã¢ãªãããè§£æŸãããªãã®ãïŒ
ãã㯠ECSïŒEC2 èµ·åã¿ã€ãïŒã§ã¯çãããããŸããã
äž»ã«æ¬¡ã® 3ã€ãçç±ã§ãã
â Docker ã®åããã»ã¹ãæ®ãïŒå€å ããã»ã¹ïŒ
ã³ã³ãã忢æã« Docker ããã¹ãŠã®åããã»ã¹ã kill ããããã
ãã¹ãOSã«æ®ã£ãŠããŸãã±ãŒã¹ããããŸãã
ç¹åŸŽïŒ
`docker stats` ã«ã¯åºãªã
`top` ã«ã¯æ®ã£ãŠãã
ã¡ã¢ãªãè§£æŸãããŠããªãããã«èŠãã
â¡ Linux ã®ããŒãžãã£ãã·ã¥ã倧éã«æºãŸã£ãŠãã
ð¡ ECSïŒEC2 èµ·åã¿ã€ãïŒã§ã¯ãâã¡ã¢ãªã®äœ¿çšéïŒãã®ç¬éã®ç©ºã容éâ ã§ã¯ãããŸããã
Linux ã¯ç©ºãã¡ã¢ãªãç©æ¥µçã«ãã£ãã·ã¥ã«äœ¿ãããã
ã¿ã¹ã¯åæ¢çŽåŸã«ç©ºãã¡ã¢ãªãæ»ããªãã®ã¯æ£åžžæåã§ãã
ãããã空ãã¡ã¢ãªãå°ãªãïŒãã£ãã·ã¥ã«äœ¿ãåããŠãããç¶æ
ãããåŸãããšããåæãæã£ãŠãããšæ··ä¹±ãã«ãããªããŸãã
Linux ã¯ç©ºãã¡ã¢ãªããç¡é§ããšå€æãã
ç©æ¥µçã«ãã£ãã·ã¥ãšããŠæ¶è²»ããŸãã
ãã®ããã
ã¿ã¹ã¯åé€
ã³ã³ããçµäº
ãããŠãããã£ãã·ã¥éšåã¯ãã°ããæ®ããŸãã
çµæãšã㊠ECS Agent ã
ã空ãã¡ã¢ãªãè¶³ããªãããšå€æããèµ·å倱æããŸãã
çŸç¶ã®ã¡ã¢ãªã»ãã£ãã·ã¥ç¶æ³ã確èªããŸãã
free -h
cat /proc/meminfo | egrep "Cached|Buffers"ãã Cached ãæ°GB å ããŠãããªããã¡ã¢ãªéŒè¿«ã®åå ã¯ãã£ãã·ã¥ã®å¯èœæ§ãé«ãã§ãã
⢠ã¡ã¢ãªäžè¶³ã§ ECS Agent èªäœãäžå®å®ã«ãªã
ã¡ã¢ãªã極端ã«è¶³ããªãå Žåã
ECS Agent ãæ¬¡ã®ãããªãšã©ãŒãè¿ãããšããããŸãã
`not enough memory`
`CannotCreateContainerError`
`ResourceInitializationError`
ã€ãŸããä»åã®ã2åèµ·å倱æ â æ°ååŸã«æåããšããæåã¯
衚é¢çã«ã¯äžå¯è§£ã§ããå
éšçã«ã¯éåžžã«æ£ãã ECS ã®æåãªã®ã§ãã
3. é害調æ»ã§ç¢ºèªãã¹ã OSãã°ïŒæéèŠé ç®ïŒ
ECSïŒEC2èµ·åã¿ã€ãïŒã§ãã©ãã«ãèµ·ãããšãã
åå 㮠倧éšå 㯠OSãã°ãèŠãã°åãããŸãã
ãã°ã¯äž»ã«æ¬¡ã®3ã€ã確èªããŸãã
â ECS Agent ã®ãã°
ã¿ã¹ã¯èµ·å倱æã®çç±ãæžãããŠããŸãã
sudo journalctl -u ecsâ» ECS Agent ã®ãŠãããå㯠AMI ã«ãã ecs ãŸã㯠amazon-ecs-agent ã®å ŽåããããŸãã
â» è£è¶³
æã® Amazon Linux 2 ECS AMI ã§ã¯
`/var/log/ecs/ecs-agent.log` ãååšããŸãããã
çŸåšã® ECS æé©å AMIïŒAL2/AL2023ïŒã¯ journalctl ãåºæ¬ã§ãã
â¡ Docker Daemon ã®ãã°
â» `/var/log/docker.log` ãç¡ãã®ã¯æ£åžžã§ãã
Docker 㯠systemd 管çã®ããã
ãã°ã¯ãã¹ãŠ journald ã«ä¿åãããŸãã
sudo journalctl -u docker ⢠Kernelãã°ïŒOOMã®èšŒæ ïŒ
sudo journalctl -kããã«ã¯ OS ã¬ãã«ã®ã¡ã¢ãªãšã©ãŒã
OOM-Killer ã®å®è¡å±¥æŽããã¹ãŠèšé²ãããŸãã
4. Dockerãã°ã âã©ãã«ããªãâ çç±
ãããã誀解ã§ããã
/var/log/docker
/var/log/docker.log
ãååšããªãã®ã¯ âãã°â ã§ã¯ãããŸããã
ECSïŒEC2 èµ·åã¿ã€ãïŒã§ã¯
䜿ã£ãŠãã AMIã»systemdã»journald ã®èšå® ã«ãã£ãŠ
Docker ã®ãã°åºåå
ãç°ãªããŸãã
ç¹ã«ãæè¿ã® AWS æäŸ AMIïŒECS Optimized AMI ãªã©ïŒã§ã¯ã
Docker 㯠systemd ã«ãã£ãŠèµ·å
ãã°ã¯ journald ã«éçŽããã
`/var/log/docker*` ã ååšããªãã®ã¯æ®é
ãšããæ§æã«ãªã£ãŠããŸãã
ãã®ãããDocker ã®ãã°ã¯æ¬¡ã®ã³ãã³ãã§ç¢ºèªããã®ãæ£è§£ã§ãïŒ
sudo journalctl -u dockerâ» `/var/log/docker` ãç¡ããããšãã£ãŠç°åžžã§ã¯ãããŸããã
å®éã«ã¯ journald ã«æ£ããä¿åãããŠããŸãã
5. Amazon Linux ã®ããŒãžã§ã³å·®ç°ã®æ³šæç¹
Amazon Linux 2 / Amazon Linux 2023 / ECS Optimized AMI ã®éãã¯
OS ããŒãžã§ã³ã AMI äžä»£ã«ãã£ãŠå€ããããã
ãâ¯â¯ã«ã¯å¿
ãÃÃãããããšããæèšã¯ã§ããŸããã
ãŸãã¯ä»¥äžã§â宿©ã®æ£ãã OS ããŒãžã§ã³âã確èªããŸãã
cat /etc/os-releaseâ å®éã®çŸå Žã§ã®ã³ã
ãã°èª¿æ»ã§ã¯æ¬¡ã®ç¹ã ãæŒãããŠããã°ååã§ãïŒ
`/var/log/messages` ããããç°å¢ããããã°ãç¡ãç°å¢ãããã
`/var/log/docker` ãç¡ãå Žå㯠journaldïŒjournalctlïŒã«éçŽãããŠããã ã
ECS Agent ã®ãã°ã
`/var/log/ecs/ecs-agent.log` ã®ç°å¢
`journalctl -u ecs` ã®ç°å¢
ãããåŸã
âOS/AMI ã®éãã«äŸåãããjournalctl ã軞ã«èŠãâ ã®ãæã確å®
6. ãã©ãã«çºçæã®ãå®å šãã§ãã¯ãªã¹ããïŒé ç®ïŒãšäžéšéè€ããŸãïŒ
å®éã®èª¿æ»ã§äœ¿ã£ãã³ãã³ãã§ãã
ðž OS確èª
皌åããŠããOSã®ããŒãžã§ã³ã確èªããŸãã
ããŒãžã§ã³ã«ããå¯ŸåŠæ¹æ³ã確èªãã€ã³ããç°ãªãå ŽåããããŸãã
cat /etc/os-releaseðž ã¡ã¢ãªç¶æ³
docker stats ã«åºãŠããªãã®ã«ãtop ã§ã¯ã¡ã¢ãªãé£ã£ãŠããããã»ã¹ããããªãã
âå€å
ããã»ã¹ïŒorphanïŒâ ãæ®ã£ãŠããå¯èœæ§ãé«ãã§ãã
top -o %MEM
free -h
ps aux --sort=-%mem | headðž ããŒãžãã£ãã·ã¥ã®ãµã€ãºç¢ºèª
Linux 㯠RAM ãã空ãã¡ã¢ãªããšããŠçœ®ããŠããã®ãå«ãã
ãã¡ã€ã«ãã£ãã·ã¥ãšããŠäœ¿ãåãããšããŸãã
â ECS ã¿ã¹ã¯ãæ¢ããŠããããŒãžãã£ãã·ã¥ã¯ããã«ã¯è§£æŸãããªãã
芳枬ãã€ã³ã
free -h
cat /proc/meminfo | egrep "Cached|Buffers"ãã Cached ãæ°GB å ããŠãããªããã¡ã¢ãªéŒè¿«ã®åå ã¯ãã£ãã·ã¥ã§ãã
ðž Dockerã³ã³ãã
docker stats
docker ps -aïŒåã ãã®ç¢ºèªã¯ïŒ
docker stats --no-stream`docker stats`ãééã£ãŠããã¯ã°ã©ã³ãã§åããŠããŸã£ããšãã¯ïŒ
ps aux | grep "docker stats"
kill <PID>ðž ECS Agent
ECSã®ãã°ã確èªããŸãã(-fã¯tailã®æå³)
sudo journalctl -u ecs -fâ» ECS Agent ã®ãŠãããå㯠AMI ã«ãã ecs ãŸã㯠amazon-ecs-agent ã®å ŽåããããŸãã
ðž Dockerãã°
Dockerã®ãã°ã確èªããŸãã(-fã¯tailã®æå³)
sudo journalctl -u docker -fðž Kernel
Kernelã®ãã°ã確èªããŸãã(-fã¯tailã®æå³)
sudo journalctl -k -f7.å¯ŸåŠæ¹æ³ïŒåªå 床é ïŒ
1. ããŒãžãã£ãã·ã¥ã®æåè§£æŸïŒå¿æ¥åŠçœ®ïŒ
ð¡ ãŸã㯠1çªïŒããŒãžãã£ãã·ã¥ã®è§£æŸïŒãã詊ããŠã¿ãã®ãããããã§ãã
çç±ãšããŠãdrop_caches 㯠âEC2 ã®åèµ·åã䌎ããâ
ãµãŒãã¹ã忢ãããã«ãã®ãŸãŸå®è¡ã§ããæã軜ã察åŠã ããã§ãã
ãã£ãã·ã¥ãåå ã§ããã°åé¡ã¯è§£æ¶ãããŸãã
sudo sync; echo 1 | sudo tee /proc/sys/vm/drop_caches# echo 1 : ããŒãžãã£ãã·ã¥ã®ã¿åé€
# echo 2 : inode / dentry ãã£ãã·ã¥ãåé€
# echo 3 : å
šãŠåé€ïŒæã匷åïŒâ ïž æ³šæ
ãã㯠æ¬çªéçšã§ã¯æšå¥šãããªã å¿æ¥åŠçœ®ã§ãã
drop_caches ã¯ç¶æ³ãäžæçã«æ¹åããå¿æ¥åŠçœ®ã§ããã
æ ¹æ¬çãªã¡ã¢ãªãªãŒã¯ã解決ãããã®ã§ã¯ãããŸããã
ç¹ã« æ¬çªç°å¢ã§ã¯ãå®è¡ããå Žåã§ã `echo 1` ãŸã§ã«ãšã©ãã
`echo 2` / `echo 3` 㯠PoCã»æ€èšŒç°å¢ã«éå®ãããããã®åæã§éçšããæ¹ãå®å
šã§ãã
`echo 3` ã¯ãã¡ã€ã«ãã£ãã·ã¥ã»inodeã»dentry ããã¹ãŠè§£æŸããããã
倧éã®åèªã¿èŸŒã¿ãçºçããäžæçã« I/O è² è·ãå¢ããå¯èœæ§ããããŸãã
2. ECS ãšãŒãžã§ã³ãã®åèµ·å
sudo systemctl restart ecs # ãŸã㯠amazon-ecs-agentECS agent ã®äžèª¿ã§ã¿ã¹ã¯ãèµ·åããªãã±ãŒã¹ã«æå¹ã§ãã
3. EC2ããæåã§åèµ·åïŒRebootïŒããã
ã¡ã¢ãªãªãŒã¯ãå€å
ããã»ã¹ã®å Žåã
Reboot ãæãæ©ã確å®ã§ãã
â ãã ã StopïŒåæ¢ïŒã§ã¯ãªã Reboot ããŠãã ãã
Stop â Start ã§ã¯å®äœãšã㊠âå¥ã®ç©çãã¹ãâ ã«æ°èŠèµ·åããããã
EBS ã«ãŒãã®ããŠã³ããå€ãã£ãã
Public IP ã®å€æŽã UserData ã®åå®è¡ãçºçããããšããããŸãã
æé
EC2 â ã€ã³ã¹ã¿ã³ã¹ â åèµ·åïŒRebootïŒ
4. EC2 ã®ã€ã³ã¹ã¿ã³ã¹ã¿ã€ãã®èŠçŽã
ã¡ã¢ãªäžè¶³ãæ ¢æ§çãªãã
t3.micro â t3.small
t3.small â t3.medium
ã®ããã«ã¡ã¢ãªãå¢ããå¿ èŠããããŸãã
ïŒåèïŒå³è§£â ïŒECS / Docker / Kernel ã®ãã°æ§é å³
ïŒAmazon Linux 2 / 2023ïŒ
ââââââââââââââââââââââââââââââââââââââââââ
systemd / journald
ââââââââââââââââââââââââââââââââââââââââââ
â ECS Agent â Docker Daemon â Kernel
â (ecs.service) â (docker.service) â (kernel)
â â â
⌠⌠âŒ
âââââââââââââ âââââââââââââ ãâââââââââââââ
â journalctlâ â journalctlâ ãâ journalctlâ
â -u ecs â â -u docker â ãâ -k â
âââââââââââââ âââââââââââââ ãâââââââââââââïŒåèïŒå³è§£â¡ïŒã¡ã¢ãªäžè¶³æã® ECS Agent ã®å€æããžãã¯å³
ïŒâ»èšäºäžã§æãéèŠãªéšåïŒ
ECS 㯠âã¿ã¹ã¯ãèµ·åã§ãããâ ãæ¬¡ã®ãããªãããŒã§æ±ºããŠããŸãã
ââââââââââââââââââââââââââââââââââââââââââââââ
ð§ ECS AgentïŒã¿ã¹ã¯èµ·å倿ããžãã¯
ââââââââââââââââââââââââââââââââââââââââââââââ
ã¿ã¹ã¯èµ·åèŠæ±ïŒDesiredCount ããïŒ
â
âŒ
ââââââââââââââââââââââââââââââââââââââââââââââ
â â EC2 ã® AvailableMemory / CPU ãç¢ºèª â
ââââââââââââââââââââââââââââââââââââââââââââââ
â
â ã¡ã¢ãªè¶³ããïŒ
âŒ
ââââââââââââââââââââââââââââ
â Yes â
âââââââââââââââââ¬âââââââââââ
âŒ
ââââââââââââââââââââââââââââââââââââââââââââââ
â â¡ Docker ã«ã³ã³ããäœæãäŸé Œ ãã â
ââââââââââââââââââââââââââââââââââââââââââââââ
âŒ
ââââââââââââââââââââââââââââââââââââââââââââââ
â ⢠Docker â Kernel ã«ãªãœãŒã¹èŠæ± ã â
ââââââââââââââââââââââââââââââââââââââââââââââ
âŒ
ââââââââââââââââââââââââââââââââââââââââââââââ
â ⣠KernelïŒã¡ã¢ãªå²åœå¯èœãïŒ ãã â
ââââââââââââââââââââââââââââââââââââââââââââââ
â â
âYes âNoïŒOOM/äžè¶³ïŒ
⌠âŒ
ââââââââââââââââââââââââââââ ââââââââââââââââââââââââââââ
â RUNNINGïŒèµ·åæåïŒ ã â â CannotCreateContainer â
ââââââââââââââââââââââââââââ â not enough memory â
âââââââââââââââââ¬âââââââââââââ
âŒ
ââââââââââââââââââââââââââââ
â ãªãã©ã€ãžïŒæ°åç§åŸïŒãããâ
ââââââââââââââââââââââââââââä»åã®ã2å倱æãã3åç®ã«æåãã¯
ãã®å€æããžãã¯ã«å®å
šã«äžèŽããŠããŸãã
ããŸãïŒããçïŒïŒãã䜿ã journalctl 5éž
journalctl -u ecs -f
# ECS Agent ã®ãã°ããªã¢ã«ã¿ã€ã è¿œå°ŸïŒæéèŠïŒ
journalctl -u docker -f
# Docker Daemon ã®ãã°ã远尟ïŒ/var/log/docker ã¯ç¡ãã®ã§ããïŒ
journalctl -k -f
# Kernelãã°ã远尟ïŒOOMãã¡ã¢ãªãšã©ãŒç¢ºèªïŒ
journalctl --since "1 hour ago"
# çŽè¿1æéã®ãã¹ãŠã®ãã°
journalctl -u ecs -n 50
# ECS Agent ã®ææ°50è¡â»äžã®5ã€ã ãèŠããŠããã°ã
ECSïŒEC2ïŒã§èµ·ãããã©ãã«ã® 8å²ã¯ç¢ºå®ã«èª¿æ»ã§ããŸãã
ãŸãšã
ä»åã®ç¶æ
ã¯äžèŠãããšãé害ãã§ããã
å®éã«ã¯ ECSïŒEC2ïŒãšããŠã¯æ£åžžãªæåã§ããã
ã¿ã¹ã¯åæ¢åŸãã¡ã¢ãªã峿ã«è§£æŸããããšã¯éããªã
ECS Agent â Docker â Kernel ã®äžå±€ã§å€æããŠãã
ãã°ã¯ journalctl ã«äžå åãããŠãã
Amazon Linux ã«ãããã°ã®å Žæãéã
ã¡ã¢ãªãå°ããã€å埩ããéäžã§èµ·å倱æãèµ·ãã
ïŒLinux Kernel ã® âSlab Reclaimâ ã«ããã¡ã¢ãªã¯åŸã ã«è§£æŸãããããã峿ã«ã¯æ»ããªãïŒ
ããããçè§£ããŠãããšã
次ã«åãçŸè±¡ãèµ·ããŠãèœã¡çããŠå¯ŸåŠã§ããŸãã
ãã®èšäºããåã課é¡ãæ±ããæ¹ã®åèã«ãªãã°å¹žãã§ãã
ã§ã¯
ãŸã