Minimum learning rate #27

Activity

1. jzhang38 commented on Sep 16, 2023

   

   jzhang38

   on Sep 16, 2023

   Last edited by jzhang38

   Owner

   More actions

   This is a mistake made by us and thanks a million for spotting that out!（by right it should be 4e-5）
   Fortunately, we are still at the very early stage of training and there is still room to correct it. I have corrected this mistake, and the TinyLlama's lr curve will look like below:
   
   I will update the readme pointing to this issue together with the release of the 500B-token checkpoint later.

   ❤️React with ❤️4Reacted by FAb, egeres, Shengyu Liu and DIvAndrey

   👀React with 👀2Reacted by Erik Scholz and FAb
2. jzhang38

   closed this as completedon Sep 16, 2023