
RedPajama-INCITE ã®Instructã¢ãã«ã®æŽ»çšäºäŸ
以äžã®èšäºãããšã«ããRedPajama-INCITEãã®Instructã¢ãã«ã®æŽ»çšäºäŸããŸãšããŸããã
1. RedPajama-INCITE ã®Instructã¢ãã«
ãRedPajama-INCITE-3B / 7Bãã§ã¯ããBaseããInstructããChatãã®3çš®é¡ã®ã¢ãã«ããªãªãŒã¹ãããŸããã
ãBaseãã¯ãInstructãã¥ãŒãã³ã°åã®ããŒã¹ã¢ãã«ããInstructãã¯Few-Shotã¢ããªã±ãŒã·ã§ã³çšã«Instructãã¥ãŒãã³ã°ããã¢ãã«ããChatãã¯ãã£ããã¢ããªã±ãŒã·ã§ã³çšã«Instructãã¥ãŒãã³ã°ããã¢ãã«ã«ãªããŸãã
ãHELMãã®è©äŸ¡ã§ããRedPajama-INCITE-Instruct-3B-v1ãã¯ãLlama-7Bã(Instructãã¥ãŒãã³ã°ãªã) ã«å
å·®ã§äžåãããRedPajama-INCITE-Instruct-7B-v0.1ãã¯ãLlama-7Bããäžåã£ãŠããŸãã

2. RedPajama-INCITE ã®Instructã¢ãã«ã®æŽ»çšäºäŸ
å®äžçã§ã®LLMã®æŽ»çšäºäŸã®å€ãã¯ããFew-Shotããã³ãããã䜿ãããŠããŸãããªãŒã¬ã«AIã®ã¹ã¿ãŒãã¢ãããStanddãã¯ãåŒè·å£«ãéæ§é åããŒã¿ãããæ³åŸæ¥åãäºæ¥å±éã«é¢ããã€ã³ãµã€ãã®çæãæ¯æŽããŸãã圌ãã¯ããRedPajama 7Bãã䜿ã£ãŠãããŸããŸãªã¿ã¹ã¯ã«ãFew-Shotããã³ãããã䜿çšããŠããŸãã
以äžã¯ãã®äžäŸã«ãªããŸãã
2-1. éèŠãªæç« ã®ãã€ã©ã€ã
éåžžæåãããã³ããã倪åãLLMãçæããã¬ã¹ãã³ã¹ã«ãªããŸãã
You are a sentence highlighter. You highlight sentences that can be used to answer a specific question. To highlight a sentence you return the full text of a sentence in a bulleted list. If none should be highlighted return None.
Question: Who is the cow?
Paragraph: My name is Stephen. I am a cow. My favorite drink is soda.
Highlights:
- My name is Stephen.
- I am a cow.
Question: Who is the duck?
Paragraph: Today was a weird day. The sun was hot and I am tired.
Highlights:
None
Question: What is the largest deer?
Paragraph: The moose (in North America) or elk (in Eurasia) (Alces alces) is a member of the New World deer subfamily and is the only species in the genus Alces. It is the largest and heaviest extant species in the deer family. Most adult male moose have distinctive broad, palmate ("open-hand shaped") antlers; most other members of the deer family have antlers with a dendritic ("twig-like") configuration. Moose typically inhabit boreal forests and temperate broadleaf and mixed forests of the Northern Hemisphere in temperate to subarctic climates. Hunting and other human activities have caused a reduction in the size of the moose's range over time. It has been reintroduced to some of its former habitats. Currently, most moose occur in Canada, Alaska, New England (with Maine having the most of the lower 48 states), New York State, Fennoscandia, the Baltic states, Poland, Kazakhstan, and Russia.
Highlights:
- The moose (in North America) or elk (in Eurasia) (Alces alces) is a member of the New World deer subfamily and is the only species in the genus Alces
- It is the largest and heaviest extant species in the deer family.
Question: What architecture is RedPajama based on?
Paragraph: RedPajama-INCITE-Base-3B-v1 is trained over the RedPajama v1 dataset, with the same architecture as the popular Pythia model suite. We chose to start with the Pythia architecture to understand the value of training with the much larger RedPajama dataset with respect to the current leading open-source dataset, the Pile. Training on Summit leveraged the DeeperSpeed codebase developed by EleutherAI. We are excited to see that at 800B tokens, RedPajama-Base-INCITE-3B has better few-shot performance (measured in HELM, as the average score over 16 core scenarios) and better zero-shot performance (measured in Eleutherâs LM evaluation harness) compared with open models of similar size, including the well-regarded GPT-Neo and Pythia-2.8B (trained with 420B and 300B tokens, respectively, with the Pile). On HELM, it outperforms these models by 3-5 points. On a subset of tasks from lm-evaluation-harness, outperforms these open models by 2-7 points.
Highlights:
- RedPajama-INCITE-Base-3B-v1 is trained over the RedPajama v1 dataset, with the same architecture as the popular Pythia model suite.
- We chose to start with the Pythia architecture to understand the value of training with the much larger RedPajama dataset with respect to the current leading open-source dataset, the Pile.
- Training on Summit leveraged the DeeperSpeed codebase developed by EleutherAI.
ã翻蚳ã
ããªãã¯æç« ã匷調衚瀺ãã人ã§ãã ç¹å®ã®è³ªåã«çããããã«äœ¿çšã§ããæã匷調衚瀺ããŸãã æã匷調衚瀺ããã«ã¯ãæã®å šæãç®æ¡æžããªã¹ãã§è¿ããŸãã äœã匷調衚瀺ããå¿ èŠããªãå Žåã¯ãNone ãè¿ããŸãã
Question: çã¯èª°ã§ãã?
Paragraph: ç§ã®ååã¯ã¹ãã£ãŒãã³ã§ãã ç§ã¯çã§ãã ç§ã®å¥œããªé£²ã¿ç©ã¯ãœãŒãã§ãã
Highlights:
- ç§ã®ååã¯ã¹ãã£ãŒãã³ã§ãã
- ç§ã¯çã§ãã
Question: ã¢ãã«ã¯èª°ã§ãã?
Paragraph: 仿¥ã¯å€ãªæ¥ã§ããã æ¥å·®ããæããŠç²ããŸããã
Highlights:
ãªã
Question: æå€§ã®é¹¿ã¯äœã§ãã?
Paragraph: ãã©ãžã« (åã¢ã¡ãªã«) ãŸãã¯ãã©ãžã« (ãŠãŒã©ã·ã¢) (Alces alces) ã¯ãæ°äžçã·ã«äºç§ã®ã¡ã³ããŒã§ãããAlces å±ã®å¯äžã®çš®ã§ãã çŸåããã·ã«ç§ã®äžã§æå€§ãã€æãéãçš®ã§ãã æäœã®éã®ãã©ãžã«ã®ã»ãšãã©ã¯ãç¹åŸŽçãªå¹ åºã®æç¶ (ãéããæã®åœ¢ã) ã®è§ãæã£ãŠããŸãã ã·ã«ç§ã®ä»ã®ã»ãšãã©ã®ã¡ã³ããŒã¯ãæš¹æç¶ (ãå°æã®ãããªã) æ§é ã®è§ãæã£ãŠããŸãã ãã©ãžã«ã¯éåžžãååçã®æž©åž¯ããäºå¯åž¯æ°åã®åæ¹æãæž©åž¯åºèæš¹æããã³æ··äº€æã«çæ¯ããŠããŸãã ç©çããã®ä»ã®äººéã®æŽ»åã«ããããã©ãžã«ã®çæ¯ç¯å²ã¯æéã®çµéãšãšãã«æžå°ããŠããŸããã 以åã®çæ¯å°ã®äžéšã«åå°å ¥ãããŸããã çŸåšããã©ãžã«ã®ã»ãšãã©ã¯ã«ãããã¢ã©ã¹ã«ããã¥ãŒã€ã³ã°ã©ã³ãïŒã¡ã€ã³å·ãäžäœ48å·ã®ã»ãšãã©ãå ããïŒããã¥ãŒãšãŒã¯å·ããã§ãã¹ã«ã³ãã£ã¢ããã«ãäžåœãããŒã©ã³ããã«ã¶ãã¹ã¿ã³ããã·ã¢ã§çºçããŠããã
Highlights:
- ãã©ãžã« (åã¢ã¡ãªã«) ãŸãã¯ãã©ãžã« (ãŠãŒã©ã·ã¢) (Alces alces) ã¯ãæ°äžçã·ã«äºç§ã®ã¡ã³ããŒã§ãããAlces å±ã®å¯äžã®çš®ã§ãã
- çŸåããã·ã«ç§ã®äžã§æå€§ãã€æãéãçš®ã§ãã
Question: RedPajama ã¯ã©ã®ãããªã¢ãŒããã¯ãã£ã«åºã¥ããŠããŸãã?
Paragraph: RedPajama-INCITE-Base-3B-v1 ã¯ã人æ°ã®ãã Pythia ã¢ãã« ã¹ã€ãŒããšåãã¢ãŒããã¯ãã£ãåãã RedPajama v1 ããŒã¿ã»ããäžã§ãã¬ãŒãã³ã°ãããŸãã çŸåšã®äž»èŠãªãªãŒãã³ãœãŒã¹ ããŒã¿ã»ããã§ãã Pile ãšæ¯èŒããŠãã¯ããã«å€§èŠæš¡ãª RedPajama ããŒã¿ã»ããã䜿çšãããã¬ãŒãã³ã°ã®äŸ¡å€ãçè§£ããããã«ãPythia ã¢ãŒããã¯ãã£ããå§ããããšã«ããŸããã Summit ã®ãã¬ãŒãã³ã°ã§ã¯ãEleutherAI ãéçºãã DeeperSpeed ã³ãŒãããŒã¹ã掻çšããŸããã 800B ããŒã¯ã³ã§ãRedPajama-Base-INCITE-3B ã®å°æ°ã·ã§ãã ããã©ãŒãã³ã¹ (16 ã³ã¢ ã·ããªãªã®å¹³åã¹ã³ã¢ãšã㊠HELM ã§æž¬å®) ãšãŒãã·ã§ãã ããã©ãŒãã³ã¹ (Eleuther ã® LM è©äŸ¡ããŒãã¹ã§æž¬å®) ãåªããŠããããšã確èªããŠè奮ããŠããŸãã ) è©å€ã®é«ã GPT-Neo ããã³ Pythia-2.8B (Pile ã§ãããã 420B ããã³ 300B ããŒã¯ã³ã§ãã¬ãŒãã³ã°ããã) ãå«ããåæ§ã®ãµã€ãºã®ãªãŒãã³ ã¢ãã«ãšæ¯èŒããŸããã HELM ã§ã¯ããããã®ã¢ãã«ããã 3 ïœ 5 ãã€ã³ãåªããŠããŸãã lm-evaluation-harness ã®ã¿ã¹ã¯ã®ãµãã»ããã§ã¯ããããã®ãªãŒãã³ ã¢ãã«ãã 2 ïœ 7 ãã€ã³ãåªããŠããŸãã
Highlights:
- RedPajama-INCITE-Base-3B-v1 ã¯ã人æ°ã®ãã Pythia ã¢ãã« ã¹ã€ãŒããšåãã¢ãŒããã¯ãã£ãåãã RedPajama v1 ããŒã¿ã»ããäžã§ãã¬ãŒãã³ã°ãããŸãã
- çŸåšã®äž»èŠãªãªãŒãã³ãœãŒã¹ ããŒã¿ã»ããã§ãã Pile ãããã¯ããã«å€§èŠæš¡ãª RedPajama ããŒã¿ã»ããã䜿çšãããã¬ãŒãã³ã°ã®äŸ¡å€ãçè§£ããããã«ãPythia ã¢ãŒããã¯ãã£ããå§ããããšã«ããŸããã
- Summit ã®ãã¬ãŒãã³ã°ã§ã¯ãEleutherAI ãéçºãã DeeperSpeed ã³ãŒãããŒã¹ã掻çšããŸããã
2-2. å°çã®è³ªåå¿ç
Please answer the following question:
Question: What is the capital of Canada?
Answer: Ottawa
Question: What is the currency of Switzerland?
Answer: Swiss franc
Question: In which country is Wisconsin located?
Answer: United States
ã翻蚳ã
次ã®è³ªåã«çããŠãã ããã
Question: ã«ããã®éŠéœã¯ã©ãã§ãã?
Answer: ãªã¿ã¯
Question: ã¹ã€ã¹ã®é貚ã¯äœã§ãã?
Answer: ã¹ã€ã¹ãã©ã³
Question: ãŠã£ã¹ã³ã³ã·ã³å·ã¯ã©ãã®åœã«ãããŸãã?
Answer: ã¢ã¡ãªã«
2-3. ææ åæ
Label the tweets as either 'positive', 'negative', 'mixed', or 'neutral':
Tweet: I can say that there isn't anything I would change.
Label: positive
Tweet: I'm not sure about this.
Label: neutral
Tweet: I liked some parts but I didn't like other parts.
Label: mixed
Tweet: I think the background image could have been better.
Label: negative
Tweet: I really like it.
Label: positive
ã翻蚳ã
ãã€ãŒãã«ãpositiveãããnegativeãããmixedãããneutralãã®ããããã®ã©ãã«ãä»ããŸãã
Tweet: 倿Žãããã®ã¯äœããªããšèšããŸãã
Label: positive
Tweet: ããã«ã€ããŠã¯ããããŸããã
Label: neutral
Tweet: ããã€ãã®éšåã¯æ°ã«å ¥ããŸããããä»ã®éšåã¯å¥œãã§ã¯ãããŸããã§ããã
Label: mixed
Tweet: èæ¯ç»åã¯ãã£ãšè¯ãã£ããããããªããšæããŸãã
Label: negative
Tweet: ãšãŠãæ°ã«å ¥ã£ãŠããŸãã
Label: positive