id stringlengths 16 20 | lang stringclasses 1
value | page_id int64 12 960k | revision_id int64 16.5M 1.38B | page_type stringclasses 3
values | title stringlengths 1 124 | wikidata_qid stringlengths 2 10 ⌀ | paragraph_texts listlengths 0 6.06k | paragraph_sections listlengths 0 6.06k | paragraph_wikilinks listlengths 0 6.06k | paragraph_html_tags listlengths 0 6.06k | linked_articles listlengths 0 8.67k | categories listlengths 0 247 | templates listlengths 0 229 | num_inlinks int64 0 393k | redirects listlengths 1 1.37k |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
en-12-1383011684 | en | 12 | 1,383,011,684 | article | Anarchism | Q6199 | [
"Anarchism is a political philosophy and movement that seeks to abolish all institutions that perpetuate authority, coercion, or hierarchy, primarily targeting the state and capitalism. Anarchism advocates for the replacement of these hierarchical social structures with stateless societies consisting of horizontal ... | [
"__LEAD__",
"__LEAD__",
"__LEAD__",
"Etymology, terminology, and definition",
"Etymology, terminology, and definition",
"Etymology, terminology, and definition",
"Etymology, terminology, and definition",
"History",
"History",
"History",
"History",
"History",
"History",
"History",
"Histor... | [
[
{
"lang": "en",
"title": "Political philosophy",
"text": "political philosophy",
"start": 15,
"end": 35
},
{
"lang": "en",
"title": "Political movement",
"text": "movement",
"start": 40,
"end": 48
},
{
"lang": "en",
"title... | [
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"... | [
"-ism",
"1999 Seattle WTO protests",
"A Short History of Anarchism",
"A priori and a posteriori",
"A. John Simmons",
"A. S. Neill",
"ABC-Clio",
"AK Press",
"Abolition of monarchy",
"Abortion-rights movement",
"Abstentionism",
"Academy",
"Aeschylus",
"Affinity group",
"Age of Enlightenmen... | [
"Anarchism",
"Anti-authoritarianism",
"Anti-capitalism",
"Anti-fascism",
"Far-left politics",
"Left-wing ideologies",
"Libertarian socialism",
"Political ideologies",
"Political movements",
"Social theories",
"Types of socialism"
] | [
"!",
"About",
"Anarchism",
"Anarchism sidebar",
"Authority control",
"Circa",
"Citation",
"Cite book",
"Cite encyclopedia",
"Cite journal",
"Cite magazine",
"Cite news",
"Cite web",
"Good article",
"Harvc",
"ISBN",
"Lang",
"Langx",
"Libertarian socialism navbox",
"Libertarianis... | 4,583 | [
"Anar-kisim",
"Anarchism",
"Anarchism & society",
"Anarchism (political)",
"Anarchism (theory)",
"Anarchism and Society",
"Anarchism and identity politics",
"Anarchism and society",
"Anarchism and state democracy",
"Anarchism and statist democracy",
"Anarchism of the Right",
"Anarchism of the ... |
en-39-1374562957 | en | 39 | 1,374,562,957 | article | Albedo | Q101038 | [
"Albedo (/ælˈbiːdoʊ/ ⓘ al-BEE-doh; from Latin albedo 'whiteness') is the fraction of sunlight that is diffusely reflected by a body. It is measured on a scale from 0 (corresponding to a black body that absorbs all incident radiation) to 1 (corresponding to a body that reflects all incident radiation). Surface albe... | [
"__LEAD__",
"__LEAD__",
"__LEAD__",
"__LEAD__",
"Terrestrial albedo",
"Terrestrial albedo",
"Terrestrial albedo",
"Terrestrial albedo",
"Terrestrial albedo",
"Terrestrial albedo",
"Terrestrial albedo",
"Terrestrial albedo",
"Terrestrial albedo",
"Terrestrial albedo",
"Terrestrial albedo"... | [
[
{
"lang": "en",
"title": "Latin",
"text": "Latin",
"start": 39,
"end": 44
},
{
"lang": "en",
"title": "Sunlight",
"text": "sunlight",
"start": 85,
"end": 93
},
{
"lang": "en",
"title": "Diffuse reflection",
"text": "... | [
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"li",
"li",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
... | [
"Absolute magnitude",
"Aerosol",
"Afforestation",
"Air mass",
"Aluminium",
"Angle of incidence (optics)",
"Antarctic",
"Antarctica",
"Aqua (satellite)",
"Arctic",
"Asteroid",
"Asteroid belt",
"Bi-hemispherical reflectance",
"Bidirectional reflectance distribution function",
"Bifacial sol... | [
"1760s neologisms",
"Climate change feedbacks",
"Climate forcing",
"Climatology",
"Electromagnetic radiation",
"Land surface effects on climate",
"Meteorological quantities",
"Radiation",
"Radiometry",
"Scattering, absorption and radiative transfer (optics)"
] | [
"!",
"Authority control",
"Cite arXiv",
"Cite book",
"Cite encyclopedia",
"Cite journal",
"Cite news",
"Cite web",
"Climate change",
"Convert",
"Div col",
"Div col end",
"Etymology",
"Failed verification",
"Further",
"IPAc-en",
"Other uses",
"Portal bar",
"Reference page",
"Ref... | 3,139 | [
"Albedo",
"Albedo effect",
"Albedo effects",
"Albedo of the Earth",
"Albedo rate",
"Astronomical albedo",
"Earth albedo",
"Simple Albedo",
"Simple albedo",
"Solar reflectance",
"Terrestrial albedo"
] |
en-290-1378357571 | en | 290 | 1,378,357,571 | article | A | Q9659 | [
"A (minuscule: a) is the first letter and the first vowel letter of the Latin alphabet, used in the modern English alphabet, and others worldwide. Its name in English is a (pronounced as in say), plural aes.",
"It is similar in shape to the Ancient Greek letter alpha, from which it derives. The uppercase version ... | [
"__LEAD__",
"__LEAD__",
"Name",
"History",
"History",
"History",
"History",
"History",
"History",
"History",
"Use in writing systems",
"Pronunciation",
"Pronunciation",
"Pronunciation",
"Pronunciation",
"Pronunciation",
"Pronunciation",
"Pronunciation",
"Pronunciation",
"Pronun... | [
[
{
"lang": "en",
"title": "Letter case",
"text": "minuscule",
"start": 3,
"end": 12
},
{
"lang": "en",
"title": "Letter (alphabet)",
"text": "letter",
"start": 30,
"end": 36
},
{
"lang": "en",
"title": "Vowel",
"text"... | [
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"p",
"li",
"li",
"li",
"li",
"li",
"li",
"p",
"p",
"p",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"li",
"p"... | [
"8th century",
"A (Cyrillic)",
"A (Indic)",
"A with breve (Cyrillic)",
"A-list",
"ABO blood group system",
"ASCII",
"Aardvark",
"Afrikaans",
"Aleph",
"Aleph number",
"Algebra",
"Allograph",
"Alpha",
"Alphabet",
"Ancient Greece",
"Ancient Greek",
"Angstrom",
"Ansuz (rune)",
"Arc... | [
"ISO basic Latin letters",
"Vowel letters"
] | [
"!",
"About",
"Angbr IPA",
"Angle bracket",
"Authority control",
"Bots",
"Break",
"CS1 config",
"Circa",
"Circa no link",
"Citation",
"Cite AmCyc",
"Cite Collier's",
"Cite EB1911",
"Cite EB9",
"Cite NSRW",
"Cite book",
"Cite encyclopedia",
"Cite journal",
"Cite news",
"Cite w... | 573 | [
"A",
"A (letter)",
"ASCII 65",
"ASCII 97",
"Double-storey a",
"Double-story a",
"Letter A",
"Letter a",
"LetterA",
"Lowercase a",
"Single-storey a",
"The Letter A",
"U+0041",
"U+0061",
"\\x41",
"Ꞻ",
"A"
] |
en-303-1382052298 | en | 303 | 1,382,052,298 | article | Alabama | Q173 | ["Alabama (/ˌæləˈbæmə/ ⓘ, AL-ə-BAM-ə) is a state in the Southeastern and Deep Southern reg(...TRUNCATED) | ["__LEAD__","__LEAD__","__LEAD__","__LEAD__","Etymology","Etymology","Etymology","Etymology","Histor(...TRUNCATED) | [[{"lang":"en","title":"U.S. state","text":"state","start":41,"end":46},{"lang":"en","title":"Southe(...TRUNCATED) | ["p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p"(...TRUNCATED) | ["1800 United States census","1974 Super Outbreak","1993 Storm of the Century","2010 United States c(...TRUNCATED) | ["1819 establishments in the United States","Alabama","Contiguous United States","Southern United St(...TRUNCATED) | ["!","About","Alabama","Alabama County Labelled Map","Alabama rail network","Alabama weatherbox","As(...TRUNCATED) | 15,012 | ["22nd State","AL (state)","Agriculture in Alabama","Ala.","Alabahmu","Alabama","Alabama (State)","A(...TRUNCATED) |
en-305-1377070946 | en | 305 | 1,377,070,946 | article | Achilles | Q41746 | ["In Greek mythology, Achilles (/əˈkɪliːz/ ⓘ ə-KIL-eez) or Achilleus (Ancient Greek: Ἀχι(...TRUNCATED) | ["__LEAD__","__LEAD__","__LEAD__","Etymology","Etymology","Etymology","Etymology","Etymology","Etymo(...TRUNCATED) | [[{"lang":"en","title":"Greek mythology","text":"Greek mythology","start":3,"end":18},{"lang":"en","(...TRUNCATED) | ["p","p","p","p","p","p","p","li","li","li","li","li","li","li","li","li","li","p","p","p","p","p","(...TRUNCATED) | ["Acestor","Achaea Phthiotis","Achaeans (Homer)","Achelous","Acheron","Achille et Déidamie","Achill(...TRUNCATED) | ["Achaean Leaders","Achilles","Characters in the Iliad","Characters in the Odyssey","Demigods in cla(...TRUNCATED) | ["Apostrophe","Authority control","Blockquote","Characters in the Iliad","Circa","Cite Pauly","Cite (...TRUNCATED) | 1,308 | ["Achiles","Achilles","Achilleus","Achillevs","Achillies","Achillis","Achle","Akhilles","Akhilleus",(...TRUNCATED) |
en-307-1381516586 | en | 307 | 1,381,516,586 | article | Abraham Lincoln | Q91 | ["Abraham Lincoln (February 12, 1809 – April 15, 1865) was the 16th president of the United State(...TRUNCATED) | ["__LEAD__","__LEAD__","__LEAD__","__LEAD__","Early life and education","Early life and education","(...TRUNCATED) | [[{"lang":"en","title":"President of the United States","text":"president of the United States","sta(...TRUNCATED) | ["p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p"(...TRUNCATED) | ["1848 Whig National Convention","1856 Republican National Convention","1856 United States elections(...TRUNCATED) | ["1809 births","1865 deaths","19th-century United States representatives","19th-century members of t(...TRUNCATED) | ["!","1856 United States presidential election","1860 United States presidential election","1864 Uni(...TRUNCATED) | 9,196 | ["16th President of America","16th President of USA","16th President of the US","16th President of t(...TRUNCATED) |
en-308-1379112110 | en | 308 | 1,379,112,110 | article | Aristotle | Q868 | ["Aristotle (Ancient Greek: Ἀριστοτέλης, romanized: Aristotélēs; 384–322 BC) was a(...TRUNCATED) | ["__LEAD__","__LEAD__","__LEAD__","__LEAD__","Life","Life","Life","Life","Life","Theoretical philoso(...TRUNCATED) | [[{"lang":"en","title":"Ancient Greek","text":"Ancient Greek","start":11,"end":24},{"lang":"en","tit(...TRUNCATED) | ["p","p","p","p","p","p","p","p","p","p","p","li","li","li","li","li","li","p","p","p","p","blockquo(...TRUNCATED) | ["Abbasid Caliphate","Accident (philosophy)","Achaemenid Empire","Adaptation","Aeolian Islands","Aes(...TRUNCATED) | ["322 BC deaths","384 BC births","4th-century BC Greek mathematicians","4th-century BC Greek writers(...TRUNCATED) | ["!","Aesthetics","Ancient Greece topics","Aristotelianism","Authority control","Avoid wrap","BHL au(...TRUNCATED) | 5,639 | ["Aristole","Aristote","Aristotel","Aristotele","Aristotelis","Aristotle","Aristotle of Stagira","Ar(...TRUNCATED) |
en-309-1382943932 | en | 309 | 1,382,943,932 | article | An American in Paris | Q853997 | ["An American in Paris is a jazz-influenced symphonic poem (or tone poem) for orchestra by American (...TRUNCATED) | ["__LEAD__","__LEAD__","__LEAD__","Background","Background","Background","Composition","Composition"(...TRUNCATED) | [[{"lang":"en","title":"Jazz","text":"jazz-influenced","start":26,"end":41},{"lang":"en","title":"Sy(...TRUNCATED) | ["p","p","p","p","p","p","p","p","p","p","p","p","blockquote","p","p","p","p","p","p","p","p","p","p(...TRUNCATED) | ["24th Academy Awards","Alto saxophone","An American in Paris (film)","Andre Kostelanetz","Années f(...TRUNCATED) | ["1928 compositions","Compositions by George Gershwin","Compositions for symphony orchestra","Concer(...TRUNCATED) | ["!","About","Authority control","Blockquote","Cbignore","Citation","Cite book","Cite journal","Cite(...TRUNCATED) | 144 | ["American in Paris","An American In Paris","An American in Paris","An American in Paris (compositio(...TRUNCATED) |
en-316-1374921434 | en | 316 | 1,374,921,434 | article | Academy Award for Best Production Design | Q277751 | ["The Academy Award for Best Production Design recognizes achievement for art direction in film. The(...TRUNCATED) | [
"__LEAD__",
"__LEAD__",
"Shortlisted finalists",
"See also",
"See also",
"See also",
"See also"
] | [[{"lang":"en","title":"Art director","text":"art direction","start":72,"end":85},{"lang":"en","titl(...TRUNCATED) | [
"p",
"p",
"p",
"li",
"li",
"li",
"li"
] | ["10 (1979 film)","10th Academy Awards","11th Academy Awards","12 Years a Slave (film)","12th Academ(...TRUNCATED) | ["Academy Awards","Awards for best production design","Best Production Design Academy Award winners"(...TRUNCATED) | ["Academy Award for Best Production Design","Academy Awards","Cite magazine","Cite news","Cite web",(...TRUNCATED) | 1,512 | ["Academy Award for Art Direction","Academy Award for Best Art Direction","Academy Award for Best Ar(...TRUNCATED) |
en-324-1382518809 | en | 324 | 1,382,518,809 | article | Academy Awards | Q19020 | ["The Academy Awards, more commonly known as the Oscars, are awards based on artistic and technical (...TRUNCATED) | ["__LEAD__","__LEAD__","History","History","History","History","History","History","History","Histor(...TRUNCATED) | [[{"lang":"en","title":"Academy of Motion Picture Arts and Sciences","text":"Academy of Motion Pictu(...TRUNCATED) | ["p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p"(...TRUNCATED) | ["12th Academy Awards","13 Hours: The Secret Soldiers of Benghazi","14th Academy Awards","1917 (2019(...TRUNCATED) | ["1929 establishments in California","1953 American television series debuts","1953 English-language(...TRUNCATED) | ["!","Academy Awards","Academy Awards lists","Anchor","As of","Authority control","Blockquote","Bull(...TRUNCATED) | 19,163 | ["100th Academy Awards","Acadamy Award","Acadamy Awards","Academy Award","Academy Award Nominations"(...TRUNCATED) |
wikipedia-paragraphs
wikipedia-paragraphs is a dataset generated from Wikipedia, designed for natural language processing (NLP) research.
Each entry contains cleaned paragraph text and Wikilink information extracted from a Wikipedia page, along with useful metadata such as categories, templates, and the associated Wikidata QID.
Dataset structure
Configurations
The dataset is organized into multiple configurations, such as enwiki-20261101-v1.3.0.
Each configuration corresponds to a specific Wikipedia dump from which the data was sourced.
The version number denotes the version of the scripts used to process the data.
Data instances
Each instance in the dataset represents a single Wikipedia page, which can be an article, category, or template.
The following example from the enwiki-20261101-v1.3.0 configuration shows the data generated from the English Wikipedia article on Éclair.
{
"id": "en-1980219-1378865506",
"lang": "en",
"page_id": 1980219,
"revision_id": 1378865506,
"page_type": "article",
"title": "Éclair",
"wikidata_qid": "Q273426",
"paragraph_texts": [
"An éclair (English: /ɪˈklɛər/ ⓘ ih-KLAIR or /eɪˈklɛər/ ay-KLAIR) is a pastry made with choux dough filled with a cream and topped with a flavored icing. The dough, which is the same as that used for profiteroles, is typically piped into an oblong shape with a pastry bag and baked until it is crisp and hollow inside. Once cool, the pastry is filled with custard (crème pâtissière), whipped cream or chiboust cream, then iced with fondant icing. Other fillings include pistachio- and rum-flavored custard, fruit-flavored fillings or chestnut purée. When the icing is caramel, the dessert may be called a bâton de Jacob (lit.\u2009'Jacob's staff'). A similar pastry in a round, rather than oblong, shape is called a religieuse.",
"The word comes from the French éclair [eklɛːʁ] ⓘ, meaning 'flash of lightning', so named because it is eaten quickly (in a flash); however some believe that the name is due to the glistening of the frosting resembling lightning.",
"The éclair originated during the 19th century in Lyon, France where it was called pain à la Duchesse ('Duchess-style bread') or petite duchesse ('little duchess') until 1850. The word is first attested both in English and in French in the 1860s.",
...
],
"paragraph_sections": [
"__LEAD__",
"Etymology",
"History",
...
],
"paragraph_wikilinks": [ [229/346]
[
{
"lang": "en",
"title": "Pastry",
"text": "pastry",
"start": 70,
"end": 76
},
{
"lang": "en",
"title": "Choux pastry",
"text": "choux",
"start": 87,
"end": 92
},
{
"lang": "en",
"title": "Profiterole",
"text": "profiteroles",
"start": 199,
"end": 211
},
...
],
[
{
"lang": "en",
"title": "Lightning",
"text": "lightning",
"start": 68,
"end": 77
}
],
[
{
"lang": "en",
"title": "Lyon",
"text": "Lyon",
"start": 49,
"end": 53
}
],
...
],
"paragraph_html_tags": [
"p",
"p",
"p",
...
],
"linked_articles": [
"Caramel",
"Chestnut",
"Chiboust cream",
...
],
"categories": [
"Belgian desserts",
"Chocolate-covered foods",
"Choux pastry",
...
],
"templates": [
"!",
"Chocolate desserts",
"Citation needed",
...
],
"num_inlinks": 72,
"redirects": [
"Bignè",
"Chocolate Eclair",
"Chocolate eclair",
...
]
}
Data Fields
- id (string): A unique ID for the instance, composed of
lang,page_idandrevision_id - lang (string): The Wikipedia edition code of the page.
- page_id (int64): The page ID of the page.
- revision_id (int64): The revision ID of the page.
- page_type (string): The page type, one of:
"article","category", or"template". - title (string): The page title.
- wikidata_qid (string, optional): The corresponding Wikidata QID, if available.
- paragraph_texts (list of string): A list of cleaned paragraph texts extracted from the page's HTML.
- paragraph_sections (list of string): A list of section titles corresponding to each paragraph.
__LEAD__denotes the page's lead section. - paragraph_wikilinks (list of list): A nested list where each inner list contains the wikilinks for the corresponding paragraph. Each wikilink object has the following fields:
- lang (string): The Wikipedia edition code of the target page.
- title (string): The title of the target page.
- text (string): The anchor text of the wikilink.
- start (int64): The starting character index of the anchor text in the paragraph.
- end (int64): The ending character index of the anchor text in the paragraph.
- paragraph_html_tags (list of string): The HTML tag from which each paragraph was extracted, one of:
"p","li""dt","dd", or"blockquote" - linked_articles (list of string): A list of article titles linked from the page.
- categories (list of string): A list of categories assigned to the page.
- templates (list of string): A list of templates used on the page.
- num_inlinks (int64): The number of pages linking to this page. The meaning depends on the page type:
- For articles, it's the number of pages with wikilinks pointing to this page.
- For categories, it's the number of pages assigned to this category.
- For templates, it's the number of pages where this template is used.
- redirects (list of string): A list of page titles that redirect to this page.
Example usage
The following example shows how to use the dataset to build a corpus of lead paragraphs from articles.
from datasets import load_dataset
# Load the dataset in streaming mode to avoid downloading it all at once
dataset = load_dataset("singletongue/wikipedia-paragraphs", name="enwiki-20261101-v1.3.0", split="train", streaming=True)
# Define a function to filter for relevant article pages
def filter_example(example):
# Keep only article pages
if example["page_type"] != "article":
return False
# Exclude articles that are just lists
if example["title"].startswith("List of"):
return False
# Exclude disambiguation pages
if "Disambiguation" in example["templates"]:
return False
return True
dataset = dataset.filter(filter_example)
# Define a function to extract and combine all paragraphs from the lead section
def get_lead_paragraph_text(example):
lead_paragraphs = []
for text, section in zip(example["paragraph_texts"], example["paragraph_sections"]):
if section == "__LEAD__":
lead_paragraphs.append(text)
# Join the paragraphs into a single string, replacing newlines with spaces
return {"lead_paragraph_text": " ".join(lead_paragraphs).replace("\n", " ")}
dataset = dataset.map(get_lead_paragraph_text)
# Take the first 100 articles and save their lead paragraphs to a file
with open("corpus.txt", "w") as fo:
for example in dataset.take(100):
print(example["lead_paragraph_text"], file=fo)
Dataset creation process
This dataset is created from the Wikipedia CirrusSearch dumps and Wikimedia Enterprise HTML dumps. We also utilized the Wikimedia REST API to fetch HTML content that was missing from the dumps.
The code used for creating this dataset is available in the GitHub repository.
License
This dataset is available under the CC-BY-SA 4.0 and GFDL licenses.
Limitation
This dataset is created from specific snapshots of Wikipedia. It may contain outdated information or content that is no longer available on Wikipedia, as well as material that may be considered inappropriate.
- Downloads last month
- 18,833