Dataset Viewer
Auto-converted to Parquet Duplicate
id
stringlengths
16
20
lang
stringclasses
1 value
page_id
int64
12
960k
revision_id
int64
16.5M
1.38B
page_type
stringclasses
3 values
title
stringlengths
1
124
wikidata_qid
stringlengths
2
10
⌀
paragraph_texts
listlengths
0
6.06k
paragraph_sections
listlengths
0
6.06k
paragraph_wikilinks
listlengths
0
6.06k
paragraph_html_tags
listlengths
0
6.06k
linked_articles
listlengths
0
8.67k
categories
listlengths
0
247
templates
listlengths
0
229
num_inlinks
int64
0
393k
redirects
listlengths
1
1.37k
en-12-1383011684
en
12
1,383,011,684
article
Anarchism
Q6199
[ "Anarchism is a political philosophy and movement that seeks to abolish all institutions that perpetuate authority, coercion, or hierarchy, primarily targeting the state and capitalism. Anarchism advocates for the replacement of these hierarchical social structures with stateless societies consisting of horizontal ...
[ "__LEAD__", "__LEAD__", "__LEAD__", "Etymology, terminology, and definition", "Etymology, terminology, and definition", "Etymology, terminology, and definition", "Etymology, terminology, and definition", "History", "History", "History", "History", "History", "History", "History", "Histor...
[ [ { "lang": "en", "title": "Political philosophy", "text": "political philosophy", "start": 15, "end": 35 }, { "lang": "en", "title": "Political movement", "text": "movement", "start": 40, "end": 48 }, { "lang": "en", "title...
[ "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "...
[ "-ism", "1999 Seattle WTO protests", "A Short History of Anarchism", "A priori and a posteriori", "A. John Simmons", "A. S. Neill", "ABC-Clio", "AK Press", "Abolition of monarchy", "Abortion-rights movement", "Abstentionism", "Academy", "Aeschylus", "Affinity group", "Age of Enlightenmen...
[ "Anarchism", "Anti-authoritarianism", "Anti-capitalism", "Anti-fascism", "Far-left politics", "Left-wing ideologies", "Libertarian socialism", "Political ideologies", "Political movements", "Social theories", "Types of socialism" ]
[ "!", "About", "Anarchism", "Anarchism sidebar", "Authority control", "Circa", "Citation", "Cite book", "Cite encyclopedia", "Cite journal", "Cite magazine", "Cite news", "Cite web", "Good article", "Harvc", "ISBN", "Lang", "Langx", "Libertarian socialism navbox", "Libertarianis...
4,583
[ "Anar-kisim", "Anarchism", "Anarchism & society", "Anarchism (political)", "Anarchism (theory)", "Anarchism and Society", "Anarchism and identity politics", "Anarchism and society", "Anarchism and state democracy", "Anarchism and statist democracy", "Anarchism of the Right", "Anarchism of the ...
en-39-1374562957
en
39
1,374,562,957
article
Albedo
Q101038
[ "Albedo (/ælˈbiːdoʊ/ ⓘ al-BEE-doh; from Latin albedo 'whiteness') is the fraction of sunlight that is diffusely reflected by a body. It is measured on a scale from 0 (corresponding to a black body that absorbs all incident radiation) to 1 (corresponding to a body that reflects all incident radiation). Surface albe...
[ "__LEAD__", "__LEAD__", "__LEAD__", "__LEAD__", "Terrestrial albedo", "Terrestrial albedo", "Terrestrial albedo", "Terrestrial albedo", "Terrestrial albedo", "Terrestrial albedo", "Terrestrial albedo", "Terrestrial albedo", "Terrestrial albedo", "Terrestrial albedo", "Terrestrial albedo"...
[ [ { "lang": "en", "title": "Latin", "text": "Latin", "start": 39, "end": 44 }, { "lang": "en", "title": "Sunlight", "text": "sunlight", "start": 85, "end": 93 }, { "lang": "en", "title": "Diffuse reflection", "text": "...
[ "p", "p", "p", "p", "p", "p", "p", "p", "p", "li", "li", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", ...
[ "Absolute magnitude", "Aerosol", "Afforestation", "Air mass", "Aluminium", "Angle of incidence (optics)", "Antarctic", "Antarctica", "Aqua (satellite)", "Arctic", "Asteroid", "Asteroid belt", "Bi-hemispherical reflectance", "Bidirectional reflectance distribution function", "Bifacial sol...
[ "1760s neologisms", "Climate change feedbacks", "Climate forcing", "Climatology", "Electromagnetic radiation", "Land surface effects on climate", "Meteorological quantities", "Radiation", "Radiometry", "Scattering, absorption and radiative transfer (optics)" ]
[ "!", "Authority control", "Cite arXiv", "Cite book", "Cite encyclopedia", "Cite journal", "Cite news", "Cite web", "Climate change", "Convert", "Div col", "Div col end", "Etymology", "Failed verification", "Further", "IPAc-en", "Other uses", "Portal bar", "Reference page", "Ref...
3,139
[ "Albedo", "Albedo effect", "Albedo effects", "Albedo of the Earth", "Albedo rate", "Astronomical albedo", "Earth albedo", "Simple Albedo", "Simple albedo", "Solar reflectance", "Terrestrial albedo" ]
en-290-1378357571
en
290
1,378,357,571
article
A
Q9659
[ "A (minuscule: a) is the first letter and the first vowel letter of the Latin alphabet, used in the modern English alphabet, and others worldwide. Its name in English is a (pronounced as in say), plural aes.", "It is similar in shape to the Ancient Greek letter alpha, from which it derives. The uppercase version ...
[ "__LEAD__", "__LEAD__", "Name", "History", "History", "History", "History", "History", "History", "History", "Use in writing systems", "Pronunciation", "Pronunciation", "Pronunciation", "Pronunciation", "Pronunciation", "Pronunciation", "Pronunciation", "Pronunciation", "Pronun...
[ [ { "lang": "en", "title": "Letter case", "text": "minuscule", "start": 3, "end": 12 }, { "lang": "en", "title": "Letter (alphabet)", "text": "letter", "start": 30, "end": 36 }, { "lang": "en", "title": "Vowel", "text"...
[ "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "p", "li", "li", "li", "li", "li", "li", "p", "p", "p", "li", "li", "li", "li", "li", "li", "li", "li", "li", "li", "li", "li", "li", "li", "li", "li", "li", "li", "li", "li", "p"...
[ "8th century", "A (Cyrillic)", "A (Indic)", "A with breve (Cyrillic)", "A-list", "ABO blood group system", "ASCII", "Aardvark", "Afrikaans", "Aleph", "Aleph number", "Algebra", "Allograph", "Alpha", "Alphabet", "Ancient Greece", "Ancient Greek", "Angstrom", "Ansuz (rune)", "Arc...
[ "ISO basic Latin letters", "Vowel letters" ]
[ "!", "About", "Angbr IPA", "Angle bracket", "Authority control", "Bots", "Break", "CS1 config", "Circa", "Circa no link", "Citation", "Cite AmCyc", "Cite Collier's", "Cite EB1911", "Cite EB9", "Cite NSRW", "Cite book", "Cite encyclopedia", "Cite journal", "Cite news", "Cite w...
573
[ "A", "A (letter)", "ASCII 65", "ASCII 97", "Double-storey a", "Double-story a", "Letter A", "Letter a", "LetterA", "Lowercase a", "Single-storey a", "The Letter A", "U+0041", "U+0061", "\\x41", "Ꞻ", "A" ]
en-303-1382052298
en
303
1,382,052,298
article
Alabama
Q173
["Alabama (/ˌæləˈbæmə/ ⓘ, AL-ə-BAM-ə) is a state in the Southeastern and Deep Southern reg(...TRUNCATED)
["__LEAD__","__LEAD__","__LEAD__","__LEAD__","Etymology","Etymology","Etymology","Etymology","Histor(...TRUNCATED)
[[{"lang":"en","title":"U.S. state","text":"state","start":41,"end":46},{"lang":"en","title":"Southe(...TRUNCATED)
["p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p"(...TRUNCATED)
["1800 United States census","1974 Super Outbreak","1993 Storm of the Century","2010 United States c(...TRUNCATED)
["1819 establishments in the United States","Alabama","Contiguous United States","Southern United St(...TRUNCATED)
["!","About","Alabama","Alabama County Labelled Map","Alabama rail network","Alabama weatherbox","As(...TRUNCATED)
15,012
["22nd State","AL (state)","Agriculture in Alabama","Ala.","Alabahmu","Alabama","Alabama (State)","A(...TRUNCATED)
en-305-1377070946
en
305
1,377,070,946
article
Achilles
Q41746
["In Greek mythology, Achilles (/əˈkɪliːz/ ⓘ ə-KIL-eez) or Achilleus (Ancient Greek: Ἀχι(...TRUNCATED)
["__LEAD__","__LEAD__","__LEAD__","Etymology","Etymology","Etymology","Etymology","Etymology","Etymo(...TRUNCATED)
[[{"lang":"en","title":"Greek mythology","text":"Greek mythology","start":3,"end":18},{"lang":"en","(...TRUNCATED)
["p","p","p","p","p","p","p","li","li","li","li","li","li","li","li","li","li","p","p","p","p","p","(...TRUNCATED)
["Acestor","Achaea Phthiotis","Achaeans (Homer)","Achelous","Acheron","Achille et Déidamie","Achill(...TRUNCATED)
["Achaean Leaders","Achilles","Characters in the Iliad","Characters in the Odyssey","Demigods in cla(...TRUNCATED)
["Apostrophe","Authority control","Blockquote","Characters in the Iliad","Circa","Cite Pauly","Cite (...TRUNCATED)
1,308
["Achiles","Achilles","Achilleus","Achillevs","Achillies","Achillis","Achle","Akhilles","Akhilleus",(...TRUNCATED)
en-307-1381516586
en
307
1,381,516,586
article
Abraham Lincoln
Q91
["Abraham Lincoln (February 12, 1809 – April 15, 1865) was the 16th president of the United State(...TRUNCATED)
["__LEAD__","__LEAD__","__LEAD__","__LEAD__","Early life and education","Early life and education","(...TRUNCATED)
[[{"lang":"en","title":"President of the United States","text":"president of the United States","sta(...TRUNCATED)
["p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p"(...TRUNCATED)
["1848 Whig National Convention","1856 Republican National Convention","1856 United States elections(...TRUNCATED)
["1809 births","1865 deaths","19th-century United States representatives","19th-century members of t(...TRUNCATED)
["!","1856 United States presidential election","1860 United States presidential election","1864 Uni(...TRUNCATED)
9,196
["16th President of America","16th President of USA","16th President of the US","16th President of t(...TRUNCATED)
en-308-1379112110
en
308
1,379,112,110
article
Aristotle
Q868
["Aristotle (Ancient Greek: Ἀριστοτέλης, romanized: Aristotélēs; 384–322 BC) was a(...TRUNCATED)
["__LEAD__","__LEAD__","__LEAD__","__LEAD__","Life","Life","Life","Life","Life","Theoretical philoso(...TRUNCATED)
[[{"lang":"en","title":"Ancient Greek","text":"Ancient Greek","start":11,"end":24},{"lang":"en","tit(...TRUNCATED)
["p","p","p","p","p","p","p","p","p","p","p","li","li","li","li","li","li","p","p","p","p","blockquo(...TRUNCATED)
["Abbasid Caliphate","Accident (philosophy)","Achaemenid Empire","Adaptation","Aeolian Islands","Aes(...TRUNCATED)
["322 BC deaths","384 BC births","4th-century BC Greek mathematicians","4th-century BC Greek writers(...TRUNCATED)
["!","Aesthetics","Ancient Greece topics","Aristotelianism","Authority control","Avoid wrap","BHL au(...TRUNCATED)
5,639
["Aristole","Aristote","Aristotel","Aristotele","Aristotelis","Aristotle","Aristotle of Stagira","Ar(...TRUNCATED)
en-309-1382943932
en
309
1,382,943,932
article
An American in Paris
Q853997
["An American in Paris is a jazz-influenced symphonic poem (or tone poem) for orchestra by American (...TRUNCATED)
["__LEAD__","__LEAD__","__LEAD__","Background","Background","Background","Composition","Composition"(...TRUNCATED)
[[{"lang":"en","title":"Jazz","text":"jazz-influenced","start":26,"end":41},{"lang":"en","title":"Sy(...TRUNCATED)
["p","p","p","p","p","p","p","p","p","p","p","p","blockquote","p","p","p","p","p","p","p","p","p","p(...TRUNCATED)
["24th Academy Awards","Alto saxophone","An American in Paris (film)","Andre Kostelanetz","Années f(...TRUNCATED)
["1928 compositions","Compositions by George Gershwin","Compositions for symphony orchestra","Concer(...TRUNCATED)
["!","About","Authority control","Blockquote","Cbignore","Citation","Cite book","Cite journal","Cite(...TRUNCATED)
144
["American in Paris","An American In Paris","An American in Paris","An American in Paris (compositio(...TRUNCATED)
en-316-1374921434
en
316
1,374,921,434
article
Academy Award for Best Production Design
Q277751
["The Academy Award for Best Production Design recognizes achievement for art direction in film. The(...TRUNCATED)
[ "__LEAD__", "__LEAD__", "Shortlisted finalists", "See also", "See also", "See also", "See also" ]
[[{"lang":"en","title":"Art director","text":"art direction","start":72,"end":85},{"lang":"en","titl(...TRUNCATED)
[ "p", "p", "p", "li", "li", "li", "li" ]
["10 (1979 film)","10th Academy Awards","11th Academy Awards","12 Years a Slave (film)","12th Academ(...TRUNCATED)
["Academy Awards","Awards for best production design","Best Production Design Academy Award winners"(...TRUNCATED)
["Academy Award for Best Production Design","Academy Awards","Cite magazine","Cite news","Cite web",(...TRUNCATED)
1,512
["Academy Award for Art Direction","Academy Award for Best Art Direction","Academy Award for Best Ar(...TRUNCATED)
en-324-1382518809
en
324
1,382,518,809
article
Academy Awards
Q19020
["The Academy Awards, more commonly known as the Oscars, are awards based on artistic and technical (...TRUNCATED)
["__LEAD__","__LEAD__","History","History","History","History","History","History","History","Histor(...TRUNCATED)
[[{"lang":"en","title":"Academy of Motion Picture Arts and Sciences","text":"Academy of Motion Pictu(...TRUNCATED)
["p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p","p"(...TRUNCATED)
["12th Academy Awards","13 Hours: The Secret Soldiers of Benghazi","14th Academy Awards","1917 (2019(...TRUNCATED)
["1929 establishments in California","1953 American television series debuts","1953 English-language(...TRUNCATED)
["!","Academy Awards","Academy Awards lists","Anchor","As of","Authority control","Blockquote","Bull(...TRUNCATED)
19,163
["100th Academy Awards","Acadamy Award","Acadamy Awards","Academy Award","Academy Award Nominations"(...TRUNCATED)
End of preview. Expand in Data Studio

wikipedia-paragraphs

wikipedia-paragraphs is a dataset generated from Wikipedia, designed for natural language processing (NLP) research.

Each entry contains cleaned paragraph text and Wikilink information extracted from a Wikipedia page, along with useful metadata such as categories, templates, and the associated Wikidata QID.

Dataset structure

Configurations

The dataset is organized into multiple configurations, such as enwiki-20261101-v1.3.0. Each configuration corresponds to a specific Wikipedia dump from which the data was sourced. The version number denotes the version of the scripts used to process the data.

Data instances

Each instance in the dataset represents a single Wikipedia page, which can be an article, category, or template.

The following example from the enwiki-20261101-v1.3.0 configuration shows the data generated from the English Wikipedia article on Éclair.

{
  "id": "en-1980219-1378865506",
  "lang": "en",
  "page_id": 1980219,
  "revision_id": 1378865506,
  "page_type": "article",
  "title": "Éclair",
  "wikidata_qid": "Q273426",
  "paragraph_texts": [
    "An éclair (English: /ɪˈklɛər/ ⓘ ih-KLAIR or /eɪˈklɛər/ ay-KLAIR) is a pastry made with choux dough filled with a cream and topped with a flavored icing. The dough, which is the same as that used for profiteroles, is typically piped into an oblong shape with a pastry bag and baked until it is crisp and hollow inside. Once cool, the pastry is filled with custard (crème pâtissière), whipped cream or chiboust cream, then iced with fondant icing. Other fillings include pistachio- and rum-flavored custard, fruit-flavored fillings or chestnut purée. When the icing is caramel, the dessert may be called a bâton de Jacob (lit.\u2009'Jacob's staff'). A similar pastry in a round, rather than oblong, shape is called a religieuse.",
    "The word comes from the French éclair [eklɛːʁ] ⓘ, meaning 'flash of lightning', so named because it is eaten quickly (in a flash); however some believe that the name is due to the glistening of the frosting resembling lightning.",
    "The éclair originated during the 19th century in Lyon, France where it was called pain à la Duchesse ('Duchess-style bread') or petite duchesse ('little duchess') until 1850. The word is first attested both in English and in French in the 1860s.",
    ...
  ],
  "paragraph_sections": [
    "__LEAD__",
    "Etymology",
    "History",
    ...
  ],
  "paragraph_wikilinks": [                                                                                                                             [229/346]
    [
      {
        "lang": "en",
        "title": "Pastry",
        "text": "pastry",
        "start": 70,
        "end": 76
      },
      {
        "lang": "en",
        "title": "Choux pastry",
        "text": "choux",
        "start": 87,
        "end": 92
      },
      {
        "lang": "en",
        "title": "Profiterole",
        "text": "profiteroles",
        "start": 199,
        "end": 211
      },
      ...
    ],
    [
      {
        "lang": "en",
        "title": "Lightning",
        "text": "lightning",
        "start": 68,
        "end": 77
      }
    ],
    [
      {
        "lang": "en",
        "title": "Lyon",
        "text": "Lyon",
        "start": 49,
        "end": 53
      }
    ],
    ...
  ],
  "paragraph_html_tags": [
    "p",
    "p",
    "p",
    ...
  ],
  "linked_articles": [
    "Caramel",
    "Chestnut",
    "Chiboust cream",
    ...
  ],
  "categories": [
    "Belgian desserts",
    "Chocolate-covered foods",
    "Choux pastry",
    ...
  ],
  "templates": [
    "!",
    "Chocolate desserts",
    "Citation needed",
    ...
  ],
  "num_inlinks": 72,
  "redirects": [
    "Bignè",
    "Chocolate Eclair",
    "Chocolate eclair",
    ...
  ]
}

Data Fields

  • id (string): A unique ID for the instance, composed of lang, page_id and revision_id
  • lang (string): The Wikipedia edition code of the page.
  • page_id (int64): The page ID of the page.
  • revision_id (int64): The revision ID of the page.
  • page_type (string): The page type, one of: "article", "category", or "template".
  • title (string): The page title.
  • wikidata_qid (string, optional): The corresponding Wikidata QID, if available.
  • paragraph_texts (list of string): A list of cleaned paragraph texts extracted from the page's HTML.
  • paragraph_sections (list of string): A list of section titles corresponding to each paragraph. __LEAD__ denotes the page's lead section.
  • paragraph_wikilinks (list of list): A nested list where each inner list contains the wikilinks for the corresponding paragraph. Each wikilink object has the following fields:
    • lang (string): The Wikipedia edition code of the target page.
    • title (string): The title of the target page.
    • text (string): The anchor text of the wikilink.
    • start (int64): The starting character index of the anchor text in the paragraph.
    • end (int64): The ending character index of the anchor text in the paragraph.
  • paragraph_html_tags (list of string): The HTML tag from which each paragraph was extracted, one of: "p", "li" "dt", "dd", or "blockquote"
  • linked_articles (list of string): A list of article titles linked from the page.
  • categories (list of string): A list of categories assigned to the page.
  • templates (list of string): A list of templates used on the page.
  • num_inlinks (int64): The number of pages linking to this page. The meaning depends on the page type:
    • For articles, it's the number of pages with wikilinks pointing to this page.
    • For categories, it's the number of pages assigned to this category.
    • For templates, it's the number of pages where this template is used.
  • redirects (list of string): A list of page titles that redirect to this page.

Example usage

The following example shows how to use the dataset to build a corpus of lead paragraphs from articles.

from datasets import load_dataset

# Load the dataset in streaming mode to avoid downloading it all at once
dataset = load_dataset("singletongue/wikipedia-paragraphs", name="enwiki-20261101-v1.3.0", split="train", streaming=True)

# Define a function to filter for relevant article pages
def filter_example(example):
    # Keep only article pages
    if example["page_type"] != "article":
        return False
    
    # Exclude articles that are just lists
    if example["title"].startswith("List of"):
        return False
    
    # Exclude disambiguation pages
    if "Disambiguation" in example["templates"]:
        return False
    
    return True

dataset = dataset.filter(filter_example)

# Define a function to extract and combine all paragraphs from the lead section
def get_lead_paragraph_text(example):
    lead_paragraphs = []
    for text, section in zip(example["paragraph_texts"], example["paragraph_sections"]):
        if section == "__LEAD__":
            lead_paragraphs.append(text)
    
    # Join the paragraphs into a single string, replacing newlines with spaces
    return {"lead_paragraph_text": " ".join(lead_paragraphs).replace("\n", " ")}

dataset = dataset.map(get_lead_paragraph_text)

# Take the first 100 articles and save their lead paragraphs to a file
with open("corpus.txt", "w") as fo:
    for example in dataset.take(100):
        print(example["lead_paragraph_text"], file=fo)

Dataset creation process

This dataset is created from the Wikipedia CirrusSearch dumps and Wikimedia Enterprise HTML dumps. We also utilized the Wikimedia REST API to fetch HTML content that was missing from the dumps.

The code used for creating this dataset is available in the GitHub repository.

License

This dataset is available under the CC-BY-SA 4.0 and GFDL licenses.

Limitation

This dataset is created from specific snapshots of Wikipedia. It may contain outdated information or content that is no longer available on Wikipedia, as well as material that may be considered inappropriate.

Downloads last month
18,833