Dataset: Comparable Corpus


Dumps Date: 20-09-2017

Arabic: https://dumps.wikimedia.org/arwiki/20170920/
French: https://dumps.wikimedia.org/frwiki/20170920/
English: https://dumps.wikimedia.org/enwiki/20170920/

For each language, I downloaded the flowing files:
  • [LAN]wiki-[DATE]-pages-articles.xml.bz2 
  • [LAN]wiki-[DATE]-pagelinks.sql.gz 
  • [LAN]wiki-[DATE]-categorylinks.sql.gz 
  • [LAN]wiki-[DATE]-langlinks.sql.gz 
  • [LAN]wiki-[DATE]-page.sql.gz 
  • [LAN]wiki-[DATE]-redirect.sql.gz


ISO 639-2 language code
Wikipedia Main Category Name
Wikipedia Disambiguation Category Name

Comments

Post a Comment

Popular posts from this blog

Wikipedia Corpora

Image
 To Do : Create a table comparing WikiQA and SQuAD and say we are going to choose WikiQA (for the moment) WikiQA:  WikiQA is a set of question and sentence pairs, collected and annotated for research on open-domain QA. It includes questions for which there are no correct sentences, enabling researchers to work on answer triggering. WikiQA introduced the task of answer triggering and was the only answer triggering dataset. Questions type The questions are originally sampled from  Bing query logs . This corpus has 3047 questions (in raw data file   WikiQA.tsv  ). It contains both general and factoid questions.  -HOW: eg.  how did Athenians make money? how does interlibrary loan work? ( NO WHY Questions ) -What, How many, Who, when, Where etc. Date/Version  Version 1.0: August 25, 2015 Answer triggering - We propose the answer triggering task, a new challenge for the question answering problem, which requires QA s...
Read more

Links

DISEQuA Multisix corpus Bitbucket Account JWPL DataMachine Book:Speech and Language Processing(IR, IE, linguistic definitions) An example of a bilingual corpus Datasets for Natural Language Processing RNN QCRI Live Translator QCRI MT QCRI MT Article Multilingual QAS (steps explanation) Link for QA English datasets TF-IDF Links From Dr. Alberto Book : Machine Translation evaluation-Likert Scale Corpora referneces - QCRI -Memex "DISEQuA" and "Multisix" corpora
Read more

Extraction Model

Extraction Model:A nswer sentence selection Due to increasing complexity in question answering, deep learning has become a popular trend in solving difficult problems. Approaches Word Count:  counts the number of non-stopwords in the question that also occur in the answer sentence.  Weighted Word Count:  re-weights the counts by the IDF values of the question words. LCLR :makes use of rich lexical semantic features, including word/lemma matching, WordNet and vector-space lexical semantic models. Convolutional Neural Network Recurrent Neural Network
Read more