Data Formats

You can download training and test data from here.

  • Japanese Wikipedia categorized into Extend Named Entity [JSON]
{"pageid" : 347190,
"title" : "バッハ家", 
"ENE_id" : "1.4.3",         <-”Family name”
"ENE_score" : 1,
"ENE_annotation" : "HAND.AIP_201904}
  • Language links among 31 language Wikipedias [JSON]
{"source" : { "pageid" : 347190, "lang" : "ja", "title" : "バッハ家" },
 "destination" : { "lang" : "en", "title" : "Bach family", "pageid" : 713599 } }
{"source" : { "pageid" : 347190, "lang" : "ja", "title" : "バッハ家" },
 "destination" : { "lang" : "es", "title" : "Familia Bach", "pageid" : 436582 } }
{"source" : { "pageid" : 347190, "lang" : "ja", "title" : "バッハ家" },
 "destination" : { "lang" : "fa", "title" : "خاندان باخ “, “pageid" : 4515101 } }
{"source" : { "pageid" : 347190, "lang" : "ja", "title" : "バッハ家" },
 "destination" : { "lang" : "fr", "title" : "Famille Bach", "pageid" : 813257 } }
{"source" : { "pageid" : 347190, "lang" : "ja", "title" : "バッハ家" },
 "destination" : { "lang" : "it", "title" : "Bach (famiglia)", "pageid" : 2767443 } }
  • Training data for each language [JSON]
{"pageid": 59153, "title": "Ampersand", "ja_pageid": 5, "ja_title": "アンパサンド", "_stamp": "AUTO.TOHOKU.201906", "ENEs": [{"prob": 0.923977792263031, "ENE_id": "0", "ENE_name": "Concept"}]}
{"pageid": 17524, "title": "Language", "ja_pageid": 10, "ja_title": "言語", "_stamp": "AUTO.TOHOKU.201906", "ENEs": [{"prob": 0.9261491894721985, "ENE_id": "0", "ENE_name": "Concept"}]}
{"pageid": 15606, "title": "Japanese language", "ja_pageid": 11, "ja_title": "日本語", "_stamp": "AUTO.TOHOKU.201906", "ENEs": [{"prob": 0.7623794078826904, "ENE_id": "1.7.24.1", "ENE_name": "National_Language"}]}
  • Wikipedia contents in 31 languages
    • Wikipedia Dump [XML, JSON]
    • Cirrus Search Dump [JSON]
  • Extend Named Entity Definition (English/Japanese) [JSON]

※ The time stamp of All Wikipedia related data is January 20, 2019