SHINRA2021-ML: FAQ
- Changes
- Participation
- Account
- Target Languages
- Evaluation
Changes
Has the task changed since last year?
Changes from SHINRA2020-ML are as follows:
– You can participate only in Leaderboard.
– You have an option to utilize the system results of SHINRA2020-ML. In the official evaluation of the subtasks, we will distinguish between the submissions from systems as to whether they use the SHINRA2020-ML system results or not and evaluate them separately.
– SHINRA2021-ML will be carried out independently from NTCIR due to the task schedule.
Participation
How can I participate?
Please follow the participation procedures. Once you create your SHINRA account, you will be able to download SHINRA2021-ML task datasets.
Can I see the data before deciding to participate?
Yes. You can download any of the data once you create your SHINRA account. You are regarded as participants of the task only when you submit your run results. Please check the participation procedures.
Account
Can I download the SHINRA2021-ML task data using my leaderboard account?
To download the SHINRA2021-ML task data, you need to create a SHINRA account apart from the leaderboard account and sign in using the SHINRA account. Please create your SHINRA account at SHINRA: Sign in page.
Target Languages
Do I have to classify all the 30 languages to be a participant?
No. You are expected to select one or more target languages.
Evaluation
Do the official SHINRA2021-ML evaluation and SHINRA2021-ML leaderboard evaluation mean the same thing?
No. The official evaluation of SHINRA2021-ML task is independent of the evaluation of the leaderboard. The former is based on the test set hidden among the target data, while the latter is based on the leaderboard-specific target data.