SHINRA2021-ML: Results Submission

Results Submission

  • For each target language, a group can submit up to three runs based on different methods. (Please contact us if you need our special consideration.) The difference between the methods should be described clearly in your system description report.
  • Run result files should be grouped by method and compressed in a zip file. Use the submission form to upload the zip file.
  • If you might have some trouble and submitted more than one versions for a method, the final version will be used for evaluation.

File naming

Run file name

Each run result file for a target language should be named in the format:
[language code].json ex: en.json

Submission file name

The zip file to upload should be named in the format: [Group ID]_[Method ID]_[Version ID].zip


Group ID
The group ID used for the task registration.
Method ID
The ID to distinguish between the methods used for your runs. Specify the ID using any combination of ASCII alphanumeric characters other than ‘_’.  ex:’BERT’ .
Version ID
The version number used to determine the latest version when you resubmit results based on the same method for some reason. The ID should be specified as 3 digits with leading zeros and the incremental count of submission. ex: ‘001’, ‘002’, ‘003’, …

Submission format

As for submission format, please see the data format page.

Submission form

Please submit the run result files using the following submission form. Read the How to fill in the submission form beforehand.

Submission form

How to fill in the Submission form 
Group ID
Fill in your Group ID used for registration.
Fill in the e-mail address of your representative (principal investigator) or the contact person.
Brief description of the method
Fill in the brief description of the method.
Submission file
Specify the zip file to submit.
Data used for the run
All the task data are listed with checkboxes. Check all that apply.
Other external info used for the run
Make a list of all the external info used for the run including the pre-trained model.
Check that your run files include prediction for all the target data. 


If you have any questions, contact the followings:

Email to the organizers:

Slack among the participants and organizers: