SHINRA2020-ML: Run Results Submission for Additional Evaluation

Run Results Submission for Additional Evaluation

    • The participant groups who have already submitted their run results can submit run results for additional evaluation for deeper analyses such as ablation study.
    • The main difference from regular or late submissions are as follows:
      • The name of submission file(.zip):Add_[Group ID]_[Method ID]_[Version ID].zip
      • The last question in the submission form:Confirmation of the naming convention of the submission file
    • The following remains the same as regular or late submissions.
      • For each target language, a group can submit up to three runs based on different methods. (Please contact us if you need our special consideration.) The difference between the methods should be described clearly in your system description report.
      • Run result files should be grouped by method and compressed in a zip file. Use the submission form to upload the zip file.(The submission form for regular/late submissions is modified and reused.)
      • If you might have some trouble and submitted more than one versions for a method, the final version will be used for evaluation.

File naming

Run file name

Each run result file for a target language should be named in the format:
[language code].json ex: en.json

Submission file name

The zip file to upload should be named in the format: Add_[Group ID]_[Method ID]_[Version ID].zip

 ex:’Add_RIKEN_BERT_001.zip’

Group ID
The group ID used for the task registration.
Method ID
The ID to distinguish between the methods used for your runs. Specify the ID using any combination of ASCII alphanumeric characters other than ‘_’.  ex:’BERT’ .
Version ID
The version number used to determine the latest version when you resubmit results based on the same method for some reason. The ID should be specified as 3 digits with leading zeros and the incremental count of submission. ex: ‘001’, ‘002’, ‘003’, …

Submission format

As for submission format, please see the data format page.

Submission form

Please submit the run result files using the following submission form. The form is shared with regular/late submission. Read the How to fill in the submission form beforehand.

Submission form

How to fill in the Submission form 
Group ID
Fill in your Group ID used for registration.
E-mail
Fill in the e-mail address of your representative (principal investigator) or the contact person.
Brief description of the method
Fill in the brief description of the method.
Submission file
Specify the zip file to submit.
Notice that the submission file name for additional test should be specified as: Add_[Group ID]_[Method ID]_[Version ID].zip. 
Data used for the run
All the task data are listed with checkboxes. Check all that apply.
Other external info used for the run
Make a list of all the external info used for the run including the pre-trained model.
Confirmation
・Check that your run files include prediction for all the target data.
・Check that the submission file name is specified as: Add_[Group ID]_[Method ID]_[Version ID].zip. 

Contacts

If you have any questions, contact the followings:

Email to the organizers:
shinra2020ml-info@googlegroups.com

Slack among the participants and organizers:
http://shinra2020-ml.slack.com