What you will build
A rating form is the form a reviewer fills in for each abstract. By the end of this article, your form will have:
- two or three criteria, each with a scale and a weight, that make up the session's score;
- a recommendation: accept, accept as a poster, or reject;
- a comment to the author, which the author can read once you release it;
- an internal remark for the committee;
- instructions that reviewers read above the questions.
Review Stages, Rating Forms and the Score explains how the score is calculated.
Step 1: Create the form
- On your event home, open Intelligence, then the tile Forms, and choose Create form.
- Enter a Form name, for example "Abstract rating".
- Under Form type, choose Session rating form. The type cannot be changed later.
- Create the form.
You can also start from the presentation type: on its tab Forms, the yellow box under Rating form creates a new rating form in a new browser tab.
Step 2: Add the criteria
For each criterion, for example originality, scientific quality and relevance:
- Drag a field onto the page. A Rating field gives stars; a Whole number field gives a number scale, for example 1 to 10; Radio buttons give a choice with a score for each option.
- Give it a Field name and choose Save field.
- Set Required on this form to Required, so that no review is sent without it.
- Set Review criterion to Yes and give it a Weight. A criterion with weight 2 counts twice as much as one with weight 1.
- For a number scale, set the Lowest score and the Highest score. Leave Higher scores are better ticked unless a low number should mean a good abstract. For radio buttons, give each option its Score.
- In Field description, explain the criterion in one or two sentences, for example "Originality: does the study answer a question that has not been answered before?".
Step 3: Add the recommendation
- Drag a choice field, for example Radio buttons, onto the page and name it "Recommendation".
- Mark it Recommendation.
You do not write the options. run.events writes them for each session: accept; accept as each of the other presentation types the author agreed to, for example a poster; move to another bucket in evaluation; or reject.
Step 4: Add the comment to the author, and an internal remark
- Drag a Long text field onto the page, name it "Comment to the author" and mark it Comment to the author. In its description, remind reviewers that the author may read this comment, without the reviewer's name.
- Drag another Long text field onto the page and name it "Remarks for the committee". Leave it unmarked. It stays internal.
Step 5: Write the instructions
Reviewers read your instructions in two places on the tab Rating, where they fill in the form:
- Above the form: the form's title and description. Open Settings in the form editor and write them in the boxes for each form language. Write here what reviewers need before they start: the scale, how strict to be, when to declare a conflict of interest.
- Under each question: its Field description, from step 2.
The title and description that reviewers see go live with the form, when you save a version and publish it. The Description you saved with Save details only labels the form in your list of forms.
Step 6: Publish the form
Choose Save version, then Publish. Use Preview to walk through the form as a reviewer would.
Step 7: Use the form on the presentation type
- Open Presentation Types, choose the presentation type, and open its tab Forms.
- Under Rating form, choose your form.
- Under Headline score, choose the number the cards and lists show: usually Weighted average of the criteria.
- If you like, set a Disagreement threshold, for example 40: sessions whose reviewers' scores differ by at least that much are flagged.
- Save.
If you review in several stages, each stage can have its own rating form. See How do I run a screening round first, then a full review of the best abstracts?.
Good to know: if no question is marked Review criterion, sessions get no score. With Five icons instead of a rating form, there is no place for instructions; send them to your reviewers by e-mail.
Related articles