Review Stages, Rating Forms and the Score

How a scientific committee reviews abstracts in run.events: the rating form, the score it produces, the rounds of review, and who sees what.

Article Image Alt

What the review is for

After the call for abstracts closes, a scientific committee reviews the abstracts, so that the people who decide know which ones are the strongest. A small conference may simply ask each reviewer for a quick verdict. A large congress uses a rating form with several criteria, often in two rounds, and computes a score for each abstract.

run.events supports both. This article explains the parts of the review and how they fit together. The how-to articles linked at the end walk through each part on screen.

The review in six steps

  1. Prepare the rating. You decide how reviewers rate: with five icons or with a rating form. You decide how open the review is, and whether it runs in one round or several.
  2. Prepare the reviewers. You add the reviewers to the event, decide how many abstracts each one rates, which topics they prefer and which institutions they belong to.
  3. Assign the abstracts. You assign reviewers to abstracts, by hand or automatically. Conflicts of interest keep a reviewer away from abstracts they must not rate.
  4. Rate. Each reviewer rates the abstracts.
  5. Follow the progress. The Review dashboard shows what is done and what is missing.
  6. Move on. The best abstracts move to the next round, or to the decision. At the end, the reviewers' comments can be released to the authors.

Who does what

The reviewers hold the event role Content Team Member. They rate, comment, propose changes and declare conflicts of interest. They do not move sessions between buckets or decide.

The people who decide are the content admins: anyone with the organizer role Admin, or with the event role Event Manager or Content Owner. Only content admins move sessions between buckets, override conflicts and release comments. Content admins always see the authors' names, so the members of a committee that reviews blind should hold only Content Team Member.

Good to know: a reviewer is not limited to the abstracts assigned to them. By default, a reviewer sees the sessions in every bucket, and can rate any session in a bucket that is in evaluation, whether it was assigned to them or not. Assignments tell reviewers what you expect them to rate, and the dashboard measures them against it. To keep a reviewer away from some abstracts, open the card Content Team, choose the reviewer, and set Buckets to Selected buckets.

Two ways to rate: five icons or a rating form

A presentation type is rated in one of two ways, which you choose on its tab Forms under Rating form:

  • Five icons: the simple way. Each reviewer chooses one of five icons for each session, and the session's score is the average, from 1 to 5. There is no room for criteria, instructions or comments to the author.
  • A rating form: a form you build in Smart Forms, of the kind Session rating form. You decide the questions, and three settings on them turn the form into a review.

What makes a rating form a review

Each question of a rating form can carry one of these settings:

  • Review criterion, with a Weight: the question counts towards the session's score. A criterion with weight 2 counts twice as much as one with weight 1. Typical criteria are originality, scientific quality and relevance.
  • Recommendation: the question asks the reviewer what should happen with the session. You do not write its options; run.events writes them for each session: accept; accept as each of the other presentation types the author agreed to; move to another bucket in evaluation; or reject.
  • Comment to the author: the answer is what the authors may read later, once the comments are released.

Every other question stays internal, for example "Remarks for the committee".

A rating form can also let reviewers propose changes, with questions marked Proposed title, Proposed labels or Proposed keywords. A content admin then accepts or rejects each proposal.

How the score is calculated

run.events turns every answer to a review criterion into a number from 0 to 100, so that criteria with different scales can be combined:

  • Stars: one star is 0 and five stars are 100.
  • A number question: the criterion's Lowest score is 0 and its Highest score is 100. If Higher scores are better is cleared, the scale is turned round, so that a low number gives a high score.
  • A choice question: each option has its own Score.

Headline score, on the presentation type's tab Forms, decides which number the cards and lists show for a session: None, One criterion, Average of the criteria or Weighted average of the criteria.

An example of the weighted average: a session rated 80 on originality (weight 1) and 50 on scientific quality (weight 2) scores (80 + 2 × 50) / 3 = 60.

Disagreement threshold, on the same tab, flags a session whose reviewers' scores differ by at least this much, on the same scale from 0 to 100. A flagged session deserves a closer look before the decision.

If no question of the form is marked Review criterion, sessions get no score.

Who sees whose name

The review visibility decides who sees whose name during the review:

  • Open: everyone sees all names.
  • Single-blind, the event's default: reviewers see the authors' names. The authors see the reviewers only as "Reviewer 1", "Reviewer 2", and so do the other reviewers in the discussion with the authors.
  • Double-blind: in addition, reviewers do not see who wrote the abstract.

You set the default for the event in Call for Content Settings, tab Review and limits, as Default review visibility. A presentation type can set its own on its tab Forms, and a review stage can set its own too.

These modes apply fully to rating forms. With Five icons, every reviewer who opens a session sees the icons the other reviewers chose, with their names and photos, whatever the mode. For a blind review, use a rating form.

In a double-blind review, an answer in the submission form can still reveal the author, for example an answer to "Institution". Mark such questions Contains identifying information, and reviewers do not see the answers. run.events cannot remove a name that an author wrote into the abstract text itself, so ask authors in your guidelines not to name themselves or their institution there.

Review stages: one round or several

A review stage is one round of review, with its own rating form and its own number of reviews. Many large congresses review in two stages. In the first, the screening, each abstract gets a quick check with a short form. Only the best go on to the second stage, the full review, with the full form.

Every presentation type has its first review stage from the start. You find it on the presentation type's tab Review stages and change it with Edit. Add review stage adds a further round. Each stage has:

  • Stage name, for example Screening.
  • Buckets: the buckets of the type In Evaluation that belong to this stage. A session is in the stage of the bucket it sits in. Each stage needs its own bucket, so create the buckets first.
  • Rating form: Five icons or a rating form.
  • Review visibility for this stage.
  • Due date for assignments: the date by which the reviews are due.
  • Minimum reviews and Maximum reviews: how many reviews each session needs in this stage. A session is fully reviewed once it has the minimum, and automatic assignment uses these numbers.

The Review stages tab of a presentation type, with its stages

Moving abstracts to the next stage

A session moves to the next stage when it is moved into a bucket of that stage. Advance by score, in the bucket menu on the Content Organizer or in the selection bar of the Sessions page, does this for many sessions at once. It takes two scores. Sessions at or above the first move into the next stage. Sessions below the second can move into a declined bucket. Sessions in between stay where they are. Before anything moves, you see which sessions advance, stay or are declined, and you confirm.

Releasing the comments to the authors

The answers to the question marked Comment to the author stay internal until they are released. Comments to the author, on the presentation type's tab Forms, decides when: never (the default), when the speakers are informed of the decision, or as soon as a review is submitted. With Authors may reply, the authors can answer during the Reply window (days). In a blind review, the authors see the reviewers as "Reviewer 1", "Reviewer 2". This works only with a rating form; five icons have no comments.

Following the review

Review, under Report on the Content And Speakers page, opens the Review dashboard: how many of the needed reviews are done, which sessions are waiting, which reviewers are behind, which conflicts stand, and which sessions need attention first.

Good to know: run.events sends reviewers no mail when you add them to the event, when you assign abstracts to them or when a due date passes. Reviewers see their open reviews on their personal start page, on the card Reviewer with the button My reviews. Tell your reviewers yourself when the review starts and how to reach it, and remind those who are behind from your own mailbox.

Related articles