How to evaluate call for proposal submissions
As conference organisers, you want to ensure that you build a full, well-rounded and engaging programme that appeals to potential delegates. This programme will typically involve a combination of invited speakers and submissions to open calls for papers, perhaps with some workshops or breakout discussions added to the mix. But sometimes, evaluating all the submissions sent in response to your call for papers can feel like a daunting task. How can you identify the good proposals? Who will be a good speaker? How can the evaluation process be optimised while still ensuring a rigorous review process?
KNE CLUE: THE TIMELINE
The first thing to think about is the timeline – as setting clear deadlines for yourself and your review committee will help ensure the evaluation process is conducted effectively and efficiently, without sacrificing the quality of the reviews.
Review all submissions after the deadline has passed, all together.
Do not send them to the committee in small batches as each submission comes in. Instead, collect all the reviews together, catalogue them, and then send them all to the review committee. Allow at least a week for cataloguing prior to distribution.
Have a clear deadline for the delivery of the reviews. Allow at least three weeks for this process, with a further week as a buffer for late responses.
Allow one more week after this to consolidate and review the responses, and make the final selection.
This equates to a minimum of six weeks (you might require longer, set timelines accordingly but six weeks should be the minimum) – so when the submission deadline has arrived, make sure you send an email to all those who sent in proposals notifying them that acceptance decisions will be sent out by the specified date agreed by you and the review committee.
KNE CLUE: THE EVALUATION CRITERIA
Before organisers and the review committee can conduct an effective review process, they need to consider the criteria for evaluation as these will shape the types of submissions which are preferred, and the key priorities which need to be met.
This might include basic considerations such as originality; relevance to the conference theme; or value to delegates. However it might also involve more nuanced priorities such as geographic representation; practical case studies; or studies on specific sub-topics. Finally, you want to ensure diversity but also embrace thematic alignment so the presentation panels are rich and engaging with good debates during the Q&A.
An important part of the evaluation process relates to the size and format of the programme, so do try and have a rough draft of the agenda mapped out prior to the evaluation process. This will allow you to identify how many speaking spots need to be filled.
So consider:
How many sessions will you have
Will there be parallel panels or not
Will there be round tables as well, and these would take place instead of or alongside presentation panels
Will you be having online or in person speakers, or both
Are there language criteria
The evaluation review and scoring process also needs to be determined, and clearly defined. For example, will everyone review and score every paper or will selected papers go to specific specialists? Will scores be aggregated or will an average be taken? What will happen in the event of a tie break? Use the solution best suited to your review committee, that balances efficiency with feasibility, and does not sacrifice academic rigour.
The main priority is to clearly outline the priorities and fundamental requirements for the review committee.
KNE CLUE: THE PROCESS
- Collect all the reviews and catalogue them.
- Check each submission for alignment to the eligibility criteria (this could include author demographics and conference theme), and remove any that fail to meet these criteria.
- Circulate to the review committee, together with review guidelines and the form or spreadsheet where their responses should be collected.
- Send timely reminders about pending deadlines, but allow enough time to conduct a thorough review.
- Collect and catalogue responses, and break the proposals into three groups – the first tier (those which performed best during the review process): the second tier (the proposals were still relevant to the event, and met the eligibility criteria, but were not as strong or as interesting); and the third tier (those which were not good quality, or lacked substance, or did not meet the eligibility criteria)
- Send out rejection notifications to proposals which were definitely unsuitable. Remember, all rejections should be polite and explain why the submission was unsuccessful and why it was not suitable for this event.
- Send out acceptance notifications to the first-tier submissions with a specified date for confirming the presenter still wishes to attend (usually a week after the acceptance notification). For acceptance notifications, include as much relevant and helpful information as possible relating to the event, deadlines, and speaker requirements.
- Do not notify second-tier submissions immediately, as some of your strongest proposals will be withdrawn (usually anticipate around 10%), so some of the second tier should be held in reserve until all first-tier responses have been received.
For acceptance notifications, include as much relevant and helpful information as possible. If you think there might be further cancellations, have a shortlist of the best unsuccessful proposals and include a note in their notification asking, if the programme changes, would they like to be notified as a replacement speaker.