HBS Combined Panel Trial FAQs

What is the HBS combined panel trial?

In the 2018 Marsden Funding Round, the Council will trial an alternative assessment model for applications under the HUM, EHB and SOC panels.

The trial will be run in parallel to the standard assessment process for the Expression of Interest (EOI) stage only. Stage two of the assessment process will remain unchanged.

Why are you doing this trial?

MBIE’s recent Assessment of the Fund’s Strategy and Management recommended trialling different assessment models, including looking at broader panels and using a larger pool of assessors.

The Council is keen to use this opportunity to gather evidence for use in continuous improvement of Marsden Funding processes.

What are the main features of the trial?

This is a trial process for the 2018 Marsden Round, covering only the HUM, SOC and EHB panels and only the Expression of Interest stage (not the full proposal stage).

Under the parallel trial process:

• All Expressions of Interest (EOIs) to the HUM, SOC and EHB panels will be considered in a single pool
• EOIs will be scored by ten assessors drawn from a large college of experts
• The college of experts will consist of applicants to current and previous rounds of the Marsden Fund
• Of the 10 assessors, six will be chosen based on their relevant knowledge of the proposal research field, and four will be chosen randomly
• The three Marsden Council members who convene the HUM, EHB and SOC panels will make recommendations on successful EOIs based on the ranked list of assessor scores and cross-discipline moderation discussions
• If a proposal is successful in the trial process, but not in the standard assessment process, the convenors may recommend it is added to the list for progression to the full round
• The full round for applications to the HUM, SOC and EHB panels will be identical to the standard process.

How will funding decisions be made?

The standard process, with three separate panels (HUM, EHB and SOC) will be the primary method to make decisions in the EOI stage for the 2018 Round.

The panel convenors will monitor and review the operation of the parallel trial to assure it has operated as planned. If the Council and convenors are comfortable with the parallel trial process, then those applications ranking highest in the trial will be used to ‘top-up’ the list of proposals from the standard process invited to submit proposals to the full round.  The exact percentage will be determined once the degree of overlap from the two processes is known.  In this way the Council will be able to compare how proposals from the two processes perform in the full round.

What is the problem with the current panel model?

The MBIE Assessment report found that because of its fixed panel structure, the current model may limit the expertise available for proposals, including inter-disciplinary proposals, and this may contribute to risk and perceptions of disciplinary bias.

There is previous evidence from the literature that the disciplinary make-up of assessment panels influences what is funded, and that highly interdisciplinary research tends to have higher academic impact.

The model we are trialling also offers a mechanism to cope with growth of the Fund without creating more panels or increasing the burden on assessors.

How does the new model address these issues?

Under the new model, a larger pool of assessors will be assigned to proposals based on their knowledge of the relevant research fields. This should reduce the chance of bias being introduced by the disciplinary make-up of a fixed panel, and allow an appropriate mix of assessors to be assigned to interdisciplinary proposals.

There are similar models in use overseas. The Australian Research Council uses a similar, but not identical, approach to assess proposals under its Discovery funding scheme.

How will you know if the new model is better than the status quo?

In general it is hard to rigorously evaluate the performance of science funding mechanisms, because it may take many years for the results of funded research to become apparent, and, in the counterfactual, it may be difficult or impossible to track the outcomes for proposals and researchers who did not receive funding.

To evaluate this new model, the Council will look at whether it is better at addressing the known issues and success factors from the literature on science funding. This includes looking at the actual match of research fields achieved between proposals and assessors, and number and success rates of interdisciplinary research proposals, versus the existing model. The Council will also compare how proposals from the two processes perform in the full round, including on international referee scores.

At the same time, we will look at whether the new model is practical and manageable; including monitoring any change in the costs or burden it places on applicants, assessors and the Royal Society.

How did you choose the panels which would be used in the trial?

The Council considered several potential combinations of existing panels for the trial. The decision was made based on the extent of disciplinary overlap and the availability of ongoing expertise on the Council to run and evaluate the trial.

What if I don’t want, or am unable, to be an assessor?

There is an expectation that applicants to the affected panels should be ready to act as assessors in the trial for the EOI stage in the 2018 round.

We encourage applicants to be part of this new initiative to help us improve our knowledge and understanding of the funding system. This is also an opportunity to gain professional development through involvement in peer review, to gain exposure to a broad range of research proposals within and beyond your field, and to understand what makes an excellent Marsden proposal.

We understand that there may be occasional circumstances which preclude applicants from being an assessor (e.g. health, parental leave). Please discuss this with us if you foresee difficulties and we will accommodate you as far as possible.

How much time will being an assessor take?

Approximately 30-60 minutes per proposal, depending on your disciplinary match to the proposal and experience. We estimate the total commitment per assessor to be around 13-27 hours.

Will my discipline lose out?

There is potential for some shift in the mix of disciplines funded if the new model is adopted permanently in subsequent rounds.

The Marsden Council has set goals in the Marsden Investment Plan to both:

• Support bold, innovative research with high potential for scholarly impact; and
• Maintain a New Zealand community of experts in the full, and expanding, range of research fields

The Council will continue to strike a balance between these objectives in its funding decisions.

This includes monitoring the success rate and participation rate of disciplines, encouraging proposals from less successful fields and considering the strategic implications of the balance of disciplines represented across the research funded. The Council will monitor the development of new disciplines and interdisciplinary developments to ensure that important new research areas are not neglected and that assessors with relevant expertise are available.

Isn’t there a conflict of interest from applicants also being assessors?

Conflicts of interest are a risk when applicants are contributing to assessment of proposals which are competing with their own.

The Council has put measures in place to detect and control for potential conflicts of interest under the new process. Applicants will not be scoring their own applications, proposals from their host institution, nor others where they have a known conflict of interest. Any given assessor will only contribute to 10% of the assessment score for a proposal. Outlying scores will also be removed to further reduce the ability of any individual assessor to ‘game’ the system.

The Council will look at the effectiveness of these conflict of interest management measures when it evaluates the trial.

When will the Council decide whether to continue or expand the new assessment model?

The Council will evaluate the trial based on a number of factors and make a decision on assessment processes for the 2019 Round in 2018.

To evaluate the new model, the Council will look at whether it is better at matching the research fields of proposals and assessors, and at the number and success rates of interdisciplinary research proposals, versus the existing model.

The Council will also compare how proposals from the two processes perform in the full round, including on international referee scores.

At the same time, the Council will look at whether the new model is practical and manageable; including monitoring any change in the costs or burden it places on applicants, assessors and the Royal Society.