Explore as a

Information for panellists

Information for panellists assessing Rutherford Discovery Fellowship applications

The Panellists' Guidelines are available as a PDF: RDF Panellist Guidelines 2019

Proposal guidelines for Panellists

About the Rutherford Discovery Fellowships

The Rutherford Discovery Fellowships are administered by the Royal Society Te Apārangi (the Society) for the New Zealand Government.

The Fellowships will develop and foster the future leaders in the New Zealand science and innovation system[1].  They will attract and retain New Zealand’s most talented early- to mid-career researchers and encourage their career development by enabling them to establish a track record for future research leadership. It is expected that Fellows, throughout their careers, will contribute to positive outcomes for New Zealand.

Receipt of a Rutherford Discovery Fellowship is expected to have significant value in the future career development and leadership potential of a researcher.

Fellowships are awarded on a full-time basis of which at least 85% (or 0.85 FTE) of the Fellow’s time must be dedicated to the research objectives identified in the proposal, unless an exemption to this requirement has been approved by the Society. The remainder of their time may be used for other research, teaching and non-research related development opportunities.

The scheme will award a minimum contribution of $70,000 per year towards the researcher's salary, $60,000 in research related expenses, and $30,000 per year for the host organisations to support the Fellow’s research programme.

Eligibility

For the purpose of this scheme, early-to mid-career researchers are researchers whose doctoral degrees were conferred between three and eight years prior to the year in which the Fellowship is awarded. For the 2019 funding round, applicants must have a PhD conferment date between 01 January 2011 and 31 December 2016. Eligibility can be extended where applicants have an allowable career interruption, including maternity/parental leave, medical leave, part-time employment because of ongoing childcare responsibilities, or as otherwise agreed to by the Society. Applicants that are the primary caregiver of a dependent child, are also able to extend the period of eligibility by two years per child, to account for career interruptions experienced due to being the primary caregiver for young children. The extension of two years per dependent child is inclusive of any periods of parental leave.

All applications forwarded for review fulfil the eligibility requirements.

 

Timetable

Date

Activity

Thu 28 Feb 2019

Proposals On-Line web-based application system opens.

Thu 11 Apr 2019

On-Line web portal closes at 5 pm (New Zealand Standard Time).

Thu 16 May 2019

Deadline for receipt of applicant-solicited referee reports by the Secretariat of the Rutherford Discovery Fellowships at 5 pm (New Zealand Standard Time).

Thu 27 Jun 2019

Last day for discipline-based panellists to submit their recommendations to the Secretariat.

Early Jul 2019

The long-listed proposals are sent to the interview panel.

Tue 06 Aug 2019

Last day for interview panellists to submit their recommendations to the Secretariat.

Mid Aug 2019

Interview panel selects a short list of candidates to interview.

Mid Sep 2019

Interviews conducted by the interview panel.  Venues TBC

Oct 2019 (TBC)

Results announced.

 

Table 1.         Timetable for 2019

Assessment Process (in brief)

The Society will appoint a selection panel, chaired by the President of the Society, or their nominee, to oversee the selection process.  The Chair of the panel will work with the Society’s nominated manager to determine the best process to be used. The assessment of proposals is a two stage process.

Stage one is the assessment of all proposals by three discipline-based panels. Discipline-based panellists are asked to participate in a briefing video conference prior to beginning their assessment, but do not otherwise meet. Each discipline-based panellist grades the proposals within their panel and then submits their grades on an electronic form.  All proposals are graded against three criteria and panellists are obliged to consider the three applicant-solicited referees to produce their final scores for the applicants.

Once the overall scores from the panellists have been received, the Rutherford Discovery Fellowship Secretariat will produce an ordered list of applicants with the highest grades from each of the discipline-based panels.  These top ranking applications will form the Long List for consideration by the interview panel.  The number of applicants from each panel on the Long List will be determined by the number of proposals submitted (Table 2).

For information, discipline panellists will receive overall panel scores and Long List following stage 2a of the process.

 

Panel

Number of proposals submitted to each discipline-based panel

Number of proposals submitted    (% of total)

Long List (number)

HSS

49

30%

12

LFS

56

35%

14

PEM

57

35%

14

Totals

162

100%

40

Table 2.         Example distribution of proposals if 162 applications were to be received.

 Processflowdiagramforweb

(Stage 1) Discipline-based panel recommendation

 

(Stage 2a) Interview panel consideration

(Stage 2b) Call to interview

Figure 1.     Process flow

Stage two is in two parts: (a) the assessment of the Long List of applicants by the interview panel; and, (b) interviewing a shortlist of applicants and making recommendations for the successful Fellows.  The Chair and four member interview panel will conduct the interviews.

 

Discipline-based panels (Stage one)

Each of the three research areas will have a discipline-based assessment panel.  The panel comprises researchers who are experts in their field, have a broad knowledge of the research area and are experienced in assessment.  Panel members are appointed by the Society under consent from the Chair of the selection panel.  These panels are advisory only, providing recommendations on the relative merits of proposals to the interview panel.  The three research areas are:

Humanities and the Social Sciences (HSS).

Research related to the human condition or aspects of human society.

This includes, but not limited to: English; languages; history; religion; philosophy; law; classics; linguistics; literature; cultural studies; media studies; art history; film; economics; education; psychology (cognitive, social, developmental, organisational, community and health); cognitive science; linguistics; archaeology; anthropology; sociology; social, cultural and human geography; social anthropology; architecture, urban design and environmental studies; public health; nursing; public policy; marketing; political science; and business studies.

Life Sciences (LFS).

Research related to understanding the activities that occur in cells and tissues and the interrelationships between organisms and their environment.

This includes, but not limited to: physiology (plant or animal), pathology (animal or plant), pharmacology, molecular biology, genetics, cell biology, microbiology; neurobiology and neuropsychology (including animals as a model species for humans); animal behaviour; population biology genetics; functional genomics and related bioinformatics; biostatistics and modelling; animal, plant and microbial ecology; biogeography; biodiversity; phylogenetics; systematics and evolution; biophysics, chemical biology; and biochemistry.

Physical Sciences, Engineering and Mathematics (PEM).

Research related to the physical world and mathematics.

This includes, but not limited to: physics; physical chemistry; organic chemistry; analytical chemistry; inorganic chemistry; pure and applied mathematics; statistics; logic, theoretical and engineering aspects of computer and information sciences; complexity theory; operations research; nanotechnology; software and hardware engineering; applications and robotics; materials science; engineering (including bioengineering and other cross-disciplinary research activities); geology; geophysics; physical geography; oceanography; hydrology; meteorology; atmospheric science; earth sciences; astronomy; and astrophysics.

 

Assessment of proposals

Discipline-based panellists are asked to participate in a briefing video conference on one of the two dates indicated in the timetable. Each panel member will receive electronic copies of the applications for their panel. Panel members are asked to read, assess and grade each proposal based on the three selection criteria, taking into account the applicant-solicited referee reports.  Proposals are to be assessed by panel members exclusively on the information provided in the proposal and referee reports.

Panel members also need to identify proposals for which they have a conflict of interest, explaining the nature of the conflict (please refer to Conflicts of interest).

Each panel member is asked to start reading applications at different points through the order of the proposals, to avoid proposals from institutions or researchers first in the alphabet always being read first.

Each panel member will receive an electronic form on which to record their grades.  The spreadsheet should be completed and returned to the Rutherford Discovery Fellowship Secretariat by the due date.

The Vision Mātauranga and budget sections are included for panel members’ perusal but are not to be graded by the discipline panels.

It should be noted that these discipline-based panellists return their grades to the Rutherford Discovery Fellowships Secretariat and do not convene for a meeting at the end of their assessment.  The collated grades from the panellists will be used to create a ranked list of applications to be considered as the Long List.  This Long List is reviewed by the interview panel.

 

Assessment in relation to years of research experience

Panel members must consider applicants’ track record in relation to their years of research experience, which may differ from the number of years since PhD conferment. The years of research experience (R) is noted in the application header and on the first page under research area, and excludes periods of maternity/parental leave, medical leave or other relevant leave outlined in section 1e of the CV.

Consideration of referee reports

Applicant-solicited referees are used for the assessment of proposals in conjunction with the selection criteria. Where referees disagree, the panel members must use their own judgement in determining which referee reports to emphasise and what score to assign. These deliberations should be guided by considerations such as: the panel member's own level of expertise on the subject; the comments made by referees to explain their grades; the relative competencies of the referees; and, possible conflicts of interest. The applicants have been informed that referees should not be directly involved in the proposed programme of research or in your chain of line management.

Assessment Criteria

Rutherford Discovery Fellowship applications are graded on the following criteria:

 

Research Quality

Leadership Quality

  • Calibre of the applicant as a researcher
  • Calibre of the research programme the applicant intends following while a Rutherford Discovery Fellow
  • Calibre of the applicant as a research leader, which may include:
    • Vision for their field of work
    • Entrepreneurial activity
    • Team leadership
    • Knowledge transfer activity
  • Calibre of the applicant as a research leader, which may include:
    • Vision for their field of work
    • Entrepreneurial activity
    • Team leadership
    • Knowledge transfer activity

Table 3.         Assessment criteria

In the case of applicants of the same calibre, preference will be given to applicants who:

  • do not already have tenure or equivalent, or
  • who are living overseas and will use the Fellowship to return to New Zealand to continue their research careers.

 

Proposals are assessed on the information provided in the application, the accompanying forms and the applicant’s self-nominated referee reports.

For the current funding round the following weightings will be used:

  1. Calibre of the applicant as a researcher                                     60%
  2. Calibre of the applicant as a research leader                              20%
  3. Calibre of the research programme                                           20%

 

Considerations for assessing proposals to each criteria

Panel members may wish to consider the following as a guide for assessing the three criteria.

   1. Calibre of the applicant as a researcher

Consider if the applicant’s career is exceptional for a candidate in this discipline, at their career stage.

Exceptional may be determined by consideration of the merit of the applicant’s career to date and how the research compares with other New Zealand or international research in the same field.  If the applicant is at the start of his or her career the calibre must be assessed in relation to the years of research experience.  The curriculum vitae, supplied by the candidate in Section 4, should address the calibre of the applicant as a researcher.  Some expected sources of evidence include: awards/prizes; invitations to editorial boards or keynote addresses at conferences; publication record; patents awarded; and, referee reports.

   2. Assessment of the applicant’s leadership quality

Consider the leadership qualities you believe the applicant possesses, or the potential they have. 

Expected sources of evidence may include but are not limited to: team leadership roles; project management responsibilities; quality of stakeholder relationships; student numbers and completions; external grant funding as a named investigator; presence in relevant research communities; invitations to present keynote or plenary presentations; collaborator networks; significant contribution to achievement of commercialisation milestones; entrepreneurial activity; knowledge transfer activity; indications of peer-esteem; thought leadership (e.g., conceptual development of a research field internationally); leadership across Māori and other communities; direct policy facing or public engagement work; referee reports; and, performance in selection interview.

   3. Assessment of the proposed research programme

Consider the merit of the proposal and the potential of the research.

Merit may be determined by the applicant incorporating originality, insight and rigour.  Please consider the ability of the researcher to carry out the research.

Potential of the research may be assessed from the work outlined in Sections 8-10 of the proposal.  The research should significantly contribute to advances in theoretical understanding, develop new methodologies, contribute to new knowledge, or lead to advancement in a field by cross-fertilisation with ideas and results from another field.  Often the design and planning of a programme of research determines its success.  Good design and planning are determined by whether the overall proposal and its specific objectives have a clear focus, and the methods and experimental or sampling design are likely to produce high quality results.  Expected sources of evidence include: evidence of host support; and, referee reports.

 

Grades and distribution

There are six scores available; 1 (excellent) to 6 (room for improvement).  Each criterion should be assigned one of the six scores.  Each panel member should use the following target distribution for the proposals that they assess.

 

Score

1 (excellent)

2

3

4

5

6

(room for improvement)

% of proposals

10-20

15-25

20-30

15-25

10-20

0-10

Example (60 proposals)

6-12

9-15

12-18

9-15

6-12

0-6

Table 4.         Target distribution.

In the example above where 60 proposals are assessed, between 6 and 12 proposals should be assigned a score of 1, between 9 and 15 proposals should be assigned a score of 2, between 12 and 18 proposals should be assigned a score of 3, and so on.

Grading and recommendation to the interview panel

Once the overall scores from the panellists have been received, the Rutherford Discovery Fellowship Secretariat will produce an ordered list of applicants with the highest grades from each of the discipline-based panels.  These top ranking applications will form the Long List for consideration by the interview panel.  The number of applicants from each panel on the Long List will be determined by the number of proposals submitted.

Interview panel (Stage two)

The Chair and members of the interview panel will conduct the interviews.  This is a two-part process:

  • The interview panel assesses the Long List of applicants with the highest ranking grades from the discipline-based panels and will create a shortlist of applicants to be invited for interview. The applicants called to interview will be the highest ranked by the panel and does not need to reflect the number of proposals in a particular discipline.
  • The interview panel will conduct interviews and recommend the successful applicants for the Fellowships.

 

The Chair of the interview panel is responsible for the effective conduct of the assessment process.  This post will be filled by the President of the Society or their nominee.  Each panel member needs to ensure that the funding recommendations made are defensible by ensuring the framework for assessment is followed and identifying, and taking appropriate action, over conflicts of interest.

Each applicant will be asked a series of questions in an allocated 20 minute interview.  Overseas applicants will be interviewed using either teleconferencing or video-conferencing facilities.

The recommendations of the interview panel for successful applicants are ratified by the President of the Society.

 

Sensitive issues

Privacy

The Society has obligations under the Privacy Act to keep confidential certain information provided by individuals.  Moreover, the records of deliberations by panels are regarded as strictly confidential; as are the contents of applications.

  • Panel members should ensure the safe keeping of all applications and related confidential documents (e.g. applications, referee reports, scoring spreadsheets or summaries).
  • At the conclusion of the grading, panel meetings and the interviews, members should leave documentation with the Society staff and destroy any documentation remaining elsewhere.
  • Panel members should not enter into correspondence or discussion of the contents of the applications with referees, third parties, or the applicants.  Any necessary correspondence shall be addressed by the Secretariat of the Rutherford Discovery Fellowships.
  • The intellectual property of the ideas and hypotheses put forward in the applications should be treated in strict confidence.

Conflicts of interest

The Society takes the issue of conflicts of interest very seriously. A rigorous position is taken in order to maintain the credibility of the allocation process and to ensure that applications are subjected to fair and reasonable appraisal.

The Society wants to ensure that the panel members are active researchers with an excellent background in research.  As these researchers will invariably have connections with some applicants, conflicts of interest will arise. Where these occur for panel members, the following rules will apply. 

  • All conflicts of interest must be declared in writing to the Society.  Society staff will minute all conflicts of interest and actions taken.
  • Where a panel member is a family member or close friend of any applicant(s), that person will not assess the proposal or interview the candidate and take no part in the consideration of that proposal.  They will hear about the outcome of that proposal when official letters are sent to all applicants.
  • If a panel member has an interest in an application, such as collaborating with an applicant or an applicant’s group, or is conflicted with the applicant*then that member shall not assess the proposal or interview the candidate.
  • A panel member cannot be a referee for any applicant in the current funding round.
  • If the interview panel Chair has a conflict of interest then the duties of chairing the interview shall be passed to another panel member.

 

*A panel member is generally deemed to be conflicted if:

  • They work in the same department as the applicant(s). Where the department is large and contact between the panel member and applicant(s) is minimal, the Chair may deem there to be no conflict.
  • They work at the same CRI AND are in the same team as the applicant(s) (the level of conflict will depend on the size of the organisation).
  • They work at the same company as the applicant(s). The level of conflict will depend on the size of the company.
  • They have co-authored publications with the applicant(s) in the last 5 years
  • They have a low level of comfort assessing the application due to their relationship with the applicant(s).

When all conflicts of interest are taken into account, the interview panel Chair may decide that the remaining panellists’ expertise is not sufficient for assessment of a particular proposal.  In this case, an additional opinion from an external independent person may be sought.  Alternatively, a panellist who has previously left the room may be asked to return to answer technical questions only.

 

Additional Information

Feedback to applicants

In the Proposals On-Line web-based system, applicants are offered the option of receiving feedback in the form of quartiles for the three graded criteria at the conclusion of the funding round.  A general statement about the funding round will also be prepared and given to all applicants.

Applicants will also be notified:

  • if the applicant was successful in making the Long List.
  • if the applicant is considered ineligible to apply for a Rutherford Discovery Fellowship.

Disposal of applicant proposal matter

Referees are asked to return only the completed Referee report form.  Please destroy all proposal material once your report is completed.

Royal Society Staff

It is not the role of Society staff to make funding decisions.  Rather, their role is one of facilitation of and "guardianship" over the assessment process, ensuring that the process is credible and defensible.  To achieve this, staff will:

  • organise all logistical aspects of the process;
  • assist the discipline-based and interview panellists in determining realistic timetables for meetings;
  • provide a framework for assessment;
  • record funding decisions and collate generic feedback for applicants;
  • record any conflicts of interest and identify problem areas;
  • convey funding decisions to applicants and their host organisations - all discussions related to a decision should occur through Society staff; and,
  • negotiate contract details with host institutions.

Enquiries

If you require further information about the Rutherford Discovery Fellowships, please email us at rutherford.discovery@royalsociety.org.nz or phone 04 470 5764.



[1] This includes research in science, technologies, and humanities.