Psst! Pass it on! Concerns about NSERC’s ways

An NSERC evaluation group member “who would like to remain in the background” has forwarded the following memo to a colleague “who should remain anonymous” asking him if  he would forward it to the “Piece of Mind” blog. Is this how social media managed to topple the Tunisian and Egyptian bureaucracies dictatorships?
Here it is! Some of its points were discussed at length in the posts,  “NSERC Discovery Grants II: On intentions and consequence (Old vs. new)”, which has so far been viewed by 656 visitors, and “If the “binning” of Canada’s scientists is here to stay, then here is a way to fix it!”

Dear “Piece of Mind”,

Having served on the evaluation group during the transition phase to the new system of evaluation, I have some concerns that I want to share with the research community.

Selection of evaluation group members

During my time as a member of the evaluation group in my discipline, I had no idea how members were selected, although we were asked to suggest candidates during the review meeting each February. I would like to see a more transparent selection process, to ensure that well qualified members are consistently recruited.  These should be members in their respective fields who are top or near the top researchers, and very knowledgeable in most areas of the field. One suggestion is to have a call for nominations to the respective academic communities, and to have an advisory or liaison committee, separate from the evaluation group, to make recommendations.

The binning system

A major difficulty with the new binning system, which has been noted by others, is the assessment of HQP (highly qualified personnel).  The rating under HQP comprises 1/3 of the final rating of the researcher.  There is no point in rating an Early Career Researcher for HQP training. Currently the default rating is Moderate; if an applicant presents a detailed plan for training, the applicant may get a Strong rating. This simply penalizes an applicant who receives his/her PhD in the same year of the application who usually does not have any experience with HQP training.

NSERC now imposes a minimal rating for each category in order for an Established Researcher to be funded: for HQP this minimum rating is Strong. We have already seen the impact of this policy as the number of reported HQP has increased substantially. Although there is no hard evidence, I sense that many numbers are inflated, especially at the masters level. I think that NSERC needs to provide clearer guidelines on who can be listed: for example if an applicant claims he/she is a supervisor or a co-supervisor, the appointment should be official and with an emphasis on PhD and Postdoctoral training.  More importantly, the requirement of the minimal rating on HQP does not serve the goal of NSERC to advance research in the Natural Sciences and Engineering. Some excellent researchers may prefer to work alone or limit the number of graduate students they want to supervise for various reasons. It is reasonable to reduce funding but not to eliminate funding. I would suggest that the requirement of a minimum rating on HQP for success in funding be removed. A better way could be to simply add the scores to determine to the order of the bins.

Application evaluation

Currently, the evaluation panel for an application is not allowed to refer to any information that is not in the application, such as checking the impact of applicant’s publications on Google Scholar, which I kind of understand from a policy viewpoint. However this information asymmetry limits the ability of the panel to have a complete assessment of the application. There needs to be some flexibility that at the same time maintains fairness and consistency across the applications.

This entry was posted in R&D Policy. Bookmark the permalink.

2 Responses to Psst! Pass it on! Concerns about NSERC’s ways

  1. Florin Diacu says:

    Henri Poincaré never had a student. He always worked alone. Of course, he lived in a different time, when joint papers were like joint fiction books are today (they exist, in fact, but are rare). Still, Poincaré’s example raises questions, which NSERC should know about.

  2. For established researchers, NSERC requires a rating of “Strong” or better for Excellence of Researcher. This requirement does not apply to other criteria or to Early Career Researchers . Of the funded applicants in competition 2010, there were more than 340 across all Evaluation Groups, including 26 in Mathematics and Statistics, who received a rating of “Moderate” for HQP or Merit of Proposal. NSERC considers a rating of “Insufficient” on any criterion as non-fundable.

    Section 6.8.1.3. of the Peer Review Manual (www.nserc-crsng.gc.ca/_doc/Reviewers-Examinateurs/Chapter6_eng.pdf) explains how the applicant’s contributions to the Training of Highly Qualified Personnel are assessed.

    The principle of fairness dictates that all applicants must be reviewed on the same basis. This includes providing all applicants with an equal opportunity to make their case and directing reviewers to use only the information that is contained in the application package. All applicants receive the same page limits, are invited to provide the same number of samples of contribution (reprints, pre-prints, etc) and have their productivity assessed based on the previous six years.

    NSERC invites all members of Canada’s research community, individually or through scientific societies or liaison committees, to nominate potential members of peer review committees. Suggestions can be made through NSERC’s Web site at: http://www.nserc-crsng.gc.ca/NSERC-CRSNG/Committees-Comites/suggestion-suggestion_eng.asp.

    NSERC’s Guidelines Governing Membership of Selection Committees can be found at: http://www.nserc-crsng.gc.ca/NSERC-CRSNG/Policies-Politiques/committeemembers-membrescomite_eng.asp.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s