An NSERC evaluation group member “who would like to remain in the background” has forwarded the following memo to a colleague “who should remain anonymous” asking him if  he would forward it to the “Piece of Mind” blog. Is this how social media managed to topple the Tunisian and Egyptian bureaucracies dictatorships?
Here it is! Some of its points were discussed at length in the posts,  “NSERC Discovery Grants II: On intentions and consequence (Old vs. new)”, which has so far been viewed by 656 visitors, and “If the “binning” of Canada’s scientists is here to stay, then here is a way to fix it!”

Dear “Piece of Mind”,

Having served on the evaluation group during the transition phase to the new system of evaluation, I have some concerns that I want to share with the research community.

Selection of evaluation group members

During my time as a member of the evaluation group in my discipline, I had no idea how members were selected, although we were asked to suggest candidates during the review meeting each February. I would like to see a more transparent selection process, to ensure that well qualified members are consistently recruited.  These should be members in their respective fields who are top or near the top researchers, and very knowledgeable in most areas of the field. One suggestion is to have a call for nominations to the respective academic communities, and to have an advisory or liaison committee, separate from the evaluation group, to make recommendations.

The binning system

A major difficulty with the new binning system, which has been noted by others, is the assessment of HQP (highly qualified personnel).  The rating under HQP comprises 1/3 of the final rating of the researcher.  There is no point in rating an Early Career Researcher for HQP training. Currently the default rating is Moderate; if an applicant presents a detailed plan for training, the applicant may get a Strong rating. This simply penalizes an applicant who receives his/her PhD in the same year of the application who usually does not have any experience with HQP training.

NSERC now imposes a minimal rating for each category in order for an Established Researcher to be funded: for HQP this minimum rating is Strong. We have already seen the impact of this policy as the number of reported HQP has increased substantially. Although there is no hard evidence, I sense that many numbers are inflated, especially at the masters level. I think that NSERC needs to provide clearer guidelines on who can be listed: for example if an applicant claims he/she is a supervisor or a co-supervisor, the appointment should be official and with an emphasis on PhD and Postdoctoral training.  More importantly, the requirement of the minimal rating on HQP does not serve the goal of NSERC to advance research in the Natural Sciences and Engineering. Some excellent researchers may prefer to work alone or limit the number of graduate students they want to supervise for various reasons. It is reasonable to reduce funding but not to eliminate funding. I would suggest that the requirement of a minimum rating on HQP for success in funding be removed. A better way could be to simply add the scores to determine to the order of the bins.

Application evaluation

Currently, the evaluation panel for an application is not allowed to refer to any information that is not in the application, such as checking the impact of applicant’s publications on Google Scholar, which I kind of understand from a policy viewpoint. However this information asymmetry limits the ability of the panel to have a complete assessment of the application. There needs to be some flexibility that at the same time maintains fairness and consistency across the applications.