In response to the CAUT, NSERC’s Vice-President, Research Grants and Fellowships, Isabelle Blain explained how NSERC responded to the recommendations of the 2007 review of its Discovery Grants program. “ Two principles were fundamental:
- that the level of a grant should be commensurate with scientific or engineering merit,
- that within a given discipline group, proposals with similar scientific merit should have similar grant levels regardless of the applicant’s granting history with NSERC.
Sounds great! Here is a quick reality check on how these principles were upheld.
NSERC went ahead and devised a new evaluation system, the “binning” system, which was supposed to ensure that these principles are applied. For the benefit of those who told me that my last post was “over their head” (Leo, Melonie and others), here is a brief description of that system, which was first applied during the 2010 and 2011 competitions.
Applicants are rated under 3 criteria: EoR = Excellence of researcher, MoP = Merit of Proposal, HQP = Training of HQP.
There is a 6 point rating scale: Exceptional (E=6 points), Outstanding (O=5), Very Strong (VS=4), Strong (S=3), Moderate (M=2), Insufficient (I=1).
An ‘I’ or ‘M’ on any criterion means the grant is not funded. Once these applications have been removed, the 3 scores are added, giving totals between 18 and 9 points. Each possible total score is called a funding bin, labeled A (for E-E-E), B (for E-E-O), C (for E-O-O) etc… all the way to P (for I-I-I). Everyone in the same bin is supposed to get the same grant.
Once the Evaluation Group (EG) are done with the ratings, NSERC staff and the EG executive decide on a ‘bin to funding map’, which assigns $ to each bin. Here are the maps for the first two competitions for Mathematics.
One can easily see how the first principle was broken. Indeed, if you were rated O-V-V in 2010 (that is, Outstanding, Very Strong, Very Strong on the 3 criteria), you get 12K more than if you had received the same O-V-V rating in 2011. This translates to a 60K differential over the 5-year duration of the grant.
One can, of course, argue that different years lead to different budgets, hence the discrepancy. Fair enough, but then one realizes that actually both principles, one and two, are often broken within the same competition year. Indeed, take a look at the grant allocations for the following 4 applicants who were ranked in the same bin C (that is E-O-O) in the 2010 competition.
Bin Ranking A1 A2 A3 A4
C E-O-O 44K 48K 55K 60K
How could it be? Well, it turned out that there is another new rule at NSERC, which says that if you are in one of the top 3 bins (A, B, or C), then your old grant is protected if it is above the current allocation for your bin. So, Applicant A1 got 44K grant which was exactly the value of Bin C in 2010, while the 3 others got their grants from times past, which for various reasons were much larger.
So much for “proposals with similar scientific merit should have similar grant levels regardless of the applicant’s granting history with NSERC.”
The scientific community has been asked to accept an unstable and an unpredictable system in order to gain one that is supposedly more fair (equal grants for equal merit) and more dynamic (history and track record do not count). The two examples above illustrate that the gains we were told to expect from the new system may have been overrated.