In response to the CAUT, NSERC’s Vice-President, Research Grants and Fellowships, Isabelle Blain explained how NSERC responded to the recommendations of the 2007 review of its Discovery Grants program. “ Two principles were fundamental:
- that the level of a grant should be commensurate with scientific or engineering merit,
- that within a given discipline group, proposals with similar scientific merit should have similar grant levels regardless of the applicant’s granting history with NSERC.
Sounds great! Here is a quick reality check on how these principles were upheld.
NSERC went ahead and devised a new evaluation system, the “binning” system, which was supposed to ensure that these principles are applied. For the benefit of those who told me that my last post was “over their head” (Leo, Melonie and others), here is a brief description of that system, which was first applied during the 2010 and 2011 competitions.
Applicants are rated under 3 criteria: EoR = Excellence of researcher, MoP = Merit of Proposal, HQP = Training of HQP.
There is a 6 point rating scale: Exceptional (E=6 points), Outstanding (O=5), Very Strong (VS=4), Strong (S=3), Moderate (M=2), Insufficient (I=1).
An ‘I’ or ‘M’ on any criterion means the grant is not funded. Once these applications have been removed, the 3 scores are added, giving totals between 18 and 9 points. Each possible total score is called a funding bin, labeled A (for E-E-E), B (for E-E-O), C (for E-O-O) etc… all the way to P (for I-I-I). Everyone in the same bin is supposed to get the same grant.
Once the Evaluation Group (EG) are done with the ratings, NSERC staff and the EG executive decide on a ‘bin to funding map’, which assigns $ to each bin. Here are the maps for the first two competitions for Mathematics.
One can easily see how the first principle was broken. Indeed, if you were rated O-V-V in 2010 (that is, Outstanding, Very Strong, Very Strong on the 3 criteria), you get 12K more than if you had received the same O-V-V rating in 2011. This translates to a 60K differential over the 5-year duration of the grant.
One can, of course, argue that different years lead to different budgets, hence the discrepancy. Fair enough, but then one realizes that actually both principles, one and two, are often broken within the same competition year. Indeed, take a look at the grant allocations for the following 4 applicants who were ranked in the same bin C (that is E-O-O) in the 2010 competition.
Bin Ranking A1 A2 A3 A4
C E-O-O 44K 48K 55K 60K
How could it be? Well, it turned out that there is another new rule at NSERC, which says that if you are in one of the top 3 bins (A, B, or C), then your old grant is protected if it is above the current allocation for your bin. So, Applicant A1 got 44K grant which was exactly the value of Bin C in 2010, while the 3 others got their grants from times past, which for various reasons were much larger.
So much for “proposals with similar scientific merit should have similar grant levels regardless of the applicant’s granting history with NSERC.”
The scientific community has been asked to accept an unstable and an unpredictable system in order to gain one that is supposedly more fair (equal grants for equal merit) and more dynamic (history and track record do not count). The two examples above illustrate that the gains we were told to expect from the new system may have been overrated.
Piece of Mind 
Oh. That pretty much explains what happened.
I like the Fundamental Principles.
Thanks Nassif, I suspect this will help a lot of people understand the binning system and its challenges.
The challenges are pretty easily explained, but I’m not sure whether the solutions are as simple. In the first case, budgets change each year, as you suggest. Perhaps all active grants from a given bin in previous years should automatically receive the increase – hence protecting against applying in the wrong year.
The second challenge is more difficult. Clearly, it’s designed to avoid the incongruity of cutting funding to researchers who continue to produce. But you can’t also bump everyone to the highest level. Of course, the easiest approach would be to go back to letting the peer-review committee allocate funds based on the myriad considerations of each individual case. Alas, the bureaucratization of research funding at the tricouncil suggests this isn’t likely to happen.
history and track record do not count
But they do. One bin is HQP.
Lose your grant, no HQP, can’t get back to S in that bin.
Minor correction — the binning system was first applied for the 2009 competition, we’ve just been through our 3rd round of this. It seems that the bins to $$ map was different in each of these years, which may be exacerbating an old problem — that some years are ‘luckier’ than others.
I for one am hoping this nonsense doesn’t spread from NSERC to CIHR…..
What a mess!
For the researcher: you can apply in one year an be ranked two bins higher than your colleague who applies the previous year, and still get less money (SVV 2010 vs OVV 2011).
For the EG: They know the bin-to-$$ map from the previous year. They think that they can expect a similar result, based on NSERC’s principles. Then they find that people get probably only 60% of that. They must be devastated.
For NSERC: what a loss of trust from the research community. Continuity from year to year? Forget it. Previous grant size does not matter? Okay, it does for the very top ones, it is “protected”.
For all Canadians: productivity per research $$ is a decreasing function of $$. Double the $$ for any researcher and you won’t get double the results, nor double the students trained. Put more $$ to few researchers at the top will not get the same number of papers nor the same number of HQP overall, see
http://www.universityaffairs.ca/the-wrong-way-to-fund-university-research.aspx
All very depressing indeed.
In the spirit of being constructive: would you have pointers to concrete suggestions from Canada’s scientific community on how to make this system better?
Speaking solely for myself, the frightening thing about the new system is its impact on my graduate students. I personally think it’s unethical to take on a graduate student without knowing one’s grant situation for the duration of his/her studies.
In my very specific instance I’d even be happy for a system which directed funding to my *students* directly: provided the student were doing well in coursework and got good progress reports, they should keep getting funded. Then the grant I’d apply for would be small, to support travel. Variability on this scale I can stomach. The swing in the current grant sizes directly impacts the most vulnerable- the students and postdocs.
Perhaps now that we’ve figured out basic discovery grants, you can explain the accelerator grants. The amount given out in accelerator grants dwarfs the little differences we’re discussing here. In my department, in CS, of the 12 people who received discovery grants, one person in bin 11 and another in bin 12 (just a few steps away from being denied grants) received accelerator grants (40K per year in addition to what they normally get). Is there an understandable rationale for who gets these?
Yes Valerie, this has also been a puzzling issue. I did allude to this discrepancy in a previous post
https://nghoussoub.com/2011/04/12/accountability-may-be-the-biggest-casualty-of-nserc%E2%80%99s-new-ways-at-discovery/
I actually think the binning is not a bad idea. By asking these questions rather than simply coming up with a number for each researcher, you move away from a situation where you place excessive emphasis on the previous number. (I sat on the GSC pre- and post-binning. In the pre-binning world a substantial cut was reserved for the situation where someone was doing inadequate work. This meant that if a person came in with a higher level than another person for historical but not scientific reasons, they went out with a higher level too). What is much more worrying for me is the way that the bin -> $$ map is essentially decided at a bureaucratic level so that most EG members have no input into the question of funding levels. The dramatic swings year-on-year between bin levels are a nightmare.
I have 2 students and 2 postdocs and just saw my grant cut from 20K to 15K. Had I received the same evaluations last year I would have received 24K (over 50% more). Very upsetting indeed…
Pingback: NSERC Discovery Grants: What do we know about the 2011 Math/Stats competition? | Piece of Mind
Pathetic moaning. Bad work = Lower grants. Stop whining, academics don’t know what work is.
I admire your certitude, Alan. Presumably you are competent to tell us both what work is, and what good work is as well. Please go ahead, we’d like to be enlightened. Maybe, as a start, provide a link to some of your research articles. Then we’ll have concrete examples, and can learn better, and moan less.
Pingback: NSERC should stick with linear thinking…and own principles | Piece of Mind
Pingback: New university presidents at Toronto and McGill - Capital Read, Inkless Wells - Macleans.ca
Pingback: New university presidents at Toronto and McGill - Macleans.ca