A colleague –with experience on a selection committee– explains below why the new binning system may be announcing “very good days ahead for bad science”.

He makes 3 important points in his message below:

1.   The new binning system is so rigid and inflexible that it cannot detect research on tough and deep problems.

2.   The new system can now be programmed to run by a computer with a result almost identical to a committee’s work –up to an error not exceeding one bin! (our new unit system!). It will then be more efficient to replace the selection process by an automated one, eliminate all NSERC staff and return the savings to  research support.

3.   The danger of the “flat earth policy” for HQP explained below, which is now discriminating against the smaller universities but will eventually creep up to affect higher tier universities. It is as fair as a flat rate of revenue taxation and flat not as percentage but rather  as absolute value !

His analysis is subtle and worth reading carefully.

Dear all,

I want to mention here a couple of points that are relevant I think for the next competition as well as the ones after that.

There is an unhealthy lack of flexibility in the new system, which got even worse in this year’s competition.  There are two consequences of this lack of flexibility that I would like to point out

– Firstly, research on tough and deep problems ends up being supported much less than it should. The reason is that quantitative indicators are often less strong for research on hard topics and making the case that quality beats quantity in a non-specialist committee is not always easy.

– Secondly, things are actually so inflexible that for 90% of the files a computer program would arrive at the same conclusions as the committee up to a final difference of at most one bin. In only about   3% of files would this program (that would integrate only quantitative factors + some ranking of journals and would possibly have a slightly different formula for various subcommittees) miss the result by two bins (in view of this it would be much more efficient  to replace the selection process by an automated one, eliminate all the NSERC staff and return the savings to  research support – my estimate is that the net result for the scientific community would be positive).

My point here is that the NSERC is taking all the responsibility of real decision-making from the hands of the researchers and the whole process is looking more and more like a loss of energy and time and has not much content. It also makes “beating the system” very easy with the consequence that there are very good days ahead for bad science.

My second point concerns the very concrete way in which this lack of flexibility is implemented in what concerns HQP.  This year we are supposed to apply what is called, rather inauspiciously, “the flat earth policy”.

Besides contradicting evidence, what it means is that “one PhD (or Master or undergrad) of researcher A from University M” should be counted exactly the same way independently of the level of funding of A or of the quality or even the existence of a graduate program at University M (there is an allowance for the “impact” of the student but let’s imagine this being constant for this discussion).  Thus, a researcher with a grant of 10k at a place with only a MA program and who is ranked Very Strong in researcher quality and  has two MA’s each  costing 5k/year will  probably not get a grant even if,  in fact,  it is simply not possible  to expect this researcher to do more with the  resources he/she has and indeed, five  years ago when this researcher’s grant was  approved the idea was precisely to  support training on this scale.

Obviously, once A does not get a grant this year it will be even harder to get one next year despite the fact that VS as researcher ranking means that he/she does valuable work. It’s obvious that in a few years this will essentially wipe out research training at many small institutions. But people who are at bigger institutions should also worry because, after the system runs for about four years most of the people at the smaller places will not be applying anymore so that the success rates will go up (even if the number of grants will go down) and the time will come for another tightening of the screws, this times affecting the next level institutions up.  I hope you will see the danger in this, I am quite alarmed by it and it is already at work this  year. It also affects not only people at small institutions but many other examples, women in particular, as in the last post.

It might be that, at first sight, counting the “same student” the same way all over the board might look fair. But it is simply ridiculous that we are supposed not to take into account the “reasonable expectation” for A but rather the absolute outcome. This is as fair as a flat rate of revenue taxation and flat not as percentage but rather  as absolute value !