Don Fraser, Professor Emeritus at the University of Toronto has taught and trained several generations of Canadian statisticians, and he is still at it. He also continues to contribute high-level research. He was –as recently as last summer– an invited speaker at the Joint Statistical Meetings in Vancouver, attended by more than 4000 statisticians. Don has been stonewalled by NSERC’s bureaucracy and he has decided to speak out. Make sure not to miss the epilogue!
Accuracy at NSERC
By Don Fraser, FRSC
Many applications for research support pass through NSERC each year, some new, some continuing, some go up, and some come down. Applicants who go up usually don’t talk about going up, and those who come down usually stay quiet. But how could the Accuracy of this process be checked?
A real question!
The NSERC evaluative process was recently revised, and in a public presentation that I attended NSERC staff presented a graph of new grant amounts vs. old grant amounts, and gave positive interpretation to the increase in variability. A lone statistician pointed out that it is easy to increase variability by inserting randomness, but Accuracy is another issue.
Statistics Canada wanted Accuracy this past summer, the Government of Canada over-ruled them, and the Chief Statistician resigned in protest! Then it emerged that some people do care about Accuracy; and the world wide support for Statistics Canada was impressive! But the Government of Canada just saw their own political agenda — seeking Accuracy poses big challenges!
What about the NSERC process and Accuracy?
As we know, members of the relevant scientific community participate in evaluation and are guided through the process by NSERC staff. Apparently committee members are required to affirm competence in areas judged, but who would admit otherwise? Referee reports?? Rumours circulate that committee members view these reports as not very useful! What was that….not very useful? How are outside experts to be weighed against those enmeshed in the evaluation process?
How might the process be evaluated?
Statisticians have tried-and-true methods for this: run some duplicates through the system and see how the evaluations go! It might be hard to find duplicates: different applications with almost identical ingredients. But they do occur!
Does NSERC ever make mistakes?
In 2009 Nancy Reid went from $48k to $25k, and appealed. In 2010 Nancy Reid returned to … $48k!
Mathematically or Logically: One, or other, or both represent evaluative error!
Does NSERC still make mistakes?
In 2010 Nancy Reid and Don Fraser were in the same evaluation group: They work in common areas, and they work in different areas. How did their NSERC applications look?
(1) Research: Shared prime publications in leading journals. Separate prime publications, individual and with other authors, in leading journals. Joint and individual research. The numbers for the two applicants were almost identical!
(2) Proposal: Shared research endeavours. And individual research endeavors. Individual for Reid: extension of likelihood into wider application via composite likelihood: to get Accuracy with available tools. Individual for Fraser: Statistics has two methodologies giving different results (Really? Yes!) – Compare, evaluate and calibrate these methodologies so that stated results are Accurate! Again almost identical numbers!
(3) HQP: Again very similar numbers! See fraser10_100; fraser10_101; reid10_100; reid10_101.
The proposals are essentially equivalent! …. But: Reid was awarded $48k; Fraser went from $40k to $20k. Essentially equivalent proposals with un-equivalent results!
They sure got that randomness in! Or something masquerading as such. In ”bin” talk, the difference for ’identical’ applications was 5 bin units = $28,000! What is going on? Can NSERC provide an explanation? Not yet and the trail is getting devious!
Does NSERC have an appeal process?
Well yes! With a few limitations, like you’re not supposed to mention $s but maybe bins and it’s supposed to deal with errors in their process.
Does the Appeal process work?
Fraser appealed on May 31, 2010.
And Fraser received multiple requests to submit Form 180 for reapplying for 2011. And then made multiple requests to get the advisor’s report and decision, as 2011 re-application time was fast approaching.
October 12: Appeal declined. The advisor’s report arrived, with not very much content, but a surprise:
“I have reason to believe that once the committee factored in consideration for Delays in Research Productivity, that they deemed application 9436-2010 to be of superior ranking to that of 19361- 2010. As I do not have access to application 9436-2010 or to the committee’s deliberations, I cannot be certain however.”
Hmmm – delays in research productivity tends to be a code word for parental leave: from NSERC’s guidelines
NSERC Evaluation Committee members are instructed to incorporate consideration for delays in research productivity into their evaluation of an application. Section 6.8.3.1, quoted below, provides guidelines for evaluating delays in research. “NSERC recognizes that research productivity and contributions to the training of HQP may be disrupted during periods of pregnancy or early child care (parental leave), whether or not a formal leave of absence was taken, or as a result of other personal circumstances. Administrative leave, illness, disability and other situations may also result in delays in research”.
Doesn’t apply – how does Fraser know? – He and Nancy are married, and their children are young adults!
After contacting NSERC a new version of the advisor’s report arrived; REDACTED with NSERC citing privacy concerns. New version has no mention of delays in research productivity. “I can only assume that the committee judged the quality and impact of application 9436-2010 to be higher than that of the appellant in the three evaluation criteria.” Duh!
Does an emeritus professor get fair treatment?
NSERC says they get the same treatment but the community seems to think otherwise.
Can you contest an Appeal decision? Can you ask for explanations?
A response from NSERC December 10 included an offer to “address the issues that may remain unclear” was replied to with two letters outlining these issues.
No response from NSERC to date. Stone-walled, by an entrenched bureaucracy!
Piece of Mind 
I have been a research assistant for Prof. Fraser for more than 2 years now and his brilliant teaching and guidance have been instrumental in my growth as a statistician. In fact, almost all of my colleagues have been benefitted from Prof. Fraser’s wealth of knowledge and teaching. NSERC’s decision is indeed very unfair and disappointing! Hope they’ll reconsider their decision.
I just wanted to let you know how much I enjoy and appreciate reading your blog “Piece of Mind”. I think it is especially honourable that you — as someone who will never have trouble getting solid funding from NSERC — have taken it upon yourself to start the debate about the new evaluation process at NSERC. The story of Don Fraser is very telling: it is shocking (but I think typical!) that two people with essentially equal applications should receive such different funding amounts.
The situation is especially scary for people who are at smaller institutions and for whom the grant sizes are typically smaller than those of the researchers at larger universities. In this situation, the different outcomes for two applicants with essentially the same applications can be extremely devastating and unjust: one receives funding, the other nothing (except maybe a kick in the pants). I believe that the one who receives nothing is very unlikely to ever receive a grant again due to his/her lack of funding to support the training of HQP and the increased teaching load assigned to non-grant holders at most institutions. It is, I believe, very unlikely that this individual will be able to ever achieve a high level of research productivity again. To my mind, the whole process has been turned largely into a crapshoot with especially high stakes, I think, for those competing for the smaller grants (for whom the loss of a grant can mean the end of a research career).
The situation is also extremely dire for new researchers. I have to wonder how many of Canada’s best researchers today would not have received funding from their first grant application if they had submitted it under the current conditions (and would consequently have been much less likely to blossom into the researchers they are today).
Anyway, thank you so much again for fighting this battle on behalf of the Canadian math community!
Thank you very much for the kind words. Your concerns are shared by many, and the worst thing that can happen is to remain silent.
Tomas, I couldn’t agree more. It is much more serious when these anomalies mean the difference between being able to do research and not. I think it is important for people who can afford to speak out, to do so. In this way it perhaps helps, at least indirectly, many others who don’t have the luxury to publicize their case.
Pingback: The Kafkaesque grip of bureaucrats on Canada’s peer review and granting process | Piece of Mind
Professor Fraser makes some very good points. Above all else, the very agency responsible for adjudicating grants in science appears to have an aversion to the scientific method at home. In particular, neither is there any evidence that the changes in the system were driven by evidence-based reasoning and a clearly defined problem or solution, nor is there any wish to provide the kind of data that would permit monitoring the consequences of these changes for disciplines like statistics as a whole. Near replications (like Fraser-Reid) may be hard to come by but longitudinal information is not. Data on grants before/after the changes, especially in specific cross-disciplinary areas such as computational statistics or biostatistics, IS relevant to measuring the effect of the adoption of the conference model. One of the best measures of noise in the system is that evidenced in this post. If an individual with more than a half century of extraordinarily consistent research and HQP to the highest global standard possible can see a drop of 50% or more in grant level, then clearly the noise is 50% or more.
Pingback: James Colliander's Blog » NSERC Peer Review System is Broken for Mathematics
Pingback: Accountability may be the biggest casualty of NSERC’s new ways at Discovery | Piece of Mind
Pingback: James Colliander's Blog » Anticipating the 2011 Competition Statistics for NSERC Discovery Grants Program
Pingback: Don Fraser, Officer of the Order of Canada | Piece of Mind