Don Fraser, Professor Emeritus at the University of Toronto has taught and trained several generations of Canadian statisticians, and he is still at it. He also continues to contribute high-level research. He was –as recently as last summer– an invited speaker at the Joint Statistical Meetings in Vancouver, attended by more than 4000 statisticians. Don has been stonewalled by NSERC’s bureaucracy and he has decided to speak out. Make sure not to miss the epilogue!
Accuracy at NSERC
By Don Fraser, FRSC
Many applications for research support pass through NSERC each year, some new, some continuing, some go up, and some come down. Applicants who go up usually don’t talk about going up, and those who come down usually stay quiet. But how could the Accuracy of this process be checked?
A real question!
The NSERC evaluative process was recently revised, and in a public presentation that I attended NSERC staff presented a graph of new grant amounts vs. old grant amounts, and gave positive interpretation to the increase in variability. A lone statistician pointed out that it is easy to increase variability by inserting randomness, but Accuracy is another issue.
Statistics Canada wanted Accuracy this past summer, the Government of Canada over-ruled them, and the Chief Statistician resigned in protest! Then it emerged that some people do care about Accuracy; and the world wide support for Statistics Canada was impressive! But the Government of Canada just saw their own political agenda — seeking Accuracy poses big challenges!
What about the NSERC process and Accuracy?
As we know, members of the relevant scientific community participate in evaluation and are guided through the process by NSERC staff. Apparently committee members are required to affirm competence in areas judged, but who would admit otherwise? Referee reports?? Rumours circulate that committee members view these reports as not very useful! What was that….not very useful? How are outside experts to be weighed against those enmeshed in the evaluation process?
How might the process be evaluated?
Statisticians have tried-and-true methods for this: run some duplicates through the system and see how the evaluations go! It might be hard to find duplicates: different applications with almost identical ingredients. But they do occur!
Does NSERC ever make mistakes?
In 2009 Nancy Reid went from $48k to $25k, and appealed. In 2010 Nancy Reid returned to … $48k!
Mathematically or Logically: One, or other, or both represent evaluative error!
Does NSERC still make mistakes?
In 2010 Nancy Reid and Don Fraser were in the same evaluation group: They work in common areas, and they work in different areas. How did their NSERC applications look?
(1) Research: Shared prime publications in leading journals. Separate prime publications, individual and with other authors, in leading journals. Joint and individual research. The numbers for the two applicants were almost identical!
(2) Proposal: Shared research endeavours. And individual research endeavors. Individual for Reid: extension of likelihood into wider application via composite likelihood: to get Accuracy with available tools. Individual for Fraser: Statistics has two methodologies giving different results (Really? Yes!) – Compare, evaluate and calibrate these methodologies so that stated results are Accurate! Again almost identical numbers!
The proposals are essentially equivalent! …. But: Reid was awarded $48k; Fraser went from $40k to $20k. Essentially equivalent proposals with un-equivalent results!
They sure got that randomness in! Or something masquerading as such. In ”bin” talk, the difference for ’identical’ applications was 5 bin units = $28,000! What is going on? Can NSERC provide an explanation? Not yet and the trail is getting devious!
Does NSERC have an appeal process?
Well yes! With a few limitations, like you’re not supposed to mention $s but maybe bins and it’s supposed to deal with errors in their process.
Does the Appeal process work?
Fraser appealed on May 31, 2010.
And Fraser received multiple requests to submit Form 180 for reapplying for 2011. And then made multiple requests to get the advisor’s report and decision, as 2011 re-application time was fast approaching.
October 12: Appeal declined. The advisor’s report arrived, with not very much content, but a surprise:
“I have reason to believe that once the committee factored in consideration for Delays in Research Productivity, that they deemed application 9436-2010 to be of superior ranking to that of 19361- 2010. As I do not have access to application 9436-2010 or to the committee’s deliberations, I cannot be certain however.”
Hmmm – delays in research productivity tends to be a code word for parental leave: from NSERC’s guidelines
NSERC Evaluation Committee members are instructed to incorporate consideration for delays in research productivity into their evaluation of an application. Section 18.104.22.168, quoted below, provides guidelines for evaluating delays in research. “NSERC recognizes that research productivity and contributions to the training of HQP may be disrupted during periods of pregnancy or early child care (parental leave), whether or not a formal leave of absence was taken, or as a result of other personal circumstances. Administrative leave, illness, disability and other situations may also result in delays in research”.
Doesn’t apply – how does Fraser know? – He and Nancy are married, and their children are young adults!
After contacting NSERC a new version of the advisor’s report arrived; REDACTED with NSERC citing privacy concerns. New version has no mention of delays in research productivity. “I can only assume that the committee judged the quality and impact of application 9436-2010 to be higher than that of the appellant in the three evaluation criteria.” Duh!
Does an emeritus professor get fair treatment?
NSERC says they get the same treatment but the community seems to think otherwise.
Can you contest an Appeal decision? Can you ask for explanations?
A response from NSERC December 10 included an offer to “address the issues that may remain unclear” was replied to with two letters outlining these issues.
No response from NSERC to date. Stone-walled, by an entrenched bureaucracy!