Last year, the Canadian Institutes of Health Research (CIHR), which is the primary federal funding agency of health research in Canada, embarked on a bold and wide-ranging series of reforms that change virtually every aspect of how health research funding is applied for, evaluated and distributed. On July 15, 2015, the results of the first major competition under the new system were released, as were the first casualties. Judging from the social media firestorm that followed, we felt that Canada’s research community may want to know some more. So I asked Jim Woodgett, Director of Research at the Lunenfeld-Tanenbaum Research Institute, to help us out. He kindly obliged by writing the following very informative guest blog. Jim was one of the initial F-scheme awardees, so this is far from being a rant of a bitter applicant. The simple fact is that nearly 300 accomplished Canadian researchers (those who submitted to Stage 2 but were not funded) cannot expect to receive funding for the next 12 months, at a minimum. As any responsible research leader in this country should, he worries about both the short and long term impact of the funding reforms on Canada’s health research community.
A guest blog by Jim Woodgett, Director of Research, Lunenfeld-Tanenbaum Research Institute
This essay describes the background, the changes, the issues and some possible ideas for improvement. It is based on publically available information from CIHR as well as personal experience and correspondence with many applicants throughout the lengthy process.
Burning Platforms: Globally, public funding of science has not recovered well from the recession of 2008/9 but was showing signs of sickness prior to that setback. In most developed countries, both government and non-governmental funding agencies have flat-lined scientific research budgets since the halcyon days of the late 1990s/early 2000s. That period saw many organizations, like the National Institute of Health in the US double their science budgets. In Canada, CIHR emerged from the ashes of the Medical Research Council and saw its budget increase 3-fold between 1999 and 2006. An expanded cohort of highly trained and accomplished graduate students and postdoctoral fellows developed out of the increased activities of that period, fueling growth of laboratories around the world. New research facilities were built or expanded.
After 2006, increases in agency budgets began to slow and fall behind inflation. In some countries, including Canada, research funding was increasingly diverted to more applied aspects of research, at the expense of basic/discovery science. As a consequence of these pressures, success rates of grant competitions steadily fell which promoted increases in the numbers of grant applications – causing yet further downward pressure on success rates. Over time, scientists significantly increased the amount of time they spent preparing applications and reviewing them, and some labs began to be shuttered.
End of the line, all change: By 2012, the status quo had become untenable and funding agencies scrambled to adapt to the new economic realities and depressed scientific climate. The main three Federal granting agencies in Canada, the so-called tri-councils (CIHR, NSERC and SSHRC) have each made significant recent process changes but CIHR has been the most ambitious. A program of comprehensive reforms was announced by CIHR in early 2012 and included radical changes to the types of funding programs and the manner in which they were adjudicated. Most notably, CIHR proposed to shift from conventional face-to-face reviews, which are clustered by expertise, to a system of virtual review.
Virtual Review: The most worrying aspects of the proposed changes revolved around the switch from Face-to-Face to virtual peer review. The intent by CIHR was to improve peer review consistency by having 5 reviews per application. In the prior system, a grant application would be read and evaluated by two primary reviewers and a secondary reader (each reviewer would be assigned ~8 applications). Panels of 8-15 reviewers with interests focused in specific areas of research (e.g. cell physiology, cardiology, neuroscience, etc.) would meet in an Ottawa hotel for 2-3 days to discuss 20-70 applications. If the preliminary scores were high enough, there would be discussion of strengths and weakness, a consensus score reached and the other panel members would record their own score. The averaged score would then be calculated by the agency and used to rank within the panel. In the most recent competition (transitional Open Operating Grant Program; tOOGP), the success rate was 14.2% so for a panel of 50 applications, ~7 would be approved for funding. The difference between a funded and an unfunded grant could be 0.01 – especially as scored tend to cluster around the anticipated funding line.
Virtual review increases the number of reviewers assigned to each grant and also does away with the panel structure. This was desired by CIHR as panels can develop their own “cliques” and many panel mandates (the range of expertise they were intended to cover) did not fit well with applications, especially those that crossed disciplines. To overcome these issues a global ranking system was developed. Reviewers are assigned a pile of grants based on matching their expertise with keywords in the applications. Each reviewer is assigned 15 or so grants and asked to rank them within their own pile. This forms a series of 5 fractions that is assigned to each grant. If Reviewer 1 has 16 applications and ranks Application A 2nd in her pile, the fraction is 2/16. The fractions are arithmetically combined, and because there are 5, the variance between the scores can be calculated. This allows all applications to be ranked relative to one another, irrespective of research area. Reviewers do not physically meet but are expected to upload their ranking data on-line and to participate in asynchronous chats to try to reach consensus.
The agency also proposed to replace traditional open operating grants with two largely independent funding streams termed Foundation (F-scheme) and Project (P-scheme). The former is a three-stage, applicant-focussed and designed to support a program of research rather than specific projects. The idea was to increase flexibility of research and was also expected to reduce application pressure as awardees would be funded for 5-7 years – and be excluded from applying for the separate P-scheme competition (an investigator can only hold one F-scheme award). P-scheme awards were designed for smaller budgets and for 1-5 years. Multiple P-scheme awards can be held by an investigator or by a team.
Pushback: When announced, these changes were met with a fair degree of scepticism and concern by many researchers, which were expressed at town halls and through letter campaigns to CIHR. Others welcomed the changes. CIHR asked for feedback and formally responded to the collective concerns in the Fall of 2012, but the fundamental principles of the reforms were unchanged from the initial proposal.
A Reforms Advisory Working Group was set up by CIHR with comprised ten researchers across all four pillars. I was a member and the group provided input over the next 18 months. As more details about the processes, and virtual review in particular, became apparent, the Advisory Group wrote to CIHR Science Council in June 2013 to communicate their joint concerns. These included:
- Undue reliance on modeling scenarios that were based on untested assumptions as well as a concern for early and mid-career investigators.
- Doubts about the execution of virtual review while abandoning face-to-face reviews.
- Concerns over recruitment of a sufficiently qualified College of Reviewers.
- The conformist effect of structured applications and reviews – required to increase efficiency of the review process given that many more reviews are required over the prior system.
- Poor definition of the 3rd stage of the F-scheme review and definition of the “Grey Zone”.
- The reduced number of grant application opportunities during the transition from old to new systems – including the cancellation of two competitions in order to provide enough funding for the first F-scheme.
- The increased risk associated with changing so many aspects of the funding process at once – without any additional funding to smoothen the switch.
Following a rapid and rather dismissive reply, the Working Group was soon disbanded and the first F-scheme competition was launched in June of 2014. While hindsight is 20:20, the problems associated with this first F-scheme was almost entirely predicted by the Working Group.
The (first) results are in as are the first casualties: At the outset, CIHR predicted between 120 and 250 F-scheme awards would be made in the first “pilot” of this scheme and eligibility was managed such that the competition was only open to those who held CIHR funding but had an existing grant expiring in FY2015, to those who had never held CIHR funding and to those who were within the 1st five years of holding funding at the time of registration (defined as Early Career Investigators, ECIs). There were 1375 applications to the initial phase of this F-scheme, known as Stage 1. This stage largely comprised a curriculum vitae (in the form of a bespoke “Common CV”) as the adjudication criteria were evidence of leadership, significance of prior contributions and productivity. On December 10, 2014, the results of Stage 1 were released.
Along with a consolidated ranking (which combined the rankings of the 5 reviewers), the standard deviations of these ranking were reported. Of the 1366 Stage 1 applicants that successfully submitted, 467 (34%) were approved to go on to Stage 2. There were 559 ECIs who applied to Stage 1, 87 (15%) were invited to Stage 2. Notably, the degree of variance among the rankings appeared very high (average 20%, maximum 45%), as anticipated by many, including the Working Group.
The culling of ECIs was inevitable given the emphasis on qualities that young investigators simply would not have had time to demonstrate, even though reviewers were instructed to take career stage into account. ECIs were directly compared with mid- and later career scientists. Indeed, mid-career applicants also fared poorly. This may relate to the managed intake given that this required existing grantees to have at least one expiring grant and so was biased towards investigators with multiple grants who tend to be more senior. In other words, this first competition was likely atypically competitive compared to later competitions. A letter from several Stage 1 reviewers emphasized the problems encountered in this phase and made several constructive suggestions.
Stage 2 and structural damage: Of 467 applicants invited to apply for Stage 2, 455 actually did. The drop-off was likely related to uncertainty associated with the pilot F-scheme process as this was the decision point for entering the last open operating grant competition to be conducted using face to face panels (this transitional OOGP was run at the same time as the F-scheme).
Those that decided to submit to Stage 2 were not eligible to submit to the other open grant competition. Notably, this exclusion has been lifted for applicants to the 2nd F-scheme pilot who will be allowed to submit to the P-scheme in Spring, 2016. CIHR also refined the anticipated number of F-scheme award to between 150-210 at this point. Stage 2 was focused on the scientific program and comprised 5 character-limited (including spaces) sections: Concept; Approach; Expertise; Mentorship; Environment.
Personally, I found this forcing of structure to be restrictive. CIHR maintains this structuring is necessary for efficiency of review and the reviewers specifically graded each of these sections (with a supremely Canadian scale of – highest to lowest – O++, O+, O, E++, E+, E, G, F, P). These scores were weighted and used to assess the relative ranking as in Stage 2. This approach is fine if the goal is to minimize the investment required by reviewers and to increase operational efficiency of CIHR.
However, such strictures likely come at the expense of scientific creativity. As it stands, the applications for CIHR funding cover the gamut of health research from basic biology to health systems and epidemiology. Requiring each application to be framed in a specific way distorts how science is described, forces a way of thinking and quite possibly leads to conformism that discriminates against those at the margins of thought from whom the most impact often derives.
While fairness and efficiency are clearly important in adjudicating science, these should never interfere with the primary mission of supporting ideas with the potential for the greatest effect. Artificial constraint of how research may be described endangers the very diversity of thinking that is the hallmark of great scientists.
Another casualty is the loss of feedback to the applicants. In the new system, CIHR requires reviewer comments only justify the ranking, they are not intended to provide constructive critique. By contrast, face-to-face panels provided valuable experience and learning opportunities for younger investigators by exposing them to the grant applications and review critiques of others, not to mention less tangible benefits of socializing with peers and exchanging ideas.
This loss of feedback and interaction are deleterious changes that handicap the less experienced. There were opportunities of on-line discussion between Stage 2 reviewers (as well as Stage 1) in order to highlight discrepancies in scoring, although the degree of interaction was reportedly very limited, consistent with high degrees of variance between scores as reported by individuals.
Stage 2 results were released on July 14th, 2015. Of 445 applications, 189 were invited to Stage 3. The remaining 256 were sent reviews and rankings and among this cohort were many highly accomplished investigators, now facing at least a 12 months hiatus in a large chunk of their funding.
The Stage 3 review was conducted by a single, multidisciplinary panel. The top 75 ranked applicants from Stage 2 were denoted “Green Zone” and were not discussed further. Each member of the Stage 3 was tasked with evaluating about 16 applicants that fell into the “Grey Zone” – that is, the applicants ranked ~76 to 189 and were provided with the application materials and reviews from Stage 1 and Stage 2. This final stage was designed to identify anomalies in scoring and to provide sober second thought by a different set of minds. Ultimately, 150 applications were approved for funding, but a further hurdle first had to be jumped.
Show me the money! Applications to Stage 2 included a budget request. F-scheme budget calculations were to be based on existing levels of open operating grant funding from CIHR. For applicants with no prior funding from the agency, information on their other grants could be submitted. Requests for funding above these base-lines was allowed but had to be justified within a few character-limited lines. CIHR communicated to Stage 3 applicants that 90% of their budget requests were significantly above their expected amount and, if approved as is, the funds available would only fund ~54 grants. Applicants were supplied with CIHRs own base budget calculation and asked to justify any increase over this.
The announcement of the results of the F-scheme was timed to coincide with those of the tOOGP on July 15, although a technical glitch inadvertently released the results of the latter competition two days earlier (the profound impact of this on nerves of the applicants cannot be underestimated!).
Any funding agency with set budgets that supports multiple year funding and has annual competitions only has a fraction of its budget available for new competitions. Given the cancellation of a previous OOGP competition, as well as one normally timed for the Fall of 2015, the funds available for the two competitions announced on July 15th was ~$600 over the following 7 years (the longest tenure of an F-scheme award). Existing, non-expired CIHR funding of F-scheme awardees was rolled into their future F-scheme funding – in essence extending these grants to the 5 or 7 year term. Likewise, unfunded F-scheme applicants who had CIHR grants that were not due to expire in FY2015 kept those awards.
Going forward, CIHR has said it expects to support ~114 new F-scheme grants per year at steady state. The number of expected P-scheme grants is unclear, but given that these average 25% of the value of an F-scheme grant, will likely be 200-250 x 2 competitions per year. Notably, the number of open operating grants awarded in 2015 is 533 versus 800 in previous years. In future years, which will have 2 P-schemes as well as an F-scheme competition, this is likely to rise to ~600. The overall success rate of the F-scheme competition was 10.9% and the most recent OOGP as 14.8%.
Lessons and challenges: The impact of the CIHR reforms on researchers has been substantial. Setting aside for a moment the significant concerns about the effectiveness of the virtual review ranking process and the utility of the new scheme design, the simple fact is that nearly 300 accomplished Canadian researchers (those who submitted to Stage 2 but were not funded) cannot expect to receive funding for 12 months, at minimum. This huge opportunity cost is due to the need for CIHR to cancel two competitions during the transitional process.
For the next F-scheme competition, applicants will be eligible to apply for the Spring P-scheme which removes an important opportunity exclusion. It will be interesting to see how many applicants there are to this competition, given the previous experience. Notably, the review criteria have not substantially changed and this must be a disincentive to early career scientists.
There is also concern that the F-scheme will further the Matthew effect (rich get richer) and barriers to entry will increase over time. This is because demonstrable track record is needed and those with multiple grants benefit from higher eligible base budgets. For younger investigators, the fear is that more limited P-scheme opportunities (and intrinsically smaller budgets) will make it ever harder to accumulate funding to make F-scheme funding worthwhile, since the downside of this scheme is lock-in at the approved funding for 5-7 years.
CIHR should consider separate envelopes or evaluation criteria that target career stage. Indeed, mid-career investigators also appear to not have done well in the first competition. Are the review criteria weighted too much to seniority?
Change is always difficult and CIHR will no doubt improve processes over time but there remain deep concerns that the approach taken to these reforms has fundamental flaws. In my view these are:
- Multiple changes were made at once. This is not only terrible science, it precludes any meaningful assessment of the relative performance of the new systems compared with old. We will not know the outcome on quality of science for many years, by which time there will be no going back. Various aspects of the changes were piloted but these were “live”, with real scientific livelihoods at stake. As an experiment, it’s doubtful the reforms would never have passed scientific or ethical review!
- Changes were initiated without any additional funding for transition. This resulted in lost opportunities. Does CIHR intend to support fewer researchers? If so, this was not communicated. Could other new initiatives have been put on hold during the transition to free up bridge funds?
- The biggest impact of the reforms is on the confidence of early to mid-career investigators, many of whom must be seriously evaluating whether they wish to continue their careers in Canada under these circumstances. Do they have a place in the F-scheme? Can they realise their potential with only P-scheme awards? The grass is always greener elsewhere but the funding restraints facing CIHR are similar in many other jurisdictions. However, we should all be worried when many of our most talented people in whom we have already invested are second-guessing their future.
- While CIHR has made much instructional information available, critically including the reviewing parameters, the research community is dependent on CIHRs release of data to evaluate the processes. Release of de-identified information for third party evaluation is essential for researchers to assess the programs and to identify errors and poor behaviours. As many scientists know, there have been heated exchanges at town halls with senior CIHR staff that have eroded confidence in the information being shared. From these meetings, it is clear that Canadian researchers have not given up on face to face reviewing, which the NIH, for example, still regards as the gold standard. Other funding agencies are presumably following the developments at CIHR with interest as they face similar challenges. We have to hope they will benefit from the painful lessons learned here before they follow in our footsteps.