The most important new information that Isabelle Blain provided, and the most validating of NSERC’s new evaluation system, was the result of a survey –conducted by NSERC’s staff– of the panelists who dealt with both systems (from 2008-2010). They seem to prefer the new one.

I don’t have the detailed results of the survey, neither the way it was conducted,  which kind of questions were asked, how many panelists contributed, etc…. After all, if one of the questions was whether the new system was less taxing on the panelists than the old one, then Martin Barlow didn’t leave any doubt about that: “After a while, it takes you about 15 seconds 10-15 minutes to decide that a proposal is (say) an E (Exceptional) O (Outstanding) and S (Strong)”.

This said, I have tendency to believe that the results of the survey were as positive as NSERC would have liked them to be. But I also believe that what the survey picked up is more of the self-satisfaction that panelists feel after completing a given competition cycle, than a definite preference for the new system that is based on a genuine comparison of the two procedures.

My explanation of this phenomenon is through what one can describe as a “Stockholm-type syndrome” among panelists. Indeed, I have always noticed, starting with my days on the grant selection committee, that after a few days and evenings working together, the panelists start experiencing intense feelings of solidarity and mutual loyalty, based on a strong belief in the righteousness of what they have accomplished together.

For example, the leaders of the math/stats GE were very proud of what they had accomplished in this year’s competition, until they heard otherwise from their own community. Even then, they felt compelled to defend the system. Some of them eventually conceded that certain things could have been handled better, but very reluctantly.

Why? Well any one who has been on a Grant Selection Committee can remember the “good” feeling he/she had at the end of the 4-5 days that a competition normally takes. This is to be expected.  Everyone on the panel is often convinced that he/she has done their best to be as fair and as just as possible, under the rules and constraints of the game provided to them. They haven’t been asked to evaluate these rules in the middle of a competition. They are only asked to just play by them, and they do their best under the circumstances and feel good about having done so.

The bottom line is that people who feel they have been entrusted and empowered by the system, are not inclined to criticize it.

Maybe aware of this, Isabelle proceeded to ask me to join the panel next year so that I can see how it really functions from the inside. That was a nice gesture to welcome me back to Ottawa NSERC after 15 years of exile.

Thank you Isabelle, but sorry for not taking your offer. I have already paid my dues to the system.