no-18-grant-clinic-can-study-section-reviews-be-appealed

Home Back Issues No. 18: Grant Clinic: Can Study Section Reviews be Appealed?

Mar 01

2010

No. 18: Grant Clinic: Can Study Section Reviews be Appealed?

Posted by: PIA in

Tagged in: Untagged

Grant Clinic:

Can Study Section Reviews be Appealed?

Reader Question: Can study section reviews and scores be appealed? How does an appeal work? How often are such appeals successful?

Expert Comments:

There are many government agencies and foundations which award grants. Each has its own peculiarities as to “appeals”, and that would be too much to detail for a column like this. But, let me select NIH as a typical large donor, and consider them.

Scientific Review Officers (SROs) expend considerable effort to ensure that their panels are stocked with individuals with substantial expertise in the Study Section’s focus area, and that the applications are carefully matched to the reviewers. Before reviewing begins, all reviewers sign a Non-Conflict affidavit covering the applications they are reviewing. During the review process, SROs are carefully attuned to any procedural irregularities that would contaminate an unbiased review, and can terminate the review of an application any such are detected.

NIH expects appeals to be relatively rare, and focused largely along procedural grounds. These could include a biased review, an unrevealed conflict of interest, or a review in which the score was derived from errors of scientific fact. NIH makes a clear distinction between review procedural errors, and disputes with a reviewer centering on differences of scientific opinion. Procedural errors may constitute a basis for submitting an appeal, but differences in scientific opinion are not.

If you are considering an appeal, you should first discuss your review in detail with your Program Administrator. This conversation may clarify the reasons for the score obtained. Program officials frequently attend the Study Section meetings where applications in their portfolio are discussed, and can thus provide contextual information lacking in a written review. Should you believe that grounds exist for an appeal, than you must submit a formal letter, detailing the specific flaws in the review process. Debating the validity of a reviewer’s opinion with regard to the science is not likely to be persuasive; specific errors with regard to process must be articulated.

If you, the Program Administrator, and the SRO are unable to come to agreement, than the appeal falls under the purview of the Institute’s Appeal Officer. This individual will review the file, and then bring it to the attention of the Institute’s Scientific Council. Upon this further layer of review, Council will either recommend that the appeal be dismissed, or that the application be re-reviewed (in its original form without update by the applicant) by a body other than the original Study Section.

In light of the lengthy nature of the appeal process, and the fact that re-review of the application is the best outcome that can be achieved, it is probably wisest to reserve the appeal process for only the most egregious errors in the review process. In the most cases, a faster route to funding likely involves a careful revision and re-submission of your original application.

We could not find any firm statistics on number of appeals made in the most recent five or ten years, and how successful those were. However, informal checking with several experts revealed they have never heard of even one successful appeal. Hence, if there are any successes, they must be rare. Bottom line advice: don't count on an appeal to save your grant application.

Disclaimer- As an ordinary scientist, I can’t formally speak for NIH, so my comments are personal and unoffical. (The formal NIH polices are at: http://www.grants.nih.gov/grants/guide/notice-ﬁles/not97-232.html)

Comments by Christopher Francklyn, Ph.D. Dr. Francklyn is a former Study Section Chair and veteran reviewer for NIH and NSF study sections. He is a professor at University of Vermont, where his scientific expertise is in protein synthesis and RNA-Protein interactions.

Dr. Francklyn provides a regular column, Study Section Insider, in Principal Investigator Advisor monthly newsletter. View his previous article in last month's issue: The Three Reviewers Critical to your Grant’s Success .

Be on the lookout for his new column in the March issue: 5 Common Mistakes that Will Sink Your Grant .

Enjoy this article? Sign Up to receive these free every week

This eAlert is brought to you as an informational training tool by the Principal Investigators Association, which is an independent organization. Neither the eAlert nor its contents have any connection with the National Institutes of Health (NIH) or the National Science Foundation (NSF), nor are they endorsed by these agencies. All views expressed are those personally held by the author and are not official government policies or opinions.

[ Back ]

Comments (19)

...
written by Politico, February 25, 2010

Some years ago, in a Study Section we gave a low, unfundable score to what we considered a very weak application. The upset researcher responded by complaining to the U.S. Senator who represented his state. Well, believe me, the NIH moves fast when queried by a Senator (Congress controls NIH funding). Likety split, the application was brought back for re-review at the very next meeting. The Program Officer went "off the record" to explain the situation and asked us to reconsider carefully, but he also made clear NIH would not overrule a Study Section final verdict. In the discussion, we all grasped for overlooked crumbs of merit which could boost the application, but there were none. Again our vote was strong negative. So somebody in the top echelon of NIH had to send the Senator an artful letter. Just relaying this incident to show "appeals" can take many forms.

...
written by anon, March 02, 2010

Just appealed my RO1 scored baised on obvious reviewer bias. SRO agreed and allowed a new "first submission" of the grant. Very important in this environment of only 2 submissions. The reviewer stated the reason for putting the grant in the "not discussed" category was based on the PI being a poor investigator with no publications in the field and only one publication 5 years ago. Being that I have extensive publications (more than 20) in the field including one that came out in a high impact journal (Impact 9.7) a month before my application was submitted, there was obviously a bias. This, with the fact that the other 2 reviewers scored the application highly (but didn't rescue, unfortunately) gave the SRO a reason to allow the appeal. So, while appeals should be only sparingly considered and used, when there is real reason for it-there can be a positive outcome.

...
written by OldTechie, March 02, 2010

Don't appeal!! I have a 30 year record of funding. My only experience was a "successful appeal" 20 years ago on an egregious error. A committee was formed that considered the appeal and granted it, and I then got a re-review. I sensed, however, that rather than feeling the system had kept its integrity, I was labeled for a time as a troublemaker. The lesson I learned was swallow hard, and try to correct the error tactfully. Sometimes, you'll get reviewers who are ashamed of aving made a mistake, and will give you a little boost in return. Other times, the rereview will be met by "Facts be damned, how dare you diasagree with me?" Unfortunately, for all of the teaching of scientific ethics demanded by NIH, people are still people.

...
written by Adriana, March 02, 2010

First off the person that you talk to after the review is not the person that runs the study section. The program administrators are rarely at the meeting and I have never been able to get useful information from them about the discussion of a grant. It seems like a big disconnect to have to talk to someone about the grant that was not at the discussion but that is the way it is set up. Second, I have been at many study sections where clear conflict of interest exists. It is left to each individual to claim the conflict and many people do not do that. I pointed out a conflict to the SRO at a study section once and she told me that there was not conflict because the reviewer said there was not conflict even though the reviewer and applicant had several co-authored publications.

...
written by Genejock, March 02, 2010

Around a decade ago I submitted an application that I felt would really move my field of research forward. To my dismay, my first application for funding by NIH was returned unscored. However, the real surprise arrived with the 'pink sheets'. The primary reviewer wrote less than 200 words concerning my grant. The review was summarized with a dismissive statement revealing that he/she had not actually read it. On contacting my Program Officer I was told that the process of appeal was too lengthy and cumbersome to be effective. He said that I was better off addressing whatever criticisms I could prior to re-submitting it to the same study section. This I did, only to be unscored again with a nearly identical review in the next round! Preliminary data from the grant was eventually published in one of the Nature journals. And- perseverance paid off when a re-worked proposal was eventually funded with an outstanding score by another study section.

...
written by Dabs, March 02, 2010

A few years ago, I should have appealed a review that had major flaws. One reviewer misrepresented the proposal and the state of knowledge in ways that could be shown quite clearly, and this reviewer also used a mocking tone told us we made ourselves look "ridiculous". I sensed quickly from talking with my program officer that this behavior is perfectly OK with NIH, and that she was going to circle the wagons and refuse to question anything in the review no matter how egregious. Everything I have heard indicates that the appeal process is biased against claimants, who are just considered as troublemakers. My advice is never to appeal for the purpose of getting your proposal reconsidered, but only for the sake of weeding out emotionally unstable reviewers and teaching program officers and review officers a lesson in how the review process can go badly wrong. Ultimately you might be helping to reform a deeply flawed system If you appeal for the purpose of saving your proposal, the best that can happen is that you will get a re-review after a length delay. Your proposal will be out of date and you will have lost the opportunity to make revisions. If I ever choose to appeal in the future, I will extract the best parts of my research plan and submit them to a different agency while the review is in process.

...
written by Jake, March 02, 2010

In our judicial system, we never allow the person(s) who made a decision to rule on an appeal, because bias toward defending the original decision is too obvious. Federal funding agencies cannot grasp this basic idea. I agree with the opinions above -- don't bother appealing, ever. Dabs' idea that you may improve "the system" is optimistic; you probably can't change human nature.

...
written by RKS, March 02, 2010

I've been at both ends of the problem, i.e had clearly unfair reviews of my own applications and been on study sections where there is apparent evidence of reviewer bias. Reviewers on study sections have a lot of power and can easily wield it without detection: it's never difficult to find at least some faults in any application, no matter how brilliant it is. If a reviewer want's to sink your proposal, that's simply tough luck for the applicant... I really don't think there is anything that can be done about it. Any reviewer who is out to sink a proposal will not likely make factual errors but more likely to nit pick. The best proposal that I ever wrote was unscored and most likely reviewed by my competitor who happened to be on the panel that day and the only one qualified to review it so most likely did review it. I have had substantially weaker proposals funded previously. The reviewer focussed on the limitations of the study without acknowledging that there are no better alternatives. Factually he/she was correct so no real basis for me to make a strong appeal. Basically, you need an advocate on the panel to rescue your application from a reviewer who wants to sink it but it's a rare coincidence that one has one on the day it matters. My only recommendation is to request that a certain study section member not review your application next time you resubmit. I don't know if the NIH really pays much attention to such requests though. Bottom line, keep submitting your grants into a system that is seriously flawed but probably still the best in the world. The only exceptions I can think of easily are where a reviewer makes factual errors...and that is most likely because they didn't understand or make an effort to understand and their intention was probably not focussed on sinking the application in the first place.

...
written by Neo, March 02, 2010

It's rather plain that the NIH system is biased against appeals. I appealed once after receiving a review of an unscored application in which text that had identical phrasing (but only in some sections) was used by two different reviewers to critique the proposal. Now, because this proposal was unscored the two reviewers were supposedly not to have the opportunity to be influenced by one another, so this was at best a procedural error and at worst an ethical lapse. My reasoned appeal was dismissed with an explanation that a "computer error" had occurred. Nice. My "faith" in the system remains damaged. I often wonder in this instance had the institute been held to the same standards that NIH they would hold their investigators how this would have played out...

...
written by Oldreviewer, March 02, 2010

My research has been funded by NIH and other federal agencies for nearly 50 years, and I have served on various review panels for NIH and NSF. I have to agree with the practical advice of those who recommend against appealing bad reviews, since the process is strongly rigged against their success. On the other hand, I suspect that the number of worthwhile, even outstanding, new ideas that are rejected constitutes a significant loss to the agencies and to the people who ultimately pay the research bills. I am convinced that a better screening process could identify appeals that, if their claims are valid, should not only return those proposals for FAST reconsideration, but also result in some loss of points (in some form) for the reviewers that did the sloppy job. Judges are careful because they do not like being overturned, why not reviewers? An ombudsman's office with some real authority might help to reduce the number of reviewers who make casual errors (or clearly biased statements) simply because no one ever calls them on that behavior. The assumption that all those who gripe about bad reviews are merely whiners is nonsense. I have seen many examples of bad reviewer's judgements in reviews I and others have received. On the average reviewers' scientific evaluations correlate positively with what a more valid system could achieve,but there is a great deal of room for improvement...and improving that correlation by even a modest amount would result in a lot more good science being done for the same total bill.

...
written by ANON, March 02, 2010

NIH reviews are nothing but the actions of street gangs. As the gangster gets challenged, first reaction is to find technicality to reject and steal ideas. We must have a clas of reviewers that can never in their life interact with other scientists for this to be truely without bias. One can not do that, so this type of favouratism in science and revolving door mentality that originates in selfishness and greed will go on! There need to be a judiciary review to judge conflicts of interest and actions of reviewers, with huge liability [$ awards as fines] attached to agency and reviewer's behavior, so that the biased person looses heavily to a level where the reviewer looses ability to earn even a penny! Such punishments are must, and need to be incorporated in the process of reviews!

...
written by Mousedoc, March 02, 2010

Anon, you are abviously as ignorant as they come. Anyways, there are very few people that enjoy being on study section. Most do it out of a sense of duty and they try to do their best job. Why would anybody keep doing this painful chore that really does not do anything for you if you make it a "huge liability" with "$awards as fines" as an added "benefit". The real problem is that there is not enough money to support all the good research that is being done. Now if you would find a better way getting rid of terrorists that would be another matter altogether. For you I suggest think before you speak, or better yet, don't speak at all.

...
written by kai, March 02, 2010

There are always conflicts of interest (easy to spot, too) and personal bias at every study section. Nonetheless I have been impressed by the overall competence, decency and hard work of the reviewers. Much better than in Europe or anywhere else.

...
written by Sinner?, March 02, 2010

This old reviewer can now confess that late at night, shortly before deadline, faced with a ple of applications-- I engaged in a kind of triage. Sorry for those of you who slaved over the dull, almost routine parts of your forms. If you didn't catch my interest fast with your mission and approach, you were toast.Do you remember a movie where the female lead said "You had me at 'Hello'". Well it was like that.

...
written by half empty, March 02, 2010

I believe that overall the NIH review process works. Nonetheless, I have personally seen enough examples of egregious errors of ignorance, laziness, or malicious conflict of interest to know that it could be done better, and to know that there is rarely any justice available to wronged applicants. But WHO is going to fix it? Where are appeals panels gong to come from when NIH struggles to fill its regular study sections with competent reviewers? Who will take the place of the bad eggs who get banned from reviewing? I think part of the problem is that the function of scientists as individuals has become so dependent on obtaining external funding that the function of scientists as a community is beginning to break down. We are judged so heavily by the number and size of our grants, that other service is regarded as a handicap, taken on only at the peril of falling behind in the funding competition. The very best can do it all well, and the most noble one's actually serve. But many of us can't or don't, despite better intentions. Teaching also suffers enormously from this. If the value of such non-monetary components of science continues to depreciate, the review process will only get worse.

...
written by The Orwellian Philosophy, March 02, 2010

The appeal to a bureaucratically run process that is nothing more than an expensive exercise in censorship is futile. "Procedural errors may constitute a basis for submitting an appeal, but differences in scientific opinion are not."--Procedural errors are rarely the flaw. The intrinsic flaw is the myopic knowledge base of a single reviewer and the ability to express arrogant myopic opinion behind the dark cloak of anonymity with no feedback or penalty.

...
written by buggirl, March 04, 2010

I am on review panel and one reviewer advocated for a grant that was, literally, unfinished (no kidding, it actually stopped in mid-sentence, and not very close to the end, either!) We have another guy who gives 30% of his reviews a "1"! However, I suggest that if you have someone who will tank you no matter what you send in, (and we all know there are such people), include their name and institution in the cover letter and be as specific as you can be about why there is a conflict even if it means including specific information.

...
written by alcoholpolicy, March 09, 2010

I know of one successful appeal in relatively recent years, but it was one that an attending IC staff (not SRO)invited because he perceived a major scientific error in the primary reviewer's critique and in the discussion that swayed other section members. In the special review that followed our investigator prevailed. The lesson would seem to be to encourage PO's to attend and hope they have the conviction and integrity to want to set matters straight--also have a good relationship with IC staff so that they actually value and know [your] competencies. Small technicalities do not seem to tip the balance.

...
written by heartbroken, October 07, 2010

So, do you know of any application that ended up being funded after an appeal? If never, there is something seriously wrong in the system as all systems have flows (even our court systems) and there has to be a possibility for a corrective action.

Write comment