Facebook Twitter LinkedIn

Home Back Issues No. 1: Grant Clinic: Contradictory Data

Nov 02

No. 1: Grant Clinic: Contradictory Data

Posted by: PIA in

Tagged in: Untagged 

Sign up to receive these free articles weekly.


Reader Question: We are in the process of applying for a grant to extend promising preliminary results. But should we also include mention of some contradictory data we found? This would demonstrate we did extra hours of work, but will it undercut our chances for funding?

Answer: As to the "extra" hours of work you put in, nobody cares. Only results count.

The first purpose in presenting preliminary data is to demonstrate feasibility and that you have the necessary expertise to carry out the procedures you propose in the Research Methods section. I believe it doesn't matter if you have some contradictory results, or even that the data you are describing were collected for other purposes in a different study. The focus in this section is not on the results per se, but on the fact that you were successfully able to collect and analyze the data.

But pilot data, showing positive results, are potent evidence. They indicate that you are on the right track. The reviewers will weight this heavily. There are subtleties to reporting these, however. The possible variations involve (1) primacy of the outcomes; (2) positive/null/negative results; and (3) statistical power of your pilot study to detect effects. Here are the possibilities:

Implications of preliminary results

Positive/Null/ Negative Results

Statistical Power




Reviewer may wonder why you need to conduct the proposed study since you already have strong support

Null/ Negative


The biggest potential problem. You failed to detect effects although had adequate power to do so



Your most desirable pilot outcome. Because power was not adequate to detect effects, presumably your statistical test for this outcome was not significant (if it is, one must suspect chance); thus, you are on the right track: The results bear the replication you propose

Null/ Negative


A problem, but can be addressed. Inadequate power implies that difference between means is not to be believed, probable result of chance. However, don’t overstate, is certainly not a positive.


The main problem is the null result in your primary outcome. If your sample size was small, and power inadequate, this isn't necessarily fatal - but it will require careful wording when you write the section.

One would not necessarily mention the statistical power in pilot data; the reviewer will get a hint of it from the sample size. Remember that in a small sample, even a single outlier will kill your significance.

Next scenario: Null results. One can never prove the null hypothesis, so it doesn’t matter how big the sample size may be, there could always be other reasons for the null results. However, the reviewer is probably going to believe that result and penalize you for it.

Another strategy would be to run a new, quick-and-dirty pilot. The limitation is whether you have the time and resources (and inclination) to do so. And, do you have any reason to think your results will turn out differently?

Let me close on a controversial note. You decide what to report - and what not to. In circumstances like yours, some will bend the rules (note I do not say "break"). A common strategy is to design the study to address several outcomes, in hopes that at least a few will work out (known as “fishing”, not a compliment). Further, you may simply omit mention of null results. However, don't underestimate the intelligence of your reviewers for spotting such maneuvers.

Comments by William Gerin, Ph.D., Professor of Biobehavioral Health, Pennsylvania State University.
Writing the NIH Grant Proposal: A Step-by-Step Guide, SAGE Books (2006)

This eAlert is brought to you as an informational training tool by the Principal Investigators Association, which is an independent organization. Neither the eAlert nor its contents have any connection with the National Institutes of Health (NIH), nor are they endorsed by this agency. All views expressed are those personally held by the author and are not official government policies or opinions.

Comments (5)
written by Bryan Dickerson, November 04, 2009
While the author makes many good points about statistics and the perception of results by a reviewer, I dissagree with the suggestion to omit contradictory results. At Luna, we have found it very important to maintain long-term relationships of trust with our sponsors and with reviewers, and this includes showing or mentioning key problematic results. If I am aware of potential problems with our proposed approach, I prefer to present this data as well as the supporting results. Of course it is then prudent to discuss how we plan to design experiments and research to address the potential problems with a new technology, and to get to the truth of the situation. Careful experiments should be able to isolate variables to determine under what conditions the proposed technology can be successful. This effort towards unbiased communication builds trust and helps to mitigate unrealistic expectations from a sponsor. Then if problems arise during a project, there was informed consent and a forewarning of potential risks. In summary, I believe that addressing problematic results has a better long-term effect than hiding them.
written by Grant Veteran, November 04, 2009
Be careful.Too many contradictory results piled upon each other confuses what is contradiction and what is supposedly being contradicted.
written by Dr. Sue, November 04, 2009
Suggest clarify those contradictory results before applying for the grant. May improve chances for being funded. Or even lead to a better discovery instead of your "encouraging results".
written by Laurel Cooley, November 05, 2009
These implications of preliminary results seem to only apply to studies using statistical analysis. It does not address preliminary results in qualitative research, which the NSF program officers state is encouraged as well.
written by Dr. Ledbetter, November 06, 2009
One of the main problems facing science is the lack of discussion of negative/null results. This often leads to others repeating work that fails, wasted resources and deters scientific progression. I believe it is fine to introduce the contradictory data to start the section and then lead into what you had done to generate the confirmatory data that supports your hypothesis.

Write comment
smaller | bigger

Write the displayed characters