Welcome to this bonus lecture.
I was recently asked: what do you do if you have null results?
And I thought it's a very good question.
And so I decided to add this bonus lecture in which I will do my very best to give you some pointers
to deal with this problem.
So by no results we mean that for example if we are comparing a treatment and a placebo group that there
is no difference between the two groups.
And if you're using inferential statistics then it means that your p value is larger than point 0 5.
And this tends to be a relatively tricky situation because it's quite difficult to sell null results
in your writing.
So in this lecture I will give you some pointers how you can get the most out of your null results.
So first of all why are null results problematic in the first place?
And there are two main problems with null results and the first problem is that conclusions from null
results are often not very convincing.
Remember that the two qualities of an effective answer to your research question is that the question
is convincing and valuable and a null result.
In most cases will make your results less convincing.
So why is that.
Well one reason is that your results can simply reflect methodological issues.
So in the Depression study it could be that the treatment wasn't executed correctly or it could be that
the measure of depression was unreliable and both of these issues can lead to null results even if the
treatment is effective.
And that's the whole problem here.
It's very hard to conclude from a null result that there isn't anything going on for example that the
treatment doesn't work.
Another reason for the same problem is that your results may reflect statistical issues.
Most of the time that will mean that your sample size is too low.
Maybe if you just had more participants we would actually see that the treatment works.
And so for all of these reasons it's hard to draw a convincing conclusion from no results.
So that is the first problem.
And the second problem is that conclusions from no results often also do not seem valuable.
So for example even if you can make the case that there are no methodological issues in your study and
no statistical issues that you can really say with quite high certainty that the treatment didn't work
then the problem is who really cares about an ineffective treatment.
And so to deal with null results we need to fix all of these problems that you can see here.
We need to make the conclusion convincing by ruling out methodological and statistical issues.
And we need to make the case that our results are still valuable.
Okay so the first step will be to deal with methodological issues.
And one of the best solutions is to use a manipulation check if you have a manipulation check.
In your study.
So for example if you show that although the refocus treatment does not reduce depression it does cause
people to focus less on the negative than that may rule out the concern that you just didn't execute
the treatment correctly.
Because it obviously did what it was supposed to do.
It reduced the focus on the negative.
It just didn't reduce depression.
Okay.
If you don't have a manipulation check then another solution you can use is to look at other people's
research.
So the goal here would be to show that your method has been established by previous research so that
there cannot be much doubt about your method.
So for example if you followed the same treatment protocol as another study and if you use the same
measurement of depression as another study and both seem to be finding the other study then there isn't
really much reason to believe that there are methodological issues in your study.
And so by comparing yourself to similar research you can make the case that there were no methodological
issues in your study.
Okay so that is step one.
Now the next step is to rule out statistical issues.
And again there are several things you can do.
One thing you can do is to use a power analysis a power analysis can show that your sample size would
be sufficient to detect a very small effect.
In other words where the power analysis you can make the case that even if there is a treatment effect
it will be very very small.
Another solution is again to use other people's research.
So what you can do is you can show that similar research found significant results with the same or
even a lower sample size than yours.
And I've seen successful examples of both of these solutions.
OK so that's step two.
Now the last step will be to get the reader interested in your null result and usually the way you do
that is by embedding your research into a theoretical context in which the net result is interesting.
Okay that sounds very abstract.
So let me give you an example.
So in the fictional depression study the fact that the treatment is ineffective may not be very interesting.
However this study was not just about the question whether the treatment is effective but it was also
about testing different explanations of depression is depression caused by an unprocessed trauma or
is it caused by a certain information processing style where people overly focus on the negative so
in this case you could make the null result more interesting by focusing more on the implication that
the cognitive explanation of depression.
The one about the negative focus is false.
And the reason why this is more interesting is because this is not just about a specific type of treatment
but it's about a whole class of treatments.
So there's a much bigger implication and therefore a lot more interesting.
OK.
Once you made it through all of these steps then the next thing you will have to do is to write this
down and here's really important to use very cautious language.
Remember what I discussed with you about writing about limitations.
You always want to admit the limitation first so that you don't give the impression that you're just
trying to argue your way out of it.
So you want to write something like given that the results were not significant conclusions must be
drawn with caution.
Nevertheless the data may allow for a few general conclusions when taking all results into consideration
and then you could write down your solutions to all the three steps that we just discussed.
And with such a careful and skeptical introduction your reader will be much more likely to go along
with your thinking rather than thinking that you're just trying to argue your way out of it.
Okay so that's how we can deal with null results.
I hope it's helpful.
And if you haven't given a rating for this cause yet you could do me and future students a huge favor
by just quickly leaving a rating for this course.
All right.
Thank you for listening and let me know if you have more questions.