What’s a p-value and can you trust it?

The notes below are inside look on how we structured this week’s episode … If your p-value is higher than .05, you can’t publish research. Antonio: What is a p-value?   Jordy: The P value, or calculated probability, is the probability of finding the observed, or more extreme, results when the null hypothesis (H0) of Read More ...

The notes below are inside look on how we structured this week’s episode …

If your p-value is higher than .05, you can’t publish research.

Antonio: What is a p-value?  

Jordy: The P value, or calculated probability, is the probability of finding the observed, or more extreme, results when the null hypothesis (H0) of a study question is true

https://www.statsdirect.com/help/basics/p_values.htm

Antonio: Okay, so if you’re like me … not a statistician … you want to have simpler language even if the explanation is longer.  I reread that and think man … what is this null hypothesis? That’s really what got me hung up. How about you Jordy?

Jordy: ….

Antonio: So, the null hypothesis is that whatever you are trying to test has no significant difference from the population.  Your hypothesis, whatever you are trying to prove,is called the alternative hypothesis.

Okay, so let’s make up an example.  I’ll use Penn State’s List of 7 steps.

Define null hypothesis

So, let’s make up a null hypothesis – College freshmen students gain an average of 10 lbs during their first year of college.  Let’s say we have the standard deviation of 3 lbs. Our null hypothesis is that there will be no significant difference between this population and our sample.

2. Define alternative hypothesis

Our alternative hypothesis is that students that are given an electronic scale at the beginning of their college year will impact their weight by the end of the year.

3. Set probability / alpha

.05 – 5% ?

Why would you make this number lower than 5% ? What if you get it wrong?

4. Collect Data

Experimental or Observational

5. Calculate the test statistic

Okay, I think where the Statistician magic happens the most.  Essentially, this statistic measure compares the data that we collected or observed compared to our overall population.  

I say that the most magic happens here because you need to know a bit about the distribution of the data and the pluses and minuses of each statistic.  You then need to plug in the data to the formula – even I can do that part.

6 / 7 – Based on that measure – which is sometimes just how many standard deviation a our data is from the population – we then need to measure the likelihood of that

Now you have the likelihood on it and then you compare if that’s lower than your alpha value. If it is, you can now reject the Null Hypothesis.

….

It sounds decent to me so why is there such issue with p values?  Well, I think when people are given one measure for success, they’ll figure out how to beat it, fudge it, or bend the rules a bit.

https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002106

Inflation bias, also known as “p-hacking” or “selective reporting,” is the misreporting of true effect sizes in published studies (Box 1). It occurs when researchers try out several statistical analyses and/or data eligibility specifications and then selectively report those that produce significant results [12–15].

Common practices that lead to p-hacking include:

conducting analyses midway through experiments to decide whether to continue collecting data [15,16];

and stopping data exploration if an analysis yields a significant p-value [18,19].

recording many response variables and deciding which to report postanalysis [16,17],

deciding whether to include or drop outliers postanalyses [16],

excluding, combining, or splitting treatment groups postanalysis [2],

including or excluding covariates postanalysis [14],

According to one paper we found,

The Extent and Consequences of P-Hacking in Science, while p-hacking is probably common, its effect seems to be weak relative to the real effect sizes being measured.

270 authors work to repeat 100 experiments.  ‘Even with all the extra steps taken to ensure the same conditions of the original 97 studies only 35 of the studies replicated (36.1%), and if they did replicate their effects were smaller than the initial studies effects.’

Resources

https://www.statsdirect.com/help/basics/p_values.htm

Seven Steps for Testing

https://newonlinecourses.science.psu.edu/stat502/node/139/

Inflation Bias aka p-hacking

https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002106

Covariates

Confusing Statistical Terms #5: Covariate

The Extent and Consequences of P-Hacking in Science

https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002106

Reproducibility Project

https://en.m.wikipedia.org/wiki/Reproducibility_Project