Chapter 9 Statistical inference conclusion

Which one is better, frequentist, Bayesian or bootstrap? Unfortunately, there is no objective, automatic way to statistical inference. All approaches demand careful planning and consideration of the research question of interest. All approaches rely on assumptions, some more clearly stated as with Bayesian approach, and some more inherent as with frequentist approach. All approaches to statistical inference represent Small Worlds that are hoped to help explain the Large World. Rather than approaching inferential statistics as something that represents and estimates true state of the World, inferential statistics might be approached as descriptive statistic (8).

This section on statistical inference is best summarized with the following quotes:

“Statistical inference often fails to replicate. One reason is that many results may be selected for drawing inference because some threshold of a statistic like the P-value was crossed, leading to biased reported effect sizes. Nonetheless, considerable non-replication is to be expected even without selective reporting, and generalizations from single studies are rarely if ever warranted. Honestly reported results must vary from replication to replication because of varying assumption violations and random variation; excessive agreement itself would suggest deeper problems, such as failure to publish results in conflict with group expectations or desires. A general perception of a “replication crisis” may thus reflect failure to recognize that statistical tests not only test hypotheses, but countless assumptions and the entire environment in which research takes place. Because of all the uncertain and unknown assumptions that underpin statistical inferences, we should treat inferential statistics as highly unstable local descriptions of relations between assumptions and data, rather than as providing generalizable inferences about hypotheses or models. And that means we should treat statistical results as being much more incomplete and uncertain than is currently the norm. Acknowledging this uncertainty could help reduce the allure of selective reporting: Since a small P-value could be large in a replication study, and a large P-value could be small, there is simply no need to selectively report studies based on statistical results. Rather than focusing our study reports on uncertain conclusions, we should thus focus on describing accurately how the study was conducted, what problems occurred, what data were obtained, what analysis methods were used and why, and what output those methods produced." (8)

“Exercising awareness of multiple perspectives, we emphasize that we do not believe that one of these philosophies is the correct or best one, nor do we claim that reducing the different approaches to a single one would be desirable. What is lacking here is not unification, but rather, often, transparency about which interpretation of probabilistic outcomes is intended when applying statistical modeling to specific problems. Particularly, we think that, depending on the situation, both “aleatory” or “epistemic” approaches to modeling uncertainty are legitimate and worthwhile, referring to data generating processes in observer-independent reality on one hand and rational degrees of belief on the other." (56)

“Broadly speaking, 19th century statistics was Bayesian while the 20th century was frequentist, at least from the point of view of most scientific practitioners. Here in the 21st century scientists are bringing statisticians much bigger problems to solve, often comprising millions of data points and thousands of parameters. Which statistical philosophy will dominate practice? My guess, backed up with some recent examples, is that a combination of Bayesian and frequentist ideas will be needed to deal with our increasingly intense scientific environment. This will be a challenging period for statisticians, both applied and theoretical, but it also opens the opportunity for a new golden age, rivaling that of Fisher, Neyman, and the other giants of the early 1900’s.” (42)

This being said, my personal opinion is that more and more sport scientists will consider Bayesian analysis, particularly with the new tools that provide ease of Bayesian model definition and training, as well as visualization of the results. For the benefits of Bayesian over frequentist analysis please see the papers by Kruschke et al. (107,108) as well as suggested reading on Bayesian analysis at the end of this book.

References

8. Amrhein, V, Trafimow, D, and Greenland, S. Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication. The American Statistician 73: 262–270, 2019.

42. Efron, B. Bayesians, Frequentists, and Scientists. Journal of the American Statistical Association 100: 1–5, 2005.

56. Gelman, A and Hennig, C. Beyond subjective and objective in statistics. Journal of the Royal Statistical Society: Series A (Statistics in Society) 180: 967–1033, 2017.

107. Kruschke, JK and Liddell, TM. Bayesian data analysis for newcomers. Psychonomic Bulletin & Review 25: 155–177, 2018.

108. Kruschke, JK and Liddell, TM. The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review 25: 178–206, 2018.