P-values: misunderstood and misused
(January 2016)cite arxiv:1601.06805Comment: Submitted to Frontiers in Physics, Interdisciplinary Physics, "Research Topic": At the crossroads: lessons and challenges in Computational Social Science.Abstract
P-values are widely used in both the social and natural sciences to quantify
the statistical significance of observed results. The recent surge of big data
research has made p-value an even more popular tool to test the significance of
a study. However, substantial literature has been produced critiquing how
p-values are used and understood. In this paper we review this recent critical
literature, much of which is routed in the life sciences, and consider its
implications for social scientific research. We provide a coherent picture of
what the main criticisms are, and draw together and disambiguate common themes.
In particular, we explain how the False Discovery Rate is calculated, and how
this differs from a p-value. We also make explicit the Bayesian nature of many
recent criticisms, a dimension that is often underplayed or ignored. We also
identify practical steps to help remediate some of the concerns identified, and
argue that p-values need to be contextualised within (i) the specific study,
and (ii) the broader field of inquiry.
