Increasingly, I am discovering that one of the biggest sources of confusion about the foundations of statistics has to do with what it means or should mean to use “background knowledge” and “judgment” in making statistical and scientific inferences. David Cox and I address this in our “Conversation” in RMM (2011); it is one of the three or four topics in that special volume that I am keen to take up.
Insofar as humans conduct science and draw inferences, and insofar as learning about the world is not reducible to a priori deductions, it is obvious that “human judgments” are involved. True enough, but too trivial an observation to help us distinguish among the very different ways judgments should enter according to contrasting inferential accounts. When Bayesians claim that frequentists do not use or are barred from using background information, what they really mean is that frequentists do not use prior probabilities of hypotheses, at least when those hypotheses are regarded as correct or incorrect, if only approximately. So, for example, we would not assign relative frequencies to the truth of hypotheses such as (1) prion transmission is via protein folding without nucleic acid, or (2) the deflection of light is approximately 1.75” (as if, as Pierce puts it, “universes were as plenty as blackberries”). How odd it would be to try to model these hypotheses as themselves having distributions: to us, statistical hypotheses assign probabilities to outcomes or values of a random variable.
However, quite a lot of background information goes into designing, carrying out, and analyzing inquiries into hypotheses regarded as correct or incorrect. For a frequentist, that is where background knowledge enters. There is no reason to suppose that the background required in order sensibly to generate, interpret, and draw inferences about H should—or even can—enter through prior probabilities for H itself! Of course, presumably, Bayesians also require background information in order to determine that “data x” have been observed, to determine how to model and conduct the inquiry, and to check the adequacy of statistical models for the purposes of the inquiry. So the Bayesian prior only purports to add some other kind of judgment, about the degree of belief in H. It does not get away from the other background judgments that frequentists employ.
However, quite a lot of background information goes into designing, carrying out, and analyzing inquiries into hypotheses regarded as correct or incorrect. For a frequentist, that is where background knowledge enters. There is no reason to suppose that the background required in order sensibly to generate, interpret, and draw inferences about H should—or even can—enter through prior probabilities for H itself! Of course, presumably, Bayesians also require background information in order to determine that “data x” have been observed, to determine how to model and conduct the inquiry, and to check the adequacy of statistical models for the purposes of the inquiry. So the Bayesian prior only purports to add some other kind of judgment, about the degree of belief in H. It does not get away from the other background judgments that frequentists employ.
This relates to a second point that came up in our conversation when Cox asked, “Do we want to put in a lot of information external to the data, or as little as possible?”
As I understand him, he is emphasizing the fact that the frequentist conception of scientific inquiry involves piecemeal inquiries, each of which manages to restrain the probe so as to ask one question at a time reliably. We don’t want our account to demand we list all the possible hypotheses about prions, or relativistic gravity, or whatever—not to mention all the ways in which each can fail—simply to get a single inquiry going! Actual science, actual learning is hardly well captured this way. We will use plenty of background knowledge to design and put together results from multiple inquiries, but we find it neither useful nor even plausible to try to capture all of that with prior degrees of belief in the hypotheses of interest. I see no Bayesian argument otherwise, but I invite them to supply one.[i]
A fundamental tenet of the conception of inductive learning most at home with the frequentist philosophy is that inductive inference requires building up incisive arguments and inferences by putting together several different piece-meal results. Although the complexity of the story makes it more difficult to set out neatly, as, for example, if a single algorithm is thought to capture the whole of inductive inference, the payoff is an account that approaches the kind of full-bodied arguments that scientists build up in order to obtain reliable knowledge and understanding of a field. (Mayo and Cox 2010)
An analogy I used in EGEK (Mayo 1996) is the contrast between “ready-to-wear” and “designer” methods: the former do not require huge treasuries just to get one inference or outfit going!
Some mistakenly infer from the idea of Bayesian latitude toward background opinions, subjective beliefs, and desires, that the Bayesian gives us an account that appreciates the full complexity of inference—but latitude for complexity is very different from latitude for introducing beliefs and desires into the data analysis! How ironic that it is the Bayesian and not the frequentist who is keen to package inquiry into a neat and tidy algorithm, where all background enters via quantitative sum-ups of prior degrees of belief in the hypothesis under study. In the same vein, I hear people say time and again that since it is difficult to test the assumptions of models, we should recognize the subjectivity and background and be Bayesians! Wouldn’t it be better to have an account that provides methods for actually testing assumptions? One that insists that any unresolved knowledge gap be communicated in the final report in a way that allows others to critique and extend the inquiry? This is what an error-statistical account requires, and it is at the heart of that account’s objectivity. The onus of providing such information comes with the requirement to report, at least approximately, whether formally or informally, the error-probing capabilities of the given methods used. We wish to use background information, not to express degrees of subjective belief, but to avoid being misled by our subjective beliefs, biases and desires!
This is part and parcel of an overall philosophy of science that I discuss elsewhere.
[i] Of course, in some cases, a hypothesized parameter can be regarded as the result of a random trial and can be assigned probabilities kosher for a frequentist; however, computing a conditional probability is open to frequentists and Bayesians alike—if that is what is of interest—but even in those cases the prior scarcely exhausts “background information” for inference.
No comments:
Post a Comment
===========================