“You May Believe You Are a Bayesian But You Are Probably Wrong”

The following is an extract (58-63) from the contribution by

Stephen Senn  (Full article)

Head of the Methodology and Statistics Group,

Competence Center for Methodology and Statistics (CCMS), Luxembourg

.......

I am not arguing that the subjective Bayesian approach is not a good one to use.  I am claiming instead that the argument is false that because some ideal form of this approach to reasoning seems excellent in theory it therefore follows that in practice using this and only this approach to reasoning is the right thing to do.  A very standard form of argument I do object to is the one frequently encountered in many applied Bayesian papers where the first paragraphs lauds the Bayesian approach on various grounds, in particular its ability to synthesize all sources of information, and in the rest of the paper the authors assume that because they have used the Bayesian machinery of prior distributions and Bayes theorem they have therefore done a good analysis. It is this sort of author who believes that he or she is Bayesian but in practice is wrong. (58)

3. Reasons for Hesitation

The first of these is temporal coherence. De Finetti was adamant that it is not the world’s time, in a sense of the march of events (or the history of ‘one damn thing after another’), that governs rational decision making but the mind’s time, that is to say the order in which thoughts occur or evidence arises. However, I do think that he believed there was no going back. You struck out the sequences of thought-events that had not occurred in your mind and renormalized. The discipline involved is so stringent that most Bayesians seem to agree that it is intolerable and there have been various attempts to show that Bayesian inference really doesn’t mean this. I am unconvinced. I think that de Finetti’s theory really does mean this and the consequence is that the phrase ‘back to the drawing board’ is not allowed. Attempts to explain away the requirement of temporal coherence always seem to require an appeal to a deeper order of things—a level at which inference really takes place that absolves one of the necessity of doing it properly at the level of Bayesian calculation.  This is problematic, because it means that the informal has to come to the rescue of the formal. …(58)

I am not criticizing informal inference.  Indeed, I think it is inescapable. I am criticising claims to have found the perfect system of inference as some form of higher logic because the claim looks rather foolish if the only thing that can rescue it from producing silly results is the operation of the subconscious. Nor am I criticising subjective Bayesianism as a practical tool of inference. As mentioned above, I am criticising the claim that it is the only system of inference and in particular I am criticising the claim that because it is perfect in theory it must be the right thing to use in practice. (59)

It is hard to see what exactly a Bayesian statistician is doing when interacting with a client. There is an initial period in which the subjective beliefs of the client are established. These prior probabilities are taken to be valuable enough to be incorporated in subsequent calculation. However, in subsequent steps the client is not trusted to reason. The reasoning is carried out by the statistician. As an exercise in mathematics it is not superior to showing the client the data, eliciting a posterior distribution and then calculating the prior distribution; as an exercise in inference Bayesian updating does not appear to have greater claims than ‘downdating’ and indeed sometimes this point is made by Bayesians when discussing what their theory implies. (59)…..

In a paper published in Statistics in Medicine in 2005 Lambert et al. considered thirteen different Bayesian approaches to the estimation of the so-called random effects variance in meta-analysis. …..

The paper begins with a section in which the authors make various introductory statements about Bayesian inference. For example, “In addition to the philosophical advantages of the Bayesian approach, the use of these methods has led to increasingly complex, but realistic, models being fitted” and, “an advantage of the Bayesian approach is that the uncertainty in all parameter estimates is taken into account” (Lambert et al. 2005, 2402) but whereas one can neither deny that more complex models are being fitted than had been the case until fairly recently, nor that the sort of investigations presented in this paper are of interest, these claims are clearly misleading in at least two respects. (62)

First, the ‘philosophical’ advantages to which the authors refer must surely be to the subjective Bayesian approach outlined above, yet what the paper considers is no such thing. None of the thirteen prior distributions considered can possibly reflect what the authors believe about the random effect variance. One problem, which seems to be common to all thirteen prior distributions, is that they are determined independently of belief about the treatment effect. This is unreasonable since large variation in the treatment effect is much more likely if the treatment effect is large (Senn 2007b). Second, the degree of uncertainty must be determined by the degree of certainty and certainty has to be a matter of belief so that it is hard to see how prior distributions that do not incorporate what one believes can be adequate for the purpose of reflecting certainty and uncertainty. (62-3)

Certainly, another Bayesian paper on meta-analysis only a few years later (Higgins et al. 2008) agreed implicitly with this, … This latter paper by the by is also a fine contribution to practical data-analysis but it is not, despite the claim in the abstract, “We conclude that the Bayesian approach has the advantage of naturally allowing for full uncertainty, especially for prediction”, a Bayesian analysis in the De Finetti sense. Consider, for example this statement, “An effective number of degrees of freedom for such a t-distribution is difficult to determine, since it depends on the extent of the heterogeneity and the sizes of the within-study standard errors as well as the number of studies in the meta-analysis.” This may or may not be a reasonable practical approach but it is certainly not Bayesian. (63)

There are two acid tests. The first is that the method must be capable of providing meta-analytic results when there is only one trial. That is to say the want of data must be made good by subjective probability. The practical problem, of course, is that you cannot estimate the way in which the results vary from trial to trial unless you have at least two trials (in fact, in practice more are needed). But to concede this causes a problem for any committed Bayesian. (63)

The second test is that whereas the arrival of new data will, of course, require you to update your prior distribution to being a posterior distribution, no conceivable possible constellation of results can cause you to wish to change your prior distribution. If it does, you had the wrong prior distribution and this prior distribution would therefore have been wrong even for cases that did not leave you wishing to change it. This means, for example, that model checking is not allowed. (63)

I invite your philosophical reflections, (please see "U Phil" post), in "comments" or send to error@vt.edu (by 1/22)

I will post mine tomorrow, Senn's and "yours" next week

de Finetti, B. D. (1974), Theory of Probability (Volume 1), Chichester: Wiley.

— (1975), Theory of Probability (Volume 2), Chichester: Wiley.

Higgins J. P., S. Thompson and D. Spiegelhalter (2008), “A Re-evaluation of Random effects Meta-analysis”, Journal of the Royal Statistical Society, Series A 172, 137–159.

 Lambert, P. C., A. J. Sutton, P. R. Burton, K. R. Abrams and D. R. Jones (2005), “How Vague is Vague? A Simulation Study of the Impact of the Use of Vague Prior Distributions in MCMC Using WinBUGS”, Statistics in Medicine 24, 2401–2428.

Senn, S. (2007b), “Trying to Be Precise about Vagueness”, Statistics in Medicine 26, 1417–1430.

Senn, S. (2011), “You May Believe You Are a Bayesian But You Are Probably Wrong” Rationality, Markets and Morals (RMM) Vol. 2, 2011, 48–66.