Error Statistics Philosophy

MOVED: Reflections on the Paradigm Change (2/1/12): NEW ADDRESS: http://errorstatistics.com

2012-01-31T09:02:00.020-05:00

Dear Reader: (Wed. noon):  Well we've moved, in the sort of piecemeal, unsettled manner that I suppose the error statistical philosophy favors: take the leap first, then be compelled to adjust by trial and error stress tests afterward.  As with all such progressive changes, while some of the old problems are solved, new and deeper problems appear; things that were easy to explain in the old paradigm are no longer explicable (as of yet), and some activities that were problem-free and predictable before, are now filled with puzzles and uncertainties.  I'm not throwing out this glorious old typing machine just yet however...it still serves in some areas as a kind of limiting case, and standard, when the exactitude of the new paradigm is scarcely needed.  Well, see you over there...Taking a day off to unpack all those boxes that the moving people brought over to errorstatistics.com this morning. 
Best, Mayo

Dear Reader: Tuesday evening:
Well, I lied, because I'm back to using my trusty old machine while snafus with the new, improved platform are ironed out--and isn't it lucky that I didn't take it out with the trash this afternoon?  If you wish to post any comments, solutions to the riddle, or share condolences, you may still post here.  Sorry.
D. Mayo

Dear Reader: (Tues. a.m.)
Well this is my last post typed out on this old thing. I thought it perfectly good actually, even though this keyboard is a bit rusty and the fonts often have a mind of their own; I shall miss it.  You will laugh to hear that a representative from Elba actually had to fly all the way to the U.S. to ensure I’d finally stop dithering and move to the new-fangled blog site that currently exists as an alternative universe (until we turn it on).  My Elbian friends know all too well that in situations like these I have a tendency to display what Nietzsche calls the Russian Fatalism:

“by tenaciously clinging for years to all but intolerable situations, places, apartments, and society, merely because they happened to be given by accident: it was better than changing them, than feeling that they could be changed.” (Ecce Homo, I think it’s Why I am So Wise)

Anyway, as my last little puzzle from here, you may have noticed a caption under the picture of Elba (on this page) for the past several weeks (although no one has mentioned it):

able no stats on Elba

What is interesting about this phrase do you spoze?  You only have today to answer (for a w-point).

Yours Sincerely,

D. Mayo 
P.S. CLICK ON PICTURE OF PINK WASHING MACHINE TO GET IMMEDIATELY TO NEW SITE

Moving Day TODAY!

2012-01-30T21:02:00.002-05:00

We are in the process of moving the Error Statistics Philosophy Blog (including all previous posts and comments) to a new platform.

Our understanding is that the Word Press platform provides for a better reader experience and commenting should be much quicker and easier.

Starting today (January 31), you will be able to find us at:

http://errorstatistics.wordpress.com/

We hope to see you there. Best, the Sailor from Elba

No-Pain Philosophy: Skepticism, Rationality, Popper, and All That: The First of 2 Parts

2012-01-29T09:15:00.008-05:00

I want to shift to the arena of testing the adequacy ofstatistical models and misspecification testing (leading up to articles by Aris Spanos, Andrew Gelman, and David Hendry). But first, a couple of informal, philosophical mini-posts, if only to clarify terms we will need (each has a mini test at the end).

1. How do we obtain knowledge, and how can we get more of it?

Few people doubt that scienceis successful and that it makes progress. This remains true for the philosopherof science, despite her tendency to skepticism. By contrast, most of us thinkwe know a lot of things, and that science is one of our best ways of acquiringknowledge. But how do we justify our lackof skepticism? Any adequate account of the success of science has to squarewith the fact of limited data, with unobserved and unobservable phenomena, withtheories underdetermined by data, and with the slings and arrows of the threatof error. Intent on supplying such an account, I’m drawn not toward some idealform of knowledge “deep down,” nor toward some perfectly rational agent, butsimply toward illuminating how in fact we get the kinds of knowledge we manage to obtain—andhow to get more of it!

As such, the centrally relevant question is: How do we learn aboutthe world despite threats of error?

2.Inductive inference as “Evidence Transcending”

The risk of error enters because we wantto find things out—make claims or take action—based on limitedinformation. When we move beyondthe data to claims that are “evidence transcending,”the argument, strictly speaking, is inductive. The premises can be true whilethe conclusion inferred may be false—without a logical contradiction.Conceiving of inductive inference, very generally, as “evidence-transcending”or “ampliative” reasoning frees us to talk about induction without presupposingcertain special forms it can take. Notably, while mathematical probability arisesin inductive inference, there are two rival positions on its role:

· to quantify the degree of confidence, belief, or support to assignto a hypothesis or claim (given data x); and

· to quantify how reliably probed, well-tested, or corroborated a claim is (given data x).

This contrast is at the heartof a philosophical scrutiny of statistical accounts.

Confusion about induction andthe threat of the so-called philosophical problem of induction have made somepeople afraid to use the word—even statisticians, who could in fact be teachingphilosophers about it. Such fears,however, are unwarranted, once it is properly understood. But more important, even those whoclaim to restrict themselves to variations on “deductive” falsification mustwarrant their premises empirically, as that arch falsificationist Karl Popperknew only too well.

3. Popper, Probabilism and Severe Tests

Since Popper keeps popping upin the statistical literature (e.g., in Stephen Senn, Andrew Gelman), let me try without philosophical fanfare to say somethingabout him. In one sense Popper wasa skeptic: he didn’t think we could justify hypotheses as either true orprobably true—he rejected “probabilism” (also called “inductivism,” which isconfusing). Yet he still thoughtscience was successful and that there were rational methods of science.

Consider the two views ofprobability just given. The first view, that probability arises to assigndegrees of belief, truth, or support to hypotheses, goes hand in hand with theconception that claims are warranted by being either true or probable. Popperians also call this probabilism or justificationism, and we can retain those terms. When it is saidthat Popper was an anti-inductivist, what is really meant is that he rejectedprobabilism. He did not reject theidea that it was possible to warrant evidence-transcending claims. Instead, he required thatevidence-transcending claims be accepted (or preferred, or inferred) only ifthey have been subjected to, and have passed, stringent tests. Probability,accordingly, arose in the second sense. Here, probability is best seen ascharacterizing the properties of a method or rule, which we may call a methodof testing.

For example, agood testing rule might be one that with high probability would falsify ahypothesis H if false, but nototherwise. For Popper, a hypothesis was well corroborated only to the extentthat it passed a severe attempt to falsify it.

4. The Wedge between Skepticism and Irrationalism

This focus on using probability to qualify testing methods(rather than claims) is the key that allowed Popper to “drive a wedge betweenskepticism and irrationalism”. (e.g. Musgrave 1999, p. 322). We can be skeptics about inductively inferring that H is probable---wecan reject probabilism outright---while still allowing warranted inferences toH. We need only have rationaltesting methods or rules.

We can allow

Itis warranted to infer H

to be open toincluding any number of epistemological stances: It is warranted to infer(believe, accept, prefer, act in accordance with) H.

Popperians oust probabilism but retainrationality by defining rationality as following a rational method. Arational method is one that infers (or claims to have evidence for) H only tothe extent that H has passed a severe test---a test which would have, withreasonably high probability, detected a flaw in H, were it present.[i]

Popperoften viewed hypotheses as “solving problems” –where the problem could beconstrued very generally. So wemight say that an irrational method is one that would, with high probability,declare our problem solved, when in fact it was not (or not solved to a degreespecified). Such a method would to readily declare the problem solvederroneously.

Popperiansescape the need to come up with a logic of (or methods for) confirmation—but theystill need methods for severe tests, and ways to evaluate tests on theirerror-probing ability. Can this beachieved? Stay tuned.

_______________________________________________

Mini-Test:
What are thetwo uses of probability in inference?
How does Popperdrive a “wedge” between skepticism and irrationality?
What isprobabilism (or justificationism)?
What would bean irrational method for solving a problem?

____________________________________________

Lakatos, I. 1978. The Methodology of Scientific Research Programmes. Edited by J. Worrall and G. Currie. Vol. 1 of Philosophical Papers. CUP.

Mayo, D. 1996. Error and the Growth of Experimental Knowledge. Chicago: Universityof Chicago Press.

Mayo, D. 2011. “Statistical Scienceand Philosophy of Science: Where Do/Should They Meet in 2011 (and Beyond)?” Rationality, Markets and Morals (RMM) Vol. 2: 79-109.

Musgrave, A. 1999. Essays inRealism and Rationalism. Amsterdam: Rodopi; Atlanta, GA.

Popper,K. 1959. The Logic of ScientificDiscovery. New York: Basic Books.

Popper,K. 1983. Realism and the aim of Science. NJ: Rowman and Littlefield.

“Mere supporting instances are as a ruletoo cheap to be worth having…any support capable of carrying weight can onlyrest upon ingenious tests, undertaken with the aim of refuting our hypothesis,if it can be refuted (Popper 1983, 30).

A KODAK MOMENT- Contributed by Mark H. Mayo[i] and (his sister) Deborah G. Mayo

2012-01-26T22:21:00.017-05:00

Louis J Mayo and Elizabeth Mayo, Mayo Studios

A company filed for Chapter 11 last week. So what else is new?

But it’s not just any bankruptcy. It’s the bankruptcy of Kodak. Which is something other than the failure of a corporation that (or “who,” you might say) has been in business since 1889. It’s a kind of tragedy, rooted in denial—in the refusal to acknowledge that to remain the leader, they would have to change.[ii] And it’s a specifically American tragedy, obviously not on the order of losing JKF, Elvis, or Marilyn Monroe, but the loss of a piece of Americana all the same.

Eulogies for this one-time leader of photography are everywhere in the past week. After all, our generation grew up with Kodak. Our first Brownie cameras, then our single-lens-reflexes, then slides. Who among us of a certain age doesn’t still have Kodak prints, slides, or movies? Or hasn’t been posed—or posed our children or grandchildren—in front of a Kodak picture-spot sign at Disneyland or the World’s Fair? Even Simon and Garfunkel begged to keep their Kodachrome, which “makes you think all the world’s a sunny day.”

The nostalgia is a bit more personal for those long in the photography business. Growing up as a child of Louis Mayo meant growing up around film. It meant the mystery of darkrooms, the smell of emulsion chemicals, the shadows of negatives of room-settings. Harvard Business School’s Rosabeth Kanter told The Economist that Kodak’s executives simply “suffered from a mentality of perfect products, rather than the high-tech mindset of make it, launch it, fix it.”

Louis Mayo was a perfectionist, too, and staying with Kodak was part of that. When Mayo Studios’ largest clients compared the quality of Kodak to Fuji or AGFA, they insisted staying with the policy of all Kodak. When the Studio had trouble keeping colors consistent, the Kodak Company, committed to excellence, actually sent people down from Rochester to ensure that the emulsions would be precisely right.

Then there were the countless slides he shot anywhere and everywhere, which foreshadowed, with Kodak’s help, his success in business. He built a commercial photography studio that would lead the way in the home-furnishings industry. In his 38 years as president of Mayo Studios, www.mayostudios.com, he purchased, developed, and delivered millions of sheets of Kodak’s 8 by 10 color transparencies, black and white prints, and Type C, or color-photo, prints.

He applied the same perfectionism to filming our family. While everyone else had those dreary vacation slide shows, Louis and Elizabeth Mayo’s shots (notably, of France, Egypt, and sub-Saharan Africa[iii]), in the hands of Lou Mayo, became meticulously orchestrated extravaganzas, precisely set to music. With a narration he had composed on a yellow pad, he would put on the show (accompanied by lavish food). Strange to think that his painstaking effort of many weeks is today’s effort of a few hours in Power Point.

Then there are the trunks and trunks of slides: the family posing, always on precarious ledges, mountain cliffs, the edge of a waterfall . . . waiting, waiting. You couldn’t move, never mind the bug bites, sunsquint, for an interminable interval while light meters were adjusted and readjusted, until it was just right.

Today we don’t have to wait through light-meter adjustments, or worry about soaking the paper in its chemical bath a fraction of a second too long. But we’re just a little nostalgic for that cool darkroom, where, with perfect timing, life shined through glorious Kodak film.

[i] Currently president of Mayo Studios, Inc.

[ii] Kodak invented the first digital camera but did not develop it. Of course they still have numerous, valuable patents.

[iii] Incomplete, but worked on days before he died.

Updating & Downdating: One of the Pieces to Pick up on

2012-01-26T09:14:00.000-05:00

pieces to pick up on (later)

Before moving on to a couple of rather different areas, there's an issue that, while mentioned by both Senn and Gelman, did not come up for discussion; so let me just note it here as one of the pieces to pick up on later.

“It is hard to see what exactly a Bayesian statistician is doing when interacting with a client. There is an initial period in which the subjective beliefs of the client are established. These prior probabilities are taken to be valuable enough to be incorporated in subsequent calculation. However, in subsequent steps the client is not trusted to reason. The reasoning is carried out by the statistician. As an exercise in mathematics it is not superior to showing the client the data, eliciting a posterior distribution and then calculating the prior distribution; as an exercise in inference Bayesian updating does not appear to have greater claims than ‘downdating’ and indeed sometimes this point is made by Bayesians when discussing what their theory implies. (59)…..” Stephen Senn

“As I wrote in 2008, if you could really construct a subjective prior you believe in, why not just look at the data and write down your subjective posterior.” Andrew Gelman commenting on Senn

I've even heard subjective Bayesians concur on essentially this identical point, but I would think that many would take issue with it...no?

Overheard on the Ogosphere

2012-01-25T10:35:00.002-05:00

In response to Christian Robert's remark that the journal name Rationality,Markets and Morals (RMM) is “a rather weird combination, esp. for a journal name!” (in his Jan. 21, 2012 post on Senn’s article in that same journal), a RMM journal editor Max Albert responds:

Max Albert Says: January22, 2012 at 4:08 pm
Dear xi’an,
I am a bit surprised that you consider “Rationality, marketsand morals” as a “rather weird combination”. It is a classical combination oftopics in economics and philosophy. And it seems to me that just now all theworld is talking and writing about it.
Of course, we very much appreciate the fact that you havelooked at our special issue (or special topic, as we call it, since we have noissues). Reading, in addition, some of our other papers may convince you thatour journal title makes sense.
Best wishes

Max Albert

xi'an Says: January22, 2012 at 5:52 pm

Sorry for the pun: for me, markets, rationality, and morals(understood as the French translation into morale)do not seem to get together very well, esp. in the current times! This beingsaid, I appreciate this special topic and do not mean in any way that thejournal does not make sense! Again, sorry for the pun…

I guess Robert’s point isthat he finds current markets irrational, and that it has gotten his moraledown, so I take it Robert didn’t have any shares of AAPL after hours (1/24)! Tant pis (I think it is). DAL surging past $10 today (a PhilStockcall) is greatly raising my morale!

U-PHIL (3): Stephen Senn on Stephen Senn!

2012-01-24T02:24:00.007-05:00

I am grateful to Deborah Mayo for having highlighted my recentpiece. I am not sure that it deserves the attention it is receiving. Deborah has spotted a flaw in my discussion of pragmaticBayesianism. In praising the use of background knowledge I can neither be talkingabout automatic Bayesianism nor about subjective Bayesianism. It is clearthat background knowledge ought not generally to lead to uninformativepriors (whatever they might be) and so is not really what objectiveBayesianism is about. On the other hand all subjective Bayesians care about iscoherence and it is easy to produce examples where Bayesians quite logicallywill react differently to evidence, so what exactly is ‘backgroundknowledge’?.

Nevertheless, if we start at a rather different point, a point atwhich I

think most applied statistics starts, we might end up with a rather

different attitude. The point is to say that about most problems we encounter we have some background experience and it is appropriateto consider this carefully when deciding a) what data to collect and b)how to interpret them.

A favourite example of mine is cross-over trials.You cannot make a sensible analysis of a cross-over trial withoutconsidering carry-over. Standard frequentist approaches are either to assume itdoes not exist or to grant that it might be anything at all and these two extremes lead to startlingly different inferences. In principle, aBayesian has more options and can mix things. However, if (s)he wants to model carry-over claiming to have used background knowledge, then toconvince me that this has merit, I shall have to be shown that the length of the wash-out period compared to the treatment period has been reasonably incorporated into the model as has sensible belief about likelyduration of response based on general background pharmacology and in generalthat carry-over and treatments effects have been modelled as mutuallydependent phenomena. Andy Grieve and I published a paper together on analysing cross-overtrials in 1998, he from a Bayesian and I from a frequentist approach. Icame to the conclusion that I liked his approach far more than what had formany years been assumed to be the proper way to do a frequentist analysisof cross-over trials. So I have quite a friendly attitude to anybody who is prepared to be locally Bayesian and try a recipe of judiciously chosen priordistributions (based on experience) plus suitable likelihood, provided that theytake a suitably realistic and humble attitude to what they have achievedand don’t ram the ‘this is the only way to think’ attitude down my throat.Basically I regard such calculations as acceptable (and in some cases veryuseful) ‘subjective’ contributions to an ongoing objective program oftesting and verification. In particular they can provide attractive ways (in principle!) of dealing with nuisance parameters.

I meant it quite seriously when I said that there was value in being prepared to use all four systems of inference. If I look at my own practice, I use maximum likelihood and significance tests (Fisher), confidence intervals and power calculations (Neyman-Pearson) andBayesian decision analysis (De Finetti, Wald) and find uses for all of these.I am well aware that there are areas in which I could do better. Forexample, I think that most medical statisticians, myself included, have paidfar too much attention to the power approach to sample size determination.We ought to be using approaches based on Bayesian decision theory as well.

The only one of the four systems I don’t use is automatic objectiveBayes, largely because in the way it is currently applied, as far as I cansee, it is pretty much redundant. To take a field I often work in, greatclaims have been made for such approaches to meta-analysis but the choiceof frequentist or Bayesian framework (as most usually applied) seems tomake almost no difference. On the other hand, decisions such as whetherto treat the main effect of trial as fixed or random, whether to allow for arandom trial by treatment interaction and if so whether to model the effectfor the ‘average’ trial or patient are far more important. I don’t exclude, however, that if sensibly applied, Jeffreys’approach could be very useful. The problem is, that so-called Bayesians have enthusiastically embraced one-half of it (‘uninformative’ prior)whilst finding no use for the part that Jeffreys actually considered hisgreatest contribution (significance tests). I have not really tried to usethe combination of these two in the way that Jeffreys himself suggestedand maybe if I did I would be pleasantly surprised. My excuse is that innot trying Bayesian significance tests I am following the practice of byfar the great majority of ‘Bayesians’.

U-PHIL: Stephen Senn (2): Andrew Gelman

2012-01-23T00:13:00.003-05:00

I agree with Senn's comments on the impossibility of the de Finetti subjective Bayesian approach. As I wrote in 2008, if you could really construct a subjective prior you believe in, why not just look at the data and write down your subjective posterior. The immense practical difficulties with any serious system of inference render it absurd to think that it would be possible to just write down a probability distribution to represent uncertainty. I wish, however, that Senn would recognize "my" Bayesian approach (which is also that of John Carlin, Hal Stern, Don Rubin, and, I believe, others). De Finetti is no longer around, but we are!

I have to admit that my own Bayesian views and practices have changed. In particular, I resonate with Senn's point that conventional flat priors miss a lot and that Bayesian inference can work better when real prior information is used. Here I'm not talking about a subjective prior that is meant to express a personal belief but rather a distribution that represents a summary of prior scientific knowledge. Such an expression can only be approximate (as, indeed, assumptions such as logistic regressions, additive treatment effects, and all the rest, are only approximations too), and I agree with Senn that it would be rash to let philosophical foundations be a justification for using Bayesian methods. Rather, my work on the philosophy of statistics is intended to demonstrate how Bayesian inference can fit into a falsificationist philosophy that I am comfortable with on general grounds.

U-PHIL: Stephen Senn (1): C. Robert, A. Jaffe, and Mayo (brief remarks)

2012-01-22T23:41:00.025-05:00

I very much appreciate C. Robert and A. Jaffe sharing some reflections on Stephen Senn’s article for this blog, especially as I have only met these two statisticians recently, at different conferences. My only wish is that they had taken a bit more seriously my request to “hold (a portion of) the text at ‘arm’s length,’ as it were. Cycle around it, slowly. Give it a generous interpretation, then cycle around it again self-critically” (January 13, 2011). (I conceded it would feel foreign, but I strongly recommend it!)

Since these authors have given bloglinks, I’ll just note them here and give a few brief responses:

Christian Robert

http://xianblog.wordpress.com/2012/01/21/may-i-believe-i-am-a-bayesian/

Mayo’s brief remarks:

As I see it, Robert overlooks the most difficult challenge Senn raises--namely that in practice, people who claim to have carried out a (subjective) Bayesian analysis have actually done something very different—but that then they heap credit on the Bayesian ideal (what I called the “grace and amen” routine). He instead attempts to take the convenient escape route I warned of in my post on Senn (Jan 15 [i]); namely to insist that they are still approximating the subjective Bayesian way, as if, examples to the contrary, Senn is merely pointing up some minor imperfections rather than a relinquishing of what are still regarded as core principles of subjective Bayesianism.

For instance, when default Bayesian admit to violating long-held principles, they seem not merely to be making concessions to human imperfections but rather denying fundamental principles and assumptions still regarded as integral to subjective Bayesianism. e.g., the likelihood principle, Dutch Book arguments, and even that inductive inference follows Bayes’s theorem. These foundational problems demand a greater gesture than a minor compromise or admission that no one's perfect!

Robert’s claim that Senn “freezes the (Bayesian reasoning about the) Bayesian paradigm in its de Finetti phase-state” equally cuts no ice; Senn’s examples are taken from recent Bayesian work, and the onus is on Robert, a subjective Bayesian, to show that Senn’s criticisms do not stand.

As I keep asking, doesn't what is actually doing the work deserve its own epistemological grounding? I say it does.

______________________________________

Andrew Jaffe

http://www.andrewjaffe.net/blog/science/000524.html

Mayo’s brief remarks:

Andrew Jaffe protests, ‘no no Senn doesn’t understand how we really function in practice…it’s not at all like that’ (“I think that these criticisms mis-state the practice of Bayesian statistics, at least by the scientists I know (mostly cosmologists and astronomers).” But that’s precisely what Senn does understand and why he faults the Bayesians for thanking the subjective Bayesian paradigmfor what we (readers) are about to receive, alleging that they owe it all to Bayes. No credit properly goes back to Bayesian ways if they are not responsible for the touted results.

Says Jaffe, “Rather, most of us take a vaguely Jaynesian view, after the cranky Edwin Jaynes” Jaffe says. Well I agree that Jaynes was cranky. This is the objective Bayesian I cited in an earlier post (Dec. 6, 2011) as declaring (cantankerously) that outcomes other than the one observed can’t matter to inference:

“The question of how often a given situation would arise is utterly irrelevant to the question how we should reason when it does arise. I don’t know how many times this simple fact will have to be pointed out before statisticians of ‘frequentist” persuasions will take note of it.” (Jaynes 1976, 247)

To which I replied:

“What we wonder is how many times we will have to point out that to us, reasoning from the result that arose is crucially dependent on how often it would have arisen…..”

I just don’t see how a subjective Bayesian can be at all comforted by Jaffe's reference to a free and easy spirit type of Bayes, much less when he adds: “This is a point of view espoused most forcefully by Andrew Gelman” given that Gelman has pretty clearly denounced the very idea of doing inductive inference by way of Bayes’s theorem.[ii]
__________________________________

[i] I had written, “But now that nearly no Bayesians explicitly advocate the one true subjective Bayesian ideal, more is needed. Their position has shifted. While adhering to the BADD ideal, they will still describe their methods as mere approximations of that ideal. After all, they will (and do) claim they can’t be perfect,but the Bayesian ideal still lights the way, and therefore discredits all Senn's-ible criticism of their claim that all you need is Bayes.”

[ii] I now have Gelman’s to add, on the next post.

METABLOG QUERY

2012-01-21T13:48:00.003-05:00

Can anybody advise about the relative merits of Wordpress versus Google blogspot for blogging? The Elbians worked to transfer this entire blog to Wordpress which I take it is superior and might avoid a lot of the out-of-control fonts on this blog, but whenever I go over to that blog I have misgivings because, among other things, it's always asking me to pay to avoid annoyances like ads, which never even came up as an issue on blogspot. Moreover, as soon as I paid for one upgrade (just to explore it), 6 other choices of things to buy came up which made me very nervous, since it might mean having to understand what in the world they are selling (and I don't). So I just ran away. The decision about switching remains in limbo. It's not paying that I mind in the least, it's being confronted with a lot of confusing decisions that I'm worried about. I haven't paid for anything on blogspot, and the informality is appealing, perhaps because it makes the whole thing less official, and lets me feel that I am free to escape from all this at any time. I'd be grateful for advice and recommendations; I can't ask the blogsfolk to keep up the shadow blog indefinitely, while I decide. Maybe there's yet a third blogging platform....

PhilStock Blog: "Year of the Yo Yo"

2012-01-19T21:43:00.013-05:00

A couple of months ago I was about to start an "other blog" (as Google calls it), this one on philosophical reflections on the stock market (PhilStock), but as I cannot even keep up one blog, I wisely discarded that idea after a single post (on high speed trading). So if I dip into that area on this blog, I will warn readers with the “Phil Stock Blog” tag[i]. First and last rule on PhilStock: Never listen to anything I say about the stock market.

It’s a bizarre kind of comfort to see that stock analysts are much less inclined to tout their skills ever since the crash (of 08-09), admitting that, at least with today’s crazy market, performance is more “the result of luck rather than skill”. Take the Financial Page of the latest New Yorker (“Year of the Yo-Yo”, Jan. 16, 2010):

“As myriad studies have shown, investors often put their money into funds that have enjoyed recent success and take it out of funds that have been struggling. This seems logical, but, since most of a money manager’s performance, particularly in the short term, is the result of luck rather than of skill, this means that people often end up in funds that are about to go cold and leave ones that are about to do better. …

Hedge-fund investors, though supposedly more sophisticated than your average Joe, pay a similar price for chasing performance. Investing in actively managed mutual funds or in hedge funds already reduces the chances of beating the market, since, according to Vanguard, over the past decade more than sixty per cent of actively managed mutual funds underperformed the S. & P. , while hedge funds have trailed the market since 2003. But the search for the hot hand takes a bad situation and makes it worse.

….Unfortunately, the same psychological forces that make investors bad at rating money managers also make them bad are market timing: all else being equal, they’re prone to sell at the bottom and buy at the top. And, the bigger and more dramatic the swings in the market, the more likely we are to make the wrong decision.”

I guess this justifies my holding on to some stocks which I regard as “rather promising losers”; notably, DO (Diamond Offshore), finally back over $60--the deep water driller that was integral to beginning this blog[ii]; DAL (Delta) over $9 for a change[iii]; STP (SunTech Power)---down around 3.20, just lost 50 cents today. If I was to predict risky targets for the year, maybe $75, $13, and $5-6, respectively.) I’m also holding on to FTR (Frontier) with its 15% dividend, now $5, predict $6.50? Remember, though: Never ever listen to anything I say about the stock market.

[i] Perhaps if we ever get a handle on this blog, it can be a separate “page”.

[ii] Note: it has a high special dividend, not shown with the usual dividend.

[iii] It’s been in the $7 range for quite awhile; had been $14.

RMM-6: Special Volume on Stat Sci Meets Phil Sci

2012-01-19T13:03:00.001-05:00

The article "The Renegade Subjectivist: José Bernardo’s Reference Bayesianism" by Jan Sprenger has now been published in our special volume of the on-line journal, Rationality, Markets, and Morals (Special Topic: Statistical Science and Philosophy of Science: Where Do/Should They Meet?")

Abstract: This article motivates and discusses José Bernardo’s attempt to reconcile the subjective Bayesian framework with a need for objective scientific inference, leading to a special kind of objective Bayesianism, namely reference Bayesianism. We elucidate principal ideas and foundational implications of Bernardo’s approach, with particular attention to the classical problem of testing a precise null hypothesis against an unspecified alternative.

"Philosophy of Statistics": Nelder on Lindley

2012-01-17T23:42:00.008-05:00

A friend from Elba surprised me by sending the interesting paper and discussion of Dennis Lindley (2000), “The Philosophy of Statistics,” which I hadn’t seen in years. She suggested, as especially apt, J. Nelder’s remarks; I recommend the full article and discussion:

(from) Comments by J. Nelder:

Recently (Nelder,1999) I have argued that statistics should be called statistical science, and that probability theory should be called statistical mathematics (not mathematical statistics). I think that Professor Lindley's paper should be called the philosophy of statistical mathematics, and within it there is little that I disagree with. However, my interest is in the philosophy of statistical science, which I regard as different. Statistical science is not just about the study of uncertainty but rather deals with inferences about scientific theories from uncertain data. An important quality about theories is that they are essentially open ended; at anytime someone may come along and produce a new theory outside the current set. This contrasts with probability where to calculate a specific probability it is necessary to have a bounded universe of possibilities over which the probabilities are defined. When there is intrinsic open-endedness it is not enough to have a residual class of all the theories that I have not thought of yet. The best that we can do is to express relative likelihoods of different parameter values, without any implication that one of them is true. Although Lindley stresses that probabilities are conditional I do not think that this copes with the open-endedness problem. I follow Fisher in distinguishing between inferences about specific events, such as that it will rain here tomorrow and inferences about theories. .…

In analyzing data relative to one or more scientific theories, I would wish to present what is objective and not to mix this with subjective probabilities which are derived from my priors. If the experimenter whom I am working with wishes to combine likelihoods with his own set of weights based on his (doubtless more extensive) knowledge then he is at liberty to do so; it is not my job to do it for him. However, if he wishes to communicate the results to other scientists, it would be better, in my view, to stay with the objective part. (This paragraph is heavily dependent on ideas of George Barnard.)

General ideas like exchangeability and coherence are fine in themselves, but problems arise when we try to apply them to data from the real world. In particular when combining information from several data sets we can assume exchangeability, but the data themselves may strongly suggest that this assumption is not true. Similarly we can be coherent and wrong, because the world is not as assumed by Lindley. I find the procedures of scientific inference to be more complex than those defined in the paper. These latter fall into the class of ‘wouldn’t it be nice if’, i.e. would it not be nice if the philosophy of statistical mathematics sufficed for scientific inference. I do not think that it does. (325)

Lindley, D. V. (2000), “The Philosophy of Statistics,” Journal of the Royal Statistical Society, Series D (The Statistician), Vol. 49, No. 3, 293-337
Nelder, J.A. (2000), Commentary on “The Philosophy of Statistics,” Journal of the Royal Statistical Society, Series D (The Statistician), Vol. 49, No. 3, 324-5.
Nelder, J.A. (1999) “From Statistics to Statistical Science” Statistician 48, 257-267.

Mayo Philosophizes on Stephen Senn: "How Can We Cultivate Senn’s-Ability?"

2012-01-15T13:12:00.022-05:00

Where's Mayo?

Although, in one sense, Senn’s remarks echo the passage of Jim Berger’s that we deconstructed a few weeks ago, Senn at the same time seems to reach an opposite conclusion. He points out how, in practice, people who claim to have carried out a (subjective) Bayesian analysis have actually done something very different—but that then they heap credit on the Bayesian ideal. (See also the blog post “Who Is Doing the Work?”)

"A very standard form of argument I do object to is the one frequently encountered in many applied Bayesian papers where the first paragraphs laud the Bayesian approach on various grounds, in particular its ability to synthesize all sources of information, and in the rest of the paper the authors assume that because they have used the Bayesian machinery of prior distributions and Bayes theorem they have therefore done a good analysis. It is this sort of author who believes that he or she is Bayesian but in practice is wrong". (58)

Why in practice is this wrong? For starters, Senn points out, the analysis seems to violate such strictures as temporal coherence:

"Attempts to explain away the requirement of temporal coherence always seem to require an appeal to a deeper order of things—a level at which inference really takes place that absolves one of the necessity of doing it properly at the level of Bayesian calculation". (ibid.)

So even if they come out with sensible analyses, Senn is saying, it is despite rather than because they followed strict Bayesian rules and requirements. It is thanks to certain unconscious interventions, never made explicit, and perhaps not even noticed by the Bayesian reasoner. “This is problematic,” Senn thinks, “because it means that the informal has to come to the rescue of the formal.” Not that there is anything wrong with informality . . .

"Indeed, I think it is inescapable. I am criticising claims to have found the perfect system of inference as some form of higher logic because the claim looks rather foolish if the only thing that can rescue it from producing silly results is the operation of the subconscious". (59)

Now, many Bayesians would concede to Senn that in arriving at their outputs they violate strict norms laid down by De Finetti or other subjective Bayesians. But why then do they credit these outputs to some kind of philosophical Bayesianism? The answer, I take Senn to be suggesting, is the fact that they assume that there is but one philosophically righteous position—that of being a Bayesian deep down, where "Bayesian deep down" alludes to a fundamental subjective Bayesian position.

Senn's idea may be that their belief in Bayesianism deep down is a priori, so it's little wonder that no empirical facts can shatter their standpoint. (The very definition of an a priori claim is that it's not open to empirical appraisal.) I think this is generally the case. Many have simply been taught the Bayesian catechism—that subjective Bayesianism is at the foundation of all adequate statistical analyses, and offers the only way to capture uncertainty. Others are true-blue believers (not only in the Bayesian ideal but in the frequentist howlers regularly trotted out) . Either way, one can understand why so many Bayesian articles follow the pattern Senn describes: begin by saying grace and end by thanking the Bayesian account for its offer to house all their uncertainties within prior probability distributions, even if in between, the analysis immediately turns to non-Bayesian means that can more ably grapple with both the limits and the goals of the actual inquiry.

Yet Senn, as I understand him, finds this Bayesian “grace and amen routine"—my term not his—disingenuous and utterly insufficient as a foundation for statistical research. We ought to be able to look into the black box and recognize that the methods used scarcely toe the (subjective) Bayesian line, or so Senn seems to be saying:

"In a paper published in Statistics in Medicine in 2005 Lambert et al. considered thirteen different Bayesian approaches to the estimation of the so-called random effects variance in meta-analysis. . . .

The paper begins with a section in which the authors make various introductory statements about Bayesian inference. For example, “In addition to the philosophical advantages of the Bayesian approach, the use of these methods has led to increasingly complex, but realistic, models being fitted,” and “an advantage of the Bayesian approach is that the uncertainty in all parameter estimates is taken into account” (Lambert et al. 2005, 2402), but whereas one can neither deny that more complex models are being fitted than had been the case until fairly recently, nor that the sort of investigations presented in this paper are of interest, these claims are clearly misleading in at least two respects. (Senn 2011, 62)

First, the “philosophical” advantages to which the authors refer must surely be to the subjective Bayesian approach outlined above, yet what the paper considers is no such thing. None of the thirteen prior distributions considered can possibly reflect what the authors believe about the random effect variance.[i] Second, the degree of uncertainty must be determined by the degree of certainty and certainty has to be a matter of belief so that it is hard to see how prior distributions that do not incorporate what one believes can be adequate for the purpose of reflecting certainty and uncertainty". (62-3)

Now let's compare this with Jim Berger. Berger, I take it, holds to philosophical Bayesianism, while granting that, in practice, we need conventional priors that are not claimed to be expressions of uncertainty or degree of belief (see also Dec 19, Dec 26, Jan 3). Senn's second point says to Berger that, in that case, one cannot claim that the Bayesian analysis reflects uncertainty or degree of belief (be it actual or rational). But one who holds to Bayesianism Deep Down (DD?) can appeal to the position we crafted to resolve the paradox in Berger’s notion that the use of conventional priors is a way of becoming more subjective: Since being a philosophical Bayesian DD (BADD?) is assumed (a priori), and since replacing "terrible" priors with default priors is deemed an improvement, it must therefore be closer to the subjective Bayesian ideal.

Although Senn at times seems almost to grant that subjective Bayesianism is perfect in theory (or he at least admits to having a love-hate relationship with it), he's clearly “criticising the claim that it is the only system of inference and in particular I am criticising the claim that because it is perfect in theory it must be the right thing to use in practice” (59).[ii]

Despite these occasional whiffs of being (BADD), Senn's critique would seem to locate him outside the Bayesian (and perhaps any other) formal paradigm. Yet why suppose that this “metastatistical standpoint” admits of no general, non-trivial, empirical standards and principles? It seems to me that one should not suppose this, but instead try and unearth these general arguments, however "informal" or "quasi-formal" they may be. Moreover, I will argue that unless we do so, a Senn-style position here in praise of eclecticism fail at its intended aim.

Noting that another Bayesian paper a few years later effectively concedes his point, Senn remarks:

"This latter paper by the by is also a fine contribution to practical data-analysis but it is not, despite the claim in the abstract, "We conclude that the Bayesian approach has the advantage of naturally allowing for full uncertainty, especially for prediction," a Bayesian analysis in the De Finetti sense. Consider, for example this statement, "An effective number of degrees of freedom for such a t-distribution is difficult to determine, since it depends on the extent of the heterogeneity and the sizes of the within-study standard errors as well as the number of studies in the meta-analysis." This may or may not be a reasonable practical approach but it is certainly not Bayesian". (63)

Here, as elsewhere, Senn seems to have no trouble regarding the work as “a fine contribution” to statistical analysis, but one wonders: what criteria is he using to approve it? Is he content to leave those criteria at the unconscious level without making them explicit? If so, isn’t he open to the same kinds of subliminal appraisals made by the Bayesians he takes to task? Can we not learn the basis for Senn’s sensibility (senn’s-ibility?)? Does he think that the standards he uses for critically appraising, interpreting, and using statistical methods are ephemeral? Can we say nothing more than that they shouldn't be too terribly awful on any of the four strands of statistical methodology? Senn takes the Bayesian to task for showing us only how to be perfect, but not how to be "good." Let's move on to this.

To make this more concrete: How, specifically, would Senn have those authors describe what they actually did, given that it's “certainly not Bayesian”? Now, Senn is not really crediting any overarching or underlying philosophical standpoint for his expertise—but shouldn’t he? Is the choice between adopting an a priori standpoint and adopting eclecticism “all the way down”—even at the level of critically appraising, interpreting, and using statistical methods? If, as Senn himself suggests, most of the Bayesians writing the papers he takes to task are doing what they do more or less unconsciously, then how will he raise their consciousness? Saying it's not really Bayesian doesn't quite tell them what it is.

One might question my presumption that there are some overarching standards, principles, or criteria used in judging work from different schools. But we should at least try to articulate them before assuming it’s not possible. And anyway, Senn’s remarks suggest he is senn-sitive to applying a “second-order” scrutiny.

The account would be far more complex than the neat and tidy accounts often sought: ranging from determining what one wants to learn, breaking it up into piecemeal questions, collecting, modeling, interpreting data and feeding results from one stage into others. Nevertheless, I have suggested there are overarching criteria and patterns of inference (based on identifying the error or threat at the particular stage). (See Nov. 5, post).

To conclude these remarks, then, I want to laud Senn for courageously calling attention to the widespread practice of erroneously describing research as Bayesian, as well as to the tendency of a priori adulation of philosophical Bayesianism Deep Down.** But now that nearly no Bayesians explicitly advocate the one true subjective Bayesian ideal, more is needed. Their position has shifted. While adhering to the BADD ideal, they will still describe their methods as mere approximations of that ideal. After all, they will (and do)say, they can’t be perfect, but the Bayesian ideal still lights the way, and therefore discredits all Senn's-ible criticism of their claim that all you need is Bayes.

Unless Senn identifies the non-Bayesian work in-between the “grace and amen” Bayesianism, the worry (my worry) is that there will be no obligation to amend this practice. Nor is it enough, it seems to me, to merely point out that they are using tools from standard frequentist schools, since these can always be reinterpreted Bayesianly—or so they will say. If it’s just a name game, the new-styled Bayesians can say, as some already do about their favorite methods, “I dub thee Bayesian”—since “Bayesian” is in the title of my book, or since a conditional probability is used somewhere. That's the challenge I am posing to those who would advance the current state of statistical foundations.

**I accord Stephen Senn an Honorable Mention (the 2nd awarded so far).

Higgins J. P., S. Thompson and D. Spiegelhalter (2008), “A Re-evaluation of Random effects Meta-analysis”, Journal of the Royal Statistical Society, Series A 172, 137–159.

Lambert, P. C., A. J. Sutton, P. R. Burton, K. R. Abrams and D. R. Jones (2005), “How Vague is Vague? A Simulation Study of the Impact of the Use of Vague Prior Distributions in MCMC Using WinBUGS”, Statistics in Medicine 24, 2401–2428.

Senn, S. (2011), “You May Believe You Are a Bayesian But You Are Probably Wrong” (RMM) Vol. 2, 2011, 48–66.

OK, now U-PHIL! (e-mail error@vt.edu or in comments, by 1/22)

[i] He continues: “One problem, which seems to be common to all thirteen prior distributions, is that they are determined independently of belief about the treatment effect. This is unreasonable since large variation in the treatment effect is much more likely if the treatment effect is large” (Senn 2007b).

[ii] In at least one place Senn slips into the tendency to equate the use of background knowledge to being Bayesian in a subjective sense: Senn declares that a frequentist statistician who chose to set a carry-over effect to zero, in a clinical trial where it fairly obviously warranted being ignored, “would be being more Bayesian in the De Finetti sense than one who used conventional uninformative prior distributions or even Bayes' factor” (p. 62). (See, in this connection, the discussion in Cox and Mayo [also RMM 2011] on the use of background knowledge.) But there is no evidence that this background knowledge was or needs to be translated into a prior probability distribution.

“You May Believe You Are a Bayesian But You Are Probably Wrong”

2012-01-14T14:25:00.022-05:00

The following is an extract (58-63) from the contribution by

Stephen Senn (Full article)

Head of the Methodology and Statistics Group,

Competence Center for Methodology and Statistics (CCMS), Luxembourg

.......

I am not arguing that the subjective Bayesian approach is not a good one to use. I am claiming instead that the argument is false that because some ideal form of this approach to reasoning seems excellent in theory it therefore follows that in practice using this and only this approach to reasoning is the right thing to do. A very standard form of argument I do object to is the one frequently encountered in many applied Bayesian papers where the first paragraphs lauds the Bayesian approach on various grounds, in particular its ability to synthesize all sources of information, and in the rest of the paper the authors assume that because they have used the Bayesian machinery of prior distributions and Bayes theorem they have therefore done a good analysis. It is this sort of author who believes that he or she is Bayesian but in practice is wrong. (58)

3. Reasons for Hesitation

The first of these is temporal coherence. De Finetti was adamant that it is not the world’s time, in a sense of the march of events (or the history of ‘one damn thing after another’), that governs rational decision making but the mind’s time, that is to say the order in which thoughts occur or evidence arises. However, I do think that he believed there was no going back. You struck out the sequences of thought-events that had not occurred in your mind and renormalized. The discipline involved is so stringent that most Bayesians seem to agree that it is intolerable and there have been various attempts to show that Bayesian inference really doesn’t mean this. I am unconvinced. I think that de Finetti’s theory really does mean this and the consequence is that the phrase ‘back to the drawing board’ is not allowed. Attempts to explain away the requirement of temporal coherence always seem to require an appeal to a deeper order of things—a level at which inference really takes place that absolves one of the necessity of doing it properly at the level of Bayesian calculation. This is problematic, because it means that the informal has to come to the rescue of the formal. …(58)

I am not criticizing informal inference. Indeed, I think it is inescapable. I am criticising claims to have found the perfect system of inference as some form of higher logic because the claim looks rather foolish if the only thing that can rescue it from producing silly results is the operation of the subconscious. Nor am I criticising subjective Bayesianism as a practical tool of inference. As mentioned above, I am criticising the claim that it is the only system of inference and in particular I am criticising the claim that because it is perfect in theory it must be the right thing to use in practice. (59)

It is hard to see what exactly a Bayesian statistician is doing when interacting with a client. There is an initial period in which the subjective beliefs of the client are established. These prior probabilities are taken to be valuable enough to be incorporated in subsequent calculation. However, in subsequent steps the client is not trusted to reason. The reasoning is carried out by the statistician. As an exercise in mathematics it is not superior to showing the client the data, eliciting a posterior distribution and then calculating the prior distribution; as an exercise in inference Bayesian updating does not appear to have greater claims than ‘downdating’ and indeed sometimes this point is made by Bayesians when discussing what their theory implies. (59)…..

In a paper published in Statistics in Medicine in 2005 Lambert et al. considered thirteen different Bayesian approaches to the estimation of the so-called random effects variance in meta-analysis. …..

The paper begins with a section in which the authors make various introductory statements about Bayesian inference. For example, “In addition to the philosophical advantages of the Bayesian approach, the use of these methods has led to increasingly complex, but realistic, models being fitted” and, “an advantage of the Bayesian approach is that the uncertainty in all parameter estimates is taken into account” (Lambert et al. 2005, 2402) but whereas one can neither deny that more complex models are being fitted than had been the case until fairly recently, nor that the sort of investigations presented in this paper are of interest, these claims are clearly misleading in at least two respects. (62)

First, the ‘philosophical’ advantages to which the authors refer must surely be to the subjective Bayesian approach outlined above, yet what the paper considers is no such thing. None of the thirteen prior distributions considered can possibly reflect what the authors believe about the random effect variance. One problem, which seems to be common to all thirteen prior distributions, is that they are determined independently of belief about the treatment effect. This is unreasonable since large variation in the treatment effect is much more likely if the treatment effect is large (Senn 2007b). Second, the degree of uncertainty must be determined by the degree of certainty and certainty has to be a matter of belief so that it is hard to see how prior distributions that do not incorporate what one believes can be adequate for the purpose of reflecting certainty and uncertainty. (62-3)

Certainly, another Bayesian paper on meta-analysis only a few years later (Higgins et al. 2008) agreed implicitly with this, … This latter paper by the by is also a fine contribution to practical data-analysis but it is not, despite the claim in the abstract, “We conclude that the Bayesian approach has the advantage of naturally allowing for full uncertainty, especially for prediction”, a Bayesian analysis in the De Finetti sense. Consider, for example this statement, “An effective number of degrees of freedom for such a t-distribution is difficult to determine, since it depends on the extent of the heterogeneity and the sizes of the within-study standard errors as well as the number of studies in the meta-analysis.” This may or may not be a reasonable practical approach but it is certainly not Bayesian. (63)

There are two acid tests. The first is that the method must be capable of providing meta-analytic results when there is only one trial. That is to say the want of data must be made good by subjective probability. The practical problem, of course, is that you cannot estimate the way in which the results vary from trial to trial unless you have at least two trials (in fact, in practice more are needed). But to concede this causes a problem for any committed Bayesian. (63)

The second test is that whereas the arrival of new data will, of course, require you to update your prior distribution to being a posterior distribution, no conceivable possible constellation of results can cause you to wish to change your prior distribution. If it does, you had the wrong prior distribution and this prior distribution would therefore have been wrong even for cases that did not leave you wishing to change it. This means, for example, that model checking is not allowed. (63)

I invite your philosophical reflections, (please see "U Phil" post), in "comments" or send to error@vt.edu (by 1/22)

I will post mine tomorrow, Senn's and "yours" next week

de Finetti, B. D. (1974), Theory of Probability (Volume 1), Chichester: Wiley.

— (1975), Theory of Probability (Volume 2), Chichester: Wiley.

Higgins J. P., S. Thompson and D. Spiegelhalter (2008), “A Re-evaluation of Random effects Meta-analysis”, Journal of the Royal Statistical Society, Series A 172, 137–159.

Senn, S. (2007b), “Trying to Be Precise about Vagueness”, Statistics in Medicine 26, 1417–1430.

Senn, S. (2011), “You May Believe You Are a Bayesian But You Are Probably Wrong” Rationality, Markets and Morals (RMM) Vol. 2, 2011, 48–66.

U-PHIL: "So you want to do a philosophical analysis?"

2012-01-13T18:31:00.003-05:00

"Philosophy, as I have so far understood and lived it, means living voluntarily among ice and high mountains—seeking out everything strange and questionable in existence". Nietzsche*

I am about to turn to philosophical analyses/deconstructions of short portions of articles from the special issue "Statistical Science and Philosophy of Science" (RMM 2011), and I will invite contributed analyses from readers and, of course, the author(s). The first text, to be posted tomorrow, will be from Professor Stephen Senn. (Full article)

I will post some (after a week or so) as with the last “deconstruction” exercise.

Since we’ve been at this for a while, I’m assuming that you’re anxious to try out your philosophical muscles. So, to begin, hold (a portion of) the text at “arm’s length,” as it were. Cycle around it, slowly. Give it a generous interpretation, then cycle around it again self-critically. Try and challenge the text but also your initial impressions, give reasons, maybe include some pondering "aloud" of what you wrote.

Stop there. Or, try to imaginatively consider how to push the problem/argument a bit further: Which of the assumptions are shaky? What are the gaps? How would it advance the author’s goal to fill them?

If you don’t typically write philosophy, then this little exercise should produce something that feels a bit strange, even foreign. But it should be fun, and Lou Salome is saying that it would be good for you.

*Contest: Nietzsche readers will immediately find the single most glaring error in this famous (posed) picture. What is it?

Bad-Faith Assertions of Conflicts of Interest?*

2012-01-10T18:19:00.007-05:00

In response to an indication that the FDA might need to loosen conflict-of-interest (COI) rules to get sufficient experts to serve on their advisory panels, a list has been proferred of “industry-free” experts capable of serving with “clean hands” (See Oct 10 post: Junk Science ) But why not also seek “litigation-free” experts, asks lawyer, Nathan Schachtman on his interesting blog (Dec. 28) The Continuing Saga of Bad-Faith Assertions of Conflicts of Interest:

Conflicts of interest (COI), real or potential, have become a weapon used to silence the manufacturing industry in various scientific debates and discussions. Other equally “interested” parties, labor unions, advocacy groups, and consultants to the other industry – the litigation industry – have used conflicts and ethical claims to silence the manufacturing industry and to engage in unfettered false scientific speech. The public, unwilling and untrained to look at evidence on the merits, is conditioned to accepting an allegation of COI as the end of the discussion on scientific issues.

Recently, journalist Shannon Brownlee … “Is There an Independent Unbiased Expert in the House” (Aug. 3, 2011)…sent FDA Commissioner Margaret Hamburg a list of allegedly neutral experts who could advise the agency. Brownlee gave everyone on her list a clean bill of ethical health, and has published the list on multiple occasions, …. What the gullible may not appreciate is that the list fallaciously is based upon only one exclusionary criterion: having consulted for the pharmaceutical industry. The list omits other important COI exclusionary criteria, such as having consulted for the litigation industry, or having taken erroneous, unwarranted, and ideologically driven positions on scientific issues.

What litigation industry? Brownlee may have missed the fact that plaintiffs’ lawyers represent a huge financial interest in obtaining compensation for others, with 40 percent of the proceeds going to themselves. This litigation industry thrives, even with Dickie Scruggs in prison, and Stanley Chesley in disrepute.

In today’s litigation environment, with aggregation of claims in federal multi-district cases, plaintiffs’ counsel stand to profit in the billions from scientific positions espoused by their expert witnesses.

…..Now some people may claim that the litigation industry consultants, and the anti-industry zealots, take their positions not to please their sponsors, or to pursue lucrative opportunity, but because they fervently believe the positions that they take. But then why not give the pharmaceutical industry consultants the same benefit of the doubt? Indeed, why not move beyond COI allegations to creating lists of scientists and physicians who have demonstrated proficiency in advancing evidence-based judgments that have withstood the test of time? This anti-industry hypocrisy manifests not only in assertions of conflicts of interest, but also in calls for industry to disclose all underlying data from industry-funded or sponsored studies, while taking a protectionist stance on all other underlying data.

For his full article see his Dec 28 post.

There is indeed a tendency to regard suspicions of industry consultants as altogether justified while downplaying or ignoring possible biases of “anti-industry zealots”. Schachtman regards this as unfair “anti-industry hypocrisy”, and he may well be correct. Thoughts?

*BLOG-CROSSING! I just noticed Schachtman discusses some earlier posts of my blog on his blog today with a post called, "The Will to Ummph"!

Don't Birnbaumize that Experiment my Friend*

2012-01-08T09:32:00.003-05:00

(A) "It is not uncommon to see statistics texts argue that in frequentist theory one is faced with the following dilemma: either to deny the appropriateness of conditioning on the precision of the tool chosen by the toss of a coin[i], or else to embrace the strong likelihood principle which entails that frequentist sampling distributions are irrelevant to inference once the data are obtained. This is a false dilemma … The 'dilemma' argument is therefore an illusion". (Cox and Mayo 2010, p. 298)

The “illusion” stems from the sleight of hand I have been explaining in the Birnbaum argument—it starts with Birnbaumization.
(B) A reader wrote in that he awaits approval of my argument by either Sir David Cox or Christian Robert ; I cannot vouchsafe for Robert, unless he has revised his first impression in his October 6, 2011 blog (as I hope he has). For in that blog post Robert says

“If Mayo’s frequentist stance leads her to take the sampling distribution into account at all times, this is fine within her framework. But I do not see how this argument contributes to invalidate Birnbaum’s proof.”

I am taking sampling distributions into account because Birnbaum’s “proof” is supposed to be relevant for a sampling theorist! If it is not relevant for a sampling theorist (my error statistician) then there is no breakthrough and there is no special interest in the result (given that Bayesians already have the LP, as do the likelihoodists.)[ii] It is only because principles that are already part of the sampling theorist’s steady diet are alleged to entail the LP (in Birbaum's argument) that Savage declared that, once made aware of Birnbaum’s result, he doubted people would stop at the LP appetizer, but would instead go all the way to consuming the full Bayesian omelet!

Robert’s remark is just the tip of the iceberg that reveals a deep misunderstanding of sampling theory. (Although I prefer error statistics, I will use sampling theory for this post.) Even if Robert has corrected himself, as I very much hope he has, other readers may be under the same illusion. I had paused to clarify this point in my October 20, 2011 post.

_________________________________
(C) Likelihood Principle Violations

My Oct. 20 post was devoted to arguing that it is impossible to understand the whole issue without understanding how it is that frequentist sampling theory violates the LP. That it does so is not a point of controversy, so far as I know:

As Lindley (1971) stresses:

“.. sampling distributions, significance levels, power, all depend on something more [than the likelihood function]–something that is irrelevant in Bayesian inference–namely the sample space” (Lindley p. 436).

He means, once the data are known the sample space is irrelevant for appraisal. (The LP already assumes the statistical model underlying the likelihood is given or not in question.) Or, more recently, take Kadane 2011:

“Significance testing violates the Likelihood Principle, which states that, having observed the data, inference must rely only on what happened, and not on what might have happened but did not. The Bayesian methods explored in this book obey this principle” (Kadane, 439).

“Like their testing cousins, confidence intervals and sets violate the likelihood principle” (ibid. 441).

So it’s hard to see how Robert can really mean to say that sampling distribution considerations are irrelevant, when they are the heart and centerpiece of the Birnbaum argument. Far from being irrelevant, Birnbaum’s result is all about sampling distributions (even if addressed by someone who is not herself a sampling theorist!)

(D) Now to consider what Robert says in his post, with my remarks following:

Robert: “The core of Birnbaum’s proof is relatively simple: given two experiments E’ and E” about the same parameter θ with different sampling distributions f¹ and f², such that there exists a pair of outcomes (y’, y”) from those experiments with proportional likelihoods, one considers the mixture experiment where E’ and E” are each chosen with probability ½.

Then it is possible to build a sufficient statistic T that is equal to the data (j,z), except when j=2 and z=y”, in which case T(j,z)=(1,y’).”

Mayo: Put more informally, if y’ and y” is any LP violation pair (i.e., the two would yield different inferences/assessments of the evidence due to the difference in sampling distributions), then it is possible to “build” a statistic T for interpreting them such that y” (from E”) is always reported as y’ from E’.[iii] I called this Birnbaum’s statistic T-BB.[iv] It is possible, in short, to Birnbaumize the result (E’, y’) whenever there is an experiment E”, not performed, that could have resulted in y”, with a proportional likelihood (with the same parameter under investigation and the model assumptions granted).

Robert: “This statistic [T-BB] is sufficient”.

Mayo: Yes, T-BB is sufficient for an experiment that will report its inference based on the rules of Birnbaumization: The sampling distribution of T-BB is to be the convex combination of the sampling distributions of E’ and E” whenever confronted with an outcome that has an LP violation pair (for more details see posts from Dec. 6, 7, and references within).[v] Cox rightly questions even this first step, but I’m prepared to play along since the “proof” breaks down anyway.[vi]

It should be emphasized that in carrying out this Birnbaumization, one is not free from considering the accompanying sampling distribution (corresponding to the statistic T-BB just “built”): the Birnbaumization move depends on having a single sampling distribution (otherwise sufficiency would not apply)[vii].

While Robert switches our Infr_E(z) notation (Cox and Mayo 2010) to Birnbaum’s Ev(E, z), I will go ahead and leave it as Ev. Infr_E was deliberately designed to be clearer, easier to read, and less likely to hide the very equivocation that is overlooked in this example.

Robert observes:

Whether j = 1 or j = 2, Ev(E-BB, (j, z)) = Ev(E-BB, T(j,z))

This corresponds to my premise (1):
(1) Infr_E-BB(E’, y’) = Infr_E-BB(E”, y”)

In the relevant case, y’ and y” are LP violation pairs, since only those pose the threat to obeying the LP. So we can focus just on those in this note. In Mayo 2010 I used the * to indicate an outcome is part of an LP violation pair.
_____________________
(E) Next Robert gives premise (2), though he switches the order: this corresponds to two applications of weak conditionality (WCP) [combining my 2a and 2b]:

(2) Whether j = 1 or j = 2, Ev(E-BB, (j, z)) = Ev(E^j, z)

The key issue concerns a quote from me (with Robert’s substitutions of Ev for Infr). Note, by the way, that Robert is alluding to my chapter in Mayo 2010, not the short version that I posted on this blog, Dec 6, 7

Robert: "Now, Mayo argues this is wrong because [it asserts that]:

'[the mixed experiment E-BB] is appropriately identified with an inference from outcome y^j based on the sampling distribution of E^j, which is clearly false'".(p.310)
(continuing Robert's quote of me):
“ 'The sampling distribution to arrive at Ev(E-BB, (j, y^j )) would be the convex combination averaged over the two ways that y^j could have occurred. This differs from the sampling distributions of both Ev(E’, y’) and Ev(E”, y”)'. This sounds to me like a direct rejection of the conditionality principle, so I do not understand the point.” (Robert, Oct. 6, 2011 post, p.310)

Mayo: I am not at all rejecting the WCP. The passage Robert quotes merely states the obvious; namely, the assertion: the inference computed using the sampling distribution of E-BB is identical to the inference using the sampling distribution of E’ by itself (or E” by itself)—is false! If we are playing Birnbaumization, then the appropriate sampling distribution is the convex combination. (In the section from which Robert is quoting, a reader will note, I have put Birnbaum’s argument in valid form.)

But wait a minute, just a few lines later it turns out Robert does not deny my claim! He repeats it as obviously true, …..but suddenly it has become irrelevant.

Robert: “Indeed, and rather obviously, the sampling distribution of the evidence Ev(E^*,z^*) will differ depending on the experiment. But this is not what is stated by the likelihood principle, which is that the inference itself should be the same for y’ and y” Not the [sampling?] distribution of this inference” (Robert, p. 310).

Mayo: What? This just makes no sense. There is no inference apart from the sampling distribution for a sampling theorist. One cannot assume there is somehow an inference apart from the sampling distribution. Sampling theory has simply not been understood. Robert’s own rendition of the argument [my Premise 1], depends on a merged sampling distribution, thanks to Birnbaumization; it certainly does not ignore sampling distributions. So I’m afraid I don’t know what Robert is talking about here. (This same point arose in the discussion by Aris Spanos when Robert's post first appeared.)

Robert will go on to deny there are any LP counterexamples, because they all turn on pointing up the difference in sampling distributions! All I can do at this point is go back to where I bagan: listen to Birnbaum, Kadane, Lindley, Savage and everyone else who has discussed the (uncontroversial) fact that error statistics violates the LP! No one would be claiming sampling theory was incoherent were it not that it is prepared to reach different inferences from y’, y” despite their having proportional likelihoods (i.e., despite the conditions for the LP being met), and it does so solely because of a difference in sampling distributions.[viii]
[ix]

Kadane, J. (2011), Principles of Uncertainty, CRC Press.
Mayo , http://errorstatistics.blogspot.com/2011/10/blogging-likelihood-principle-2-slp.html
* The title is a distant analogue to that song “Don’t Bogart that chalk my friend, pass it on to me”.

[i] This refers to a mixture experiment where the fair coin toss outcomes determines whether to use a highly precise or a highly imprecise instrument (Cox and Mayo 2010, pp. 295-6).

[ii] But whether Bayesians should care and even greet my critique with a sigh of relief (given that they are nowadays inclined to reject the LP), is a distinct issue.

[iii] If your outcome is not part of a pair that would be an LP violation, forget the imaginary mixture and just report is in the usual way with its regular sampling distribution.

[iv] Abbreviation (1,y’) is just another way to write (E’, y’)—that is, the coin flip outcome directs you to perform E’ and y’ is the resulting outcome.

[v] Note however that the “mixture” in this “Birnbaumization” could as well have had j = 1 with probability ¼ and j=2 with probability ¾, or any other assignments to the outcomes summing to 1—so it is still ill-defined. I don’t think there is any warrant for actually interpreting one’s actual data using the results of a Birnbaumization game. I’m playing along for purposes of showing the argument still fails at the next step.

[vi] I deliberately describe Birnbaumization so that it is possible to perform the experiment, even though it isn’t a genuine mixture experiment.

[vii] That is why sufficiency is considered the “weak likelihood principle”.

[viii]Along with satisfying the other stipulations of the antecedent to the strong LP.

[ix] In referring to an inference from y in a sampling theory experiment E by means of the abbreviation Infr_E(y), we assume, for simplicity, that packed into E would be the probability model, parameters, and the sampling distribution corresponding to the inference in question. We prefer it because it underscores the need to consider the associated methodology and context. Birnbaum construes Ev(E, x) as “the evidence about the parameter arising from experiment E and result x“and allows it to range over the inference, conclusion or report, including p-values, confidence intervals and levels, posteriors. So our notation accomplishes the same, but with (hopefully) less chance of equivocations.

Model Validation and the LP-(Long Playing Vinyl Record)

2012-01-03T00:24:00.005-05:00

A Bayesian acquaintance writes:

Although the Birnbaum result is of primary importance for sampling theorists, I'm still interested in it because many Bayesian statisticians think that model checking violates the likelihood principle, as if this principle is a fundamental axiom of Bayesian statistics.

But this is puzzling for two reasons. First, if the LP does not preclude testing for assumptions (and he is right that it does not[i]), then why not simply explain that rather than appeal to a disproof of something that actually never precluded model testing? To take the disproof of the LP as grounds to announce: “So there! Now even Bayesians are free to test their models” would seem only to ingrain the original fallacy*.

The second reason it is puzzling is that the LP follows from inference by way of Bayes’s theorem (it's not a Bayesian axiom but it's a theorem).

According to Bayes’s theorem, P(x|µ) ... constitutes the entire evidence of the experiment, that is, it tells all that the experiment has to tell. More fully and more precisely, if y is the datum of some other experiment, and if it happens that P(x|µ) and P(y|µ) are proportional functions of µ (that is, constant multiples of each other), then each of the two data x and y have exactly the same thing to say about the values of µ… (Savage 1962, p. 17.)

True, over the past few years, leading developers of “default” Bayesian methods concede that “violation of principles such as the likelihood principle is the price that has to be paid for objectivity” [Berger, 2006, 394][ii], and they might wish to employ the arguments I have sketched invalidating Birnbaum’s “proof”. They are free to do so, if they are prepared to appeal to sampling distributions (and outcomes other than the one observed) even with data in hand.

Berger, J. 2006, The Case for Objective Bayesian Analysis, Bayesian Analysis, 1, 385-402.

*Not that I'm aware of fully adequate Bayesian model checks, but I don't see why they, in principle, could not use error statistical model checks.

[i] The LP always contains a statement to the effect “assuming the model for the experiment is adequate”. To actually determine its adequacy, a consideration of outcomes other than the one observed is no violation of the LP, (even for those who accept it).

[ii] Jim Berger: “Because objective Bayesian methods can violate principles such as the likelihood principle, they can be incoherent according to standard definitions of coherence. (p.394). ...Betting incoherency thus seems to be too strong a condition to apply to communication of information (p.395).

Midnight With Birnbaum

2011-12-31T21:27:00.032-05:00

You know how in that recent movie, “Midnight in Paris,” the main character (I forget who plays it, I saw it on a plane) is a writer finishing a novel, and he steps into a cab that mysteriously picks him up at midnight and transports him back in time where he gets to run his work by such famous authors as Hemingway and Virginia Wolf? He is impressed when his work earns their approval and he comes back each night in the same mysterious cab…Well, imagine an error statistical philosopher is picked up in a mysterious taxi at midnight (New Year’s Eve 2011) and is taken back fifty years and, lo and behold, finds herself in the company of Allan Birnbaum.[i]

ERROR STATISTICIAN: It’s wonderful to meet you Professor Birnbaum; I’ve always been extremely impressed with the important impact your work has had on philosophical foundations of statistics. I happen to be writing on your famous argument about the likelihood principle (LP). (whispers: I can’t believe this!)

BIRNBAUM: Ultimately you know I rejected the LP as failing to control the error probabilities needed for my Confidence concept.

ERROR STATISTICIAN: Yes, but I actually don’t think your argument shows that the LP follows from such frequentist concepts as sufficiency S and the weak conditionality principle WLP.[ii] Sorry,…I know it’s famous…

BIRNBAUM: Well, I shall happily invite you to take any case that violates the LP and allow me to demonstrate that the frequentist is led to inconsistency, provided she also wishes to adhere to the WLP and S (although less than S is needed).

ERROR STATISTICIAN: Well I happen to be a frequentist (error statistical) philosopher; I have recently 2006 found a hole in your proof,..er…well I hope we can discuss it.

BIRNBAUM: Well, well, well: I’ll bet you a bottle of Elba Grease champagne that I can demonstrate it!

ERROR STATISTICAL PHILOSOPHER: It is a great drink, I must admit that: I love lemons.

(A waiter brings a bottle, they each pour a glass and resume talking).

BIRNBAUM: OK. Whoever wins this little argument pays for this whole bottle of the best vintage Ebar or Elbow or whatever it is Grease. I have a new, more satisfactory, version of my earlier arguments.

ERROR STATISTICAL PHILOSOPHER: I really don’t mind paying for the bottle.

BIRNBAUM: Good, you will have to. Take any LP violation. Let x’ be 2-standard deviation difference from the null (asserting m = 0) in testing a normal mean from the fixed sample size experiment E’, say n = 100; and let x” be a 2-standard deviation difference from an optional stopping experiment E”, which happens to stop at 100. Do you agree that:

(0) For a frequentist, outcome x’ from E’ (fixed sample size) is NOT evidentially equivalent to x” from E” (optional stopping that stops at n)

ERROR STATISTICAL PHILOSOPHER: Yes, that’s a clear case where we reject the strong LP, and it makes perfect sense to distinguish their corresponding p-values (which we can write as p’ and p”, respectively). The searching in the optional stopping experiment makes the p-value quite a bit higher than with the fixed sample size. For n = 100, data x’ yields p’= ~.05; while p” is ~.3. Clearly, p’ is not equal to p”, I don’t see how you can make them equal.

BIRNBAUM: Suppose you’ve observed x’, a 2-standard deviation difference from E’. You admit, do you not, that this outcome could have occurred as a result of a different experiment? It could have been that a fair coin was flipped where it is agreed that heads instructs you to perform E’ and tails instructs you to perform the optional stopping test E”, and you happened to get heads, and then performed the experiment E’ (with n = 100) and obtained your 2-standard deviation difference x’.

ERROR STATISTICAL PHILOSOPHER: Well, that is not how I got x’, but ok, it could have occurred that way.

BIRNBAUM: Good. Then you must grant further that your result could have come from a special experiment I have dreamt up, call it a BB-experiment. In a BB- experiment, if the outcome from the experiment you actually performed has an outcome with a proportional likelihood to one in some other experiment not performed, E”, then we say that your result has an “LP pair”.
For any violation of the strong LP, the outcome observed, let it be x’, has an “LP pair”, call it x”, in some other experiment E”. In that case, a BB-experiment stipulates that you are to report x’ as if you had determined whether to run E’ or E” by flipping a fair coin.

(They fill their glasses again)

ERROR STATISTICAL PHILOSOPHER: You’re saying that if my outcome from the fixed sample size experiment E’ has an “LP pair” in the (optional stopping) experiment I did not perform, then I am to report x’ as if the determination to run E’ was by flipping a fair coin (which decides between E’ and E”)?

BIRNBAUM: Yes, and one more thing. If your outcome had actually come from the optional stopping experiment E”, it too would have an “LP pair” in the experiment you did not perform, E’. Whether you actually observed x’ from E’, or x” from E”, you are to report it as x’ from E’.

ERROR STATISTICAL PHILOSOPHER: So let’s see if I understand a Birnbaum BB-experiment: whether my observed 2-standard deviation difference came from E’ or E” the result is reported as x’, as if it came from E’, and as a result of this strange type of a mixture experiment.

BIRNBAUM: Yes, or equivalently you could just report x*: my result is a 2-standard deviation difference and it could have come from either E’ (fixed sampling, n= 100) or E” (optional stopping, which happens to stop at the 100^th trial). That’s how I sometimes formulate a BB-experiment.

ERROR STATISTICAL PHILOSOPHER: You’re saying in effect that if my result has an LP pair in the experiment not performed, I should act as if I accept the strong LP and just report it’s likelihood; so if the likelihoods are proportional in the two experiments (both testing the same mean), the outcomes are evidentially equivalent.

BIRNBAUM: Well, but since the BB- experiment is an imagined “mixture” it is a single experiment, so really you only need to apply the weak LP which frequentists accept. Yes?

ERROR STATISTICAL PHILOSOPHER: But what is the sampling distribution in this imaginary BB- experiment? Suppose I have Birnbaumized my experimental result, just as you describe, and observed a 2-standard deviation difference in a fixed sample size experiment E’. How do I calculate the p-value within a Birnbaumized experiment?

BIRNBAUM: I don’t think anyone has ever called it that.

ERROR STATISTICAL PHILOSOPHER: I just wanted to have a shorthand for the operation you are describing, there’s no need to use it, if you’d rather I not. So how do I calculate the p-value within a BB-experiment?

BIRNBAUM: You would report the overall p-value, which would be the average over the sampling distributions: (p’ + p”)/2

Say p’ is ~.05, and p” is ~.3; whatever they are, we now they are different, that’s what makes this a violation of the strong LP (given in premise (0)).

ERROR STATISTICAL PHILOSOPHER: So you’re saying that if I observe a 2-standard deviation difference from E’, I do not report the associated p-value p’, but instead I am to report the average p-value, averaging over some other experiment E” that could have given rise to an outcome with a proportional likelihood to the one I observed, even though I didn’t obtain it this way?

BIRNBAUM: I’m saying that you have to grant that x’ from a fixed sample size experiment E’ could have been generated through a BB- experiment.

My this drink is sour!

ERROR STATISTICAL PHILOSOPHER: Yes, I love pure lemon.

BIRNBAUM: Perhaps you’re in want of a gene; never mind.

I’m saying you have to grant that x’ from a fixed sample size experiment E’ could have been generated through a BB-experiment. If you are to interpret your experiment as if you are within the rules of a BB experiment, then x’ is evidentially equivalent to x” (is equivalent to x*). This is premise (1).

ERROR STATISTICAL PHILOSOPHER: But this is just a matter of your definitions, it is an analytical or mathematical result, so long as we grant being within your BB experiment.

BIRNBAUM: True, (1) plays the role of the sufficiency assumption, but one need not even appeal to this, it is just a matter of mathematical equivalence.

By the way, I am focusing just on LP violations, therefore, the outcome, by definition, has an LP pair. In other cases, where there is no LP pair, you just report things as usual.

ERROR STATISTICAL PHILOSOPHER: OK, but p’ still differs from p”; so I still don’t how I’m forced to infer the strong LP which identifies the two. In short, I don’t see the contradiction with my rejecting the strong LP (i.e., accepting premise (0)). (Also we should come back to the “other cases” ---those that are not part of LP violation pairs at some point….)

BIRNBAUM: Wait! Don’t be so impatient; I’m about to get to step (2). Here, let’s toast to the new year: “To Elbar Grease!”

(Birnbaum tops off their glasses.)

ERROR STATISTICAL PHILOSOPHER: To Elbar Grease!

BIRNBAUM: So far all of this was just step (1), I'm merely getting started.

ERROR STATISTICAL PHILOSOPHER: Oy, what is step 2?

BIRNBAUM: STEP 2 is this: Surely, you agree, that once you know which experiment the observed 2-standard deviation difference actually came, from you ought to report the p-value corresponding to that experiment. You ought NOT to report the average (p’ + p”)/2 as you were instructed to do in the BB experiment.

This gives us premise (2a):

(2a): Outcome x’, once it is known that it came from E’, should NOT be analyzed as in a BB- experiment where p-values are averaged. The report should instead use the sampling distribution of the fixed sample test E’, yielding the p-value ,p’ (.05).

ERROR STATISTICAL PHILOSOPHER: So, having first insisted I imagine myself in a Birnbaumized, I mean a BB-experiment, and report an average p-value, I’m now to return to my senses and “condition” in order to get back to the only place I ever wanted to be, i.e., back to where I was to begin with?

BIRNBAUM: Yes, at least if you hold to the weak conditionality principle WCP (of D. R. Cox)---surely you agree to this.

(2b): Likewise, if you knew the 2-standard deviation difference came from E”, then

x” should NOT be deemed evidentially equivalent to x’ (as in the BB experiment), the report should instead use the sampling distribution of the optional stopping test E”. This would yield p-value p’ (~.3).

ERROR STATISTICAL PHILOSOPHER: So, having first insisted I consider myself in a BB-experiment, in which I report the average p-value, I’m now to return to my senses and allow that if I know the result came from optional stopping, E”, I should “condition” on E” and report p”.

BIRNBAUM: Yes. There was scarcely a need to repeat the entire spiel.

ERROR STATISTICAL PHILOSOPHER: I just wanted to be clear I understood you.

BIRNBAUM: So you arrive at (2a) and (2b), yes?

ERROR STATISTICAL PHILOSOPHER: OK, but it might be noted that unlike premise (1), premises (2a) and (2b) are not given by definition, they concern an evidential standpoint about how one ought to interpret a result once you know which experiment it came from. In particular, premises (2a) and (2b) say I should condition and use the sampling distribution of the experiment known to have been actually performed, when interpreting the result.

BIRNBAUM: Yes, and isn’t this weak conditionality principle WCP one that you so happily accept?

ERROR STATISTICAL PHILOSOPHER: Well the WCP is defined for actual mixtures, where one flipped a coin to determine if E’ or E” is performed, whereas, you’re requiring I consider an imaginary Birnbaum mixture experiment, where the choice of the experiment not performed will vary depending on the outcome that needs an LP pair; and I cannot even determine what this might be until after I’ve observed the result that would violate the LP?

BIRNBAUM: Sure, but you admit that your observed x’ could have come about through a BB-experiment, and that’s all I need. Notice

(1), (2a) and (2b) yield the strong LP!

Outcome x’ from E’ (fixed sample size n) is evidentially equivalent to x” from E” (optional stopping that stops at n).

ERROR STATISTICAL PHILOSOPHER: Clever, but your “proof” is obviously unsound. Our account says the evidential appraisal of the two outcomes x' and x" ought to be different, just as always. The LP violation (premise (0)) leads to no inconsistency, even if we go along with Birnbaumization.

BIRNBAUM: Well, it is puzzling, but where have I gone wrong?

(The waiter come by and fills their glasses; they are so deeply engrossed in thought they do not even notice him.)

ERROR STATISTICAL PHILOSOPHER: There are many routes to explaining a fallacious argument. Here’s one. What is required for Step 1 to hold, is the denial of what’s needed for Step 2 to hold:

Step 1 requires us to analyze results in accordance with a BB- experiment. If we do so, true enough we get:

premise (1): outcome x' (in a BB experiment) is evidentially equivalent to outcome x" (in a BB experiment):

That is because in either case, the p-value would be (p' + p")/2

Step 2 now insists that we should NOT calculate evidential import as if we were in a BB- experiment. Instead we should consider the experiment from which the data actually came, E’ or E”:

premise (2a): outcome x' (within in a BB experiment) is/should be evidentially equivalent to x' from E' (fixed sample size): its p-value should be p'.

premise (2b): outcome x" (in a BB experiment) is/should be evidentially equivalent to x" from E" (optional stopping that stops at n): its p-value should be p".

If (1) is true, then (2a) and (2b) must be false!

If (1) is true and we keep fixed the stipulation of a BB experiment (which we must to apply step 2), then (2a) is asserting:

The average p-value (p' + p")/2 = p' which is false.

Likewise if (1) is true, then (2b) is asserting:

the average p-value (p' + p")/2 = p" which is false

Alternatively, we can see what goes wrong by realizing:

If (2a) and (2b) are true, then premise (1) must be false.

In short your famous argument requires us to assess evidence in a given experiment in two contradictory ways: as if we are within a BB- experiment (and report the average p-value) and also that we are not, but rather should report the actual p-value.

I can render it as formally valid, but then its premises can never all be true; alternatively, I can get the premises to come out true, but then the conclusion is false---so it is invalid. In no way does it show the frequentist is open to contradiction (by dint of accepting S, WCP, and denying the LP).

BIRNBAUM: Yet some people still think it is a breakthrough (in favor of Bayesianism).

WAITER: Who gets the tab?

BIRNBAUM: I do. To Elbar Grease!

ERROR STATISTICAL PHILOSOPHER: To Elbar Grease! Happy New Year!

[i] For the specifics on these two experiments see, Dec 6 and 7 posts. See also Mayo 2010
Other posts relating the LP are: Oct 6, Oct 20, Nov 27, Dec 11, Dec 22.
Links to Elbar Grease: Sept 30, Nov 5, Nov 23.

[ii] By the way, Ronald Giere gave me numerous original papers of yours. They’re in files in my attic library. Some are in mimeo, others typed…I mean, obviously for that time that’s what they’d be…now of course, oh never mind, sorry.

JIM BERGER ON JIM BERGER!

2011-12-29T00:13:00.012-05:00

Fortunately, we have Jim Berger interpreting himself this evening (see December 11)

Jim Berger writes:

A few comments:

1. Objective Bayesian priors are often improper (i.e., have infinite total mass), but this is not a problem when they are developed correctly. But not every improper prior is satisfactory. For instance, the constant prior is known to be unsatisfactory in many situations. The 'solution' pseudo-Bayesians often use is to choose a constant prior over a large but bounded set (a 'weakly informative' prior), saying it is now proper and so all is well. This is not true; if the constant prior on the whole parameter space is bad, so will be the constant prior over the bounded set. The problem is, in part, that some people confuse proper priors with subjective priors and, having learned that true subjective priors are fine, incorrectly presume that weakly informative proper priors are fine.

2. My more provocative comment was based on the fact that objective Bayesians worry a lot about the prior, and work hard to get a prior that is good in situations where one does not have much prior information or is obligated to use impartial priors (e.g., by regulation). True subjective Bayesians also worry a lot about the prior, attempting to model their prior information carefully, doing sensitivity studies, etc. But, in part because Bayesian analysis has become so popular and is being used by many without training in either objective Bayesian or true subjective Bayesian methods, there are many quite adhoc choices of priors being made that have no inherent justification and would, I claim, be much less 'Bayesian' than the objective Bayesian priors. All that is less exciting than my provocative comment, but it is what I had in mind when writing the comment.

------------------------------------

UPDATE: Dec. 29, 2011: Andrew Gelman today fired back with a lengthy post on his own blog. I hope others will join in the discussion in the commentary here.

Contributed Deconstructions: Irony & Bad Faith 3

2011-12-26T22:58:00.068-05:00

My efficient Errorstat Blogpeople¹ have put forward the following 3 reader-contributed interpretive efforts² as a result of the “deconstruction” exercise from December 11, (mine, from the earlier blog, is at the end) of what I consider:

“….an especially intriguing remark by Jim Berger that I think bears upon the current mindset (Jim is aware of my efforts):

Too often I see people pretending to be subjectivists, and then using “weakly informative” priors that the objective Bayesian community knows are terrible and will give ridiculous answers; subjectivism is then being used as a shield to hide ignorance. . . . In my own more provocative moments, I claim that the only true subjectivists are the objective Bayesians, because they refuse to use subjectivism as a shield against criticism of sloppy pseudo-Bayesian practice. (Berger 2006, 463)” (From blogpost, Dec. 11, 2011)

_________________________________________________
Andrew Gelman:
The statistics literature is big enough that I assume there really is some bad stuff out there that Berger is reacting to, but I think that when he's talking about weakly informative priors, Berger is not referring to the work in this area that I like, as I think of weakly informative priors as specifically being designed to give answers that are _not_ "ridiculous."

Keeping things unridiculous is what regularization's all about, and one challenge of regularization (as compared to pure subjective priors) is that the answer to the question, What is a good regularizing prior?, will depend on the likelihood. There's a lot of interesting theory and practice relating to weakly informative priors for regularization, a lot out there that goes beyond the idea of noninformativity.

To put it another way: We all know that there's no such thing as a purely noninformative prior: any model conveys some information. But, more and more, I'm coming across applied problems where I wouldn't want to be noninformative even if I could, problems where some weak prior information regularizes my inferences and keeps them sane and under control.

Finally, I think subjectivity and objectivity both are necessary parts of research. Science is objective in that it aims for reproducible findings that exist independent of the observer, and it's subjective in that the process of science involves many individual choices. And I think the statistics I do (mostly, but not always, using Bayesian methods) is both objective and subjective in that way. That said, I think I see where Berger is coming from: objectivity is a goal we are aiming for, whereas subjectivity is an unavoidable weakness that we try to minimize. I think weakly informative priors are, or can be, as objective as many other statistical choices, such as assumptions of additivity, linearity, and symmetry, choices of functional forms such as in logistic regression, and so forth. I see no particular purity in fitting a model with unconstrained parameter space: to me, it is just as scientifically objective, if not more so, to restrict the space to reasonable values. It often turns out that soft constraints work better than hard constraints, hence the value of continuous and proper priors. I agree with Berger that objectivity is a desirable goal, and I think we can get closer to that goal by stating our assumptions clearly enough that they can be defended or contradicted by scientific theory and data---a position to which I expect Deborah Mayo would agree as well.

(see also Gelman's blog)

_________________________________________________

Davidjrohde:

This comment was published in Bayesian analysis which has an obviously specialist audience, the two articles and the comments on the two articles reveals a near unanimous preference for subjective Bayes as the foundations of statistics. To this narrow specialist audience "subjective" is a complement, an idealized limiting case of an optimal statistical analysis. If you have a philosophical objection to subjective Bayes (or Bayes in general) as the foundations of statistics then you are really far outside the target audience and understandably the comment will be opaque.

I think Berger is saying that an objective Bayesian might understand the consequences of diffuse priors better than a subjective Bayesian, he is probably employing both Bayesian and non-Bayesian criteria to investigate the consequence of priors, making objective Bayes a bit of a piece meal "theory". My reading of the article is that Berger is a subjectivist, who is promoting tools outside standard subjective Bayesian theory (objective Bayes and frequentist) on practical grounds, it is interesting that the more extreme objective Bayes arguments of Jefreys and Jaynes seem to be largely abandoned now.

Of course the article reveals differences in Bayesians, but I think also reveals a remarkable convergence of opinion. Subjective Bayes is the foundations of statistics, but in an operational sense fully specifying subjective probabilities and then conditioning on observables is not remotely practical. Berger and Goldstein suggest different tools for dealing with this problem and the debate is largely carried out within this context (excluding Wasserman's comments).

_________________________________________________

Eileens34:

My guess is that there is a typo, and Berger meant to say “the only true objectivists are the objective Bayesians...” in the quote above. Mystery solved!

_________________________________________________

Deborah Mayo (from blogpost, December 11, 2011):

How might we deconstruct this fantastic remark of Berger’s?5 (Granted, this arises in his rejoinder to others, but this only heightens my interest in analyzing it.)

Here, “objective Bayesians” are understood as using (some scheme) of default or conventionally derived priors. One aspect of his remark is fairly clear: pseudo-Bayesian practice allows “terrible” priors to be used, and it would be better for them to appeal to conventional “default” priors that at least will not be so terrible (but in what respect?). It is the claim he makes in his “more provocative moments” that really invites deconstruction. Why would using the recommended conventional priors make them more like “true subjectivists”? I can think of several reasons—but none is really satisfactory, and all are (interestingly) perplexing. I am reminded of Sartre’s remarks in Being and Nothingness on bad faith and irony:

"In irony a man annihilates what he posits within one and the same act; he leads us to believe in order not to be believed; he affirms to deny and denies to affirm; he creates a positive object but it has no being other than its nothingness." (Sartre)

So true! (Of course I am being ironic!) Back to teasing out what’s behind Berger’s remarks.

Now, it would seem that if she did use priors that correctly reflected her beliefs (call these priors “really informed by subjective opinions”(riso?), and that satisfied the Bayesian formal coherency requirements, then that would be defensible for a subjective Bayesian. But Berger notices that, in actuality, many Bayesians (the pseudo-Bayesians) do not use riso priors. Rather, they use various priors (the origin of which they’re unsure of) as if these really reflected their subjective judgments. In doing so, she (thinks that she) doesn’t have to justify them—she claims that they reflect subjective judgments (and who can argue with them?).

According to Berger here, the Bayesian community (except for the pseudo-Bayesians?) knows that they’re terrible, according to a shared criterion (is it non-Bayesian? Frequentist?). But I wonder: if, as far as the agent knows, these priors really do reflect the person’s beliefs, then would they still be terrible? It seems not. Or, if they still would be terrible, doesn’t that suggest a distinct criterion other than using “really informed” (as far as the agent knows) opinions or beliefs?

Berger, J. (2006),“The Case for Objective Bayesian Analysis”, and “Rejoinder”, Bayesian Analysis 1(3), 385–402; 457-464.

Sartre, J.P Being and Nothingness: an essay in phenomenological ontology (1943, Gallimard); English 1956, Philosophical Library Inc.

[1] This is totally unrelated, I think, but the crew from Elba have wanted me to blog about some symbiotic worm one of them studies in Elba. I don’t know if there is a suggestion of an analogy between a symbiotic relationship between objective and subjective Bayesians , but I really prefer not to blog about worms, or even think about them (this footnote is all**). The truth is, I used to dig for worms with my brother when we were kids, but then had some bad experiences in Ferndale one summer. Sorry guys, start a blob blog of your own.

**Here's a website: http://www.youtube.com/watch?v=rxEC4CVswYI

[2] Strictly speaking, you could submit your creative attempts by Wednesday, Dec. 28, and of course, you can always comment, as I will. There were 4 other contributions which are fine as comments, more than deconstructions.

Little Bit of Blog Log-ic

2011-12-25T21:36:00.018-05:00

I have a logic license

My “Logic” chariot, crunched from behind before my travels, you might recall (blogpost Nov. 15, “Logic Takes a Bit of a Hit”), has been robustly repaired and beautifully corrected, all in my absence!¹ So here’s a little bit of blog logic….

In a couple of the early posts (e.g., Sept. 9 post), some logical terms were noted (e.g., the valid form of modes tollens); but it can’t hurt to review them with a mind toward the specific patterns of arguments that arise in the Birnbaum case.

An argument is deductively valid if it’s impossible for all of its premises to be true and its conclusion false at the same time (on pain of logical contradiction). By validity, in this post, I will always mean deductively valid. The conclusion is what is inferred from the premises that purport to provide evidence for its truth.² To say an argument is valid is not to say its premises or its conclusion are true, but only that if all the premises are true, then it’s conclusion would also have to be true.   It's an if-then claim.

A valid argument can have a false conclusion, but if it does, then at least one of the premises must be false.

Deductive validity is a matter of pure form (hence the term formal logic). If an argument (argument form) is valid, then any argument with that same form is also valid. Likewise for arguments that are invalid.

For an argument to be sound, however, it must have true premises as well as be formally valid. Since we want to infer the conclusion, i.e., detach it from its premises, we would like sound arguments. Making an argument valid (by adding premises) is easy; ensuring those premises are true is hard.   In evaluating an argument for validity, we do not add premises, but evaluate it just as given. (A different activity could be to consider what additional premises, if added, would convert an invalid argument into a valid one.)

EXAMPLE 1: Here’s a valid argument:

Premises:

1. Any two entrees ordered off the special Dec. 25 menu M* have the same price.

2. The duck and the mahi mahi were both ordered off the special Dec. 25 menu M*.

3. The price of the duck entrée is $29.99.

Conclusion:, the price of the mahi mahi entree is $29.99.

To give a partial symbolization with (hopefully) obvious assignments to the abbreviations,we might have:

1. For all x, y, If M*(x) and M*(y), then $(x) = $(y)

2. The duck and the mahi mahi have property M*, that is,

M*(duck) and M*(mahi mahi)

3. $(duck) = $29.99

Conclusion: Therefore, $(mahi mahi) = $29.99.

Note: M* is a property (of entrees), $ is a function (from entrees to prices).

If the bill shows your mahi mahi order is not $29.99, then one of the three premises is false. Once you discover, say, that the mahi mahi was actually not ordered off the special menu M*, but rather off the regular menu M’, (and you were charged the mahi mahi price listed in menu M’), then you know the second premise is false. The argument is valid but unsound. For an argument to be sound, it must have true premises as well as be formally valid.

EXAMPLE 2: An Invalid argument

1. Any two entrees ordered off the special Dec. 25 menu M* have the same price.

2. Duck and mahi mahi were ordered, and the duck was ordered off the special Dec. 25 menu M*.

3. The price of the duck is $29.99.

Conclusion: the price of the mahi mahi is $29.99.

It’s easy to see that the premises of example 2 can be all true and yet the conclusion false. The way we actually show truth values of sentences requires assigning interpretations to the elements of the argument: the domain over which variables x, y range, the names of objects in the domain, and the various properties, relations, and functions on the domain.³ An interpretation that makes all the premises true is a "model" of those statements.

EXAMPLE #3: Exercise: Valid or invalid? (Try to symbolize)

1. For any married couple (x,y) filing federal taxes jointly (in the U.S.), x and y have the same tax liability; namely, the amount in the "married filing jointly" column.

2. If a married couple in the U.S. does not file jointly but each files separately, then each owes the amount in the "married, filing separately" column.

3. Deborah and George are a married couple in the U.S.

4. If Deborah files separately, then the amount of tax Deborah owes equals d.

5. If George files separately, then the amount of tax George owes equals g.

Conclusion: d = g

You may assume these premises are true, e.g., that d and g are dollar numbers given in the respective "married filing separately columns". Never mind deductions or the like.

[1] Credit goes to George!

[2] An inference can refer to the entire argument or the conclusion drawn from the premises. I will always indicate which is meant.

[3] 1. For all x, y If M*(x) and M*(y) ithen $(x) = $(y)

     2. M*(duck) (i.e., the duck order has property M*)

     3. $(duck) = $29.99

     Therefore, $(mahi mahi) = $29.99.