Rough Theory

Theory In The Rough

Insomnia [Updated After Some Sleep...]

Updated to add, after more sleep and a chance to backtrack the history in a way I couldn’t stay awake to do at 4 a.m., I’m backing cautiously away from the sympathetic read of the piece as satire. Which is a pity, since replicating standard psychology experimental methods in a way that so successfully draws out the limitations of some common problematic research practices would have been just brilliant – much more brilliant than a “straight” read of the paper… (What can I say – my tendency to look for sarcasm has a wish-fulfilment element: I want more brilliant things in the world – and many things would in fact be more brilliant – if only they were written as satires…) The rebuttal gets it right nevertheless:

Do these results mean that psi can now be considered real, replicable, and reliable?

We think that the answer to this question is negative, and that the take home message of Bem’s research is in fact of a completely different nature. One of the discussants of the Utts review paper made the insightful remark that “Parapsychology is worth serious study. (…) if it is wrong [i.e., psi does not exist], it offers a truly alarming massive case study of how statistics can mislead and be misused.” (Diaconis, 1991, p. 386). And this, we suggest, is precisely what Bem’s research really shows. Instead of revising our beliefs regarding psi, Bem’s research should instead cause us to revise our beliefs on methodology: the field of psychology currently uses methodological and statistical strategies that are too weak, too malleable, and offer far too many opportunities for researchers to befuddle themselves and their peers.

***

I try to keep this off the blog, but, in my spare time – and sometimes in time that shouldn’t be so spared – I read sort of absurd quantities of research in the “hard” sciences. I’m particularly fond of medical research, but I’ll read more or less anything that I can manage to follow – it’s a sort of indiscriminate, unfocused, random consumption of indifferent scientific research. Often, this reading starts informally enough, trawling through the science sections of newspapers – a quick read usually generates at least one article in an average science section that just doesn’t feel right, and I backtrack from there to the original studies (some easier to find than others…). Sometimes the gratification is just working out what the news account got wrong; sometimes the gratification comes in the form of irritation at a badly designed study that was, unfortunately, pretty accurately reported in the press; sometimes the research will be genuinely interesting and well designed – and then I often end up tracking from the paper where I landed, to other material written on the field. This gives me scattered tidbits of information on all sorts of random stuff, without much expertise in anything in particular. But I enjoy it. It’s how I relax.

It’s very late here at the moment, and I’m trying to come down from a fairly intense day, so I’m trawling through science sections… and I ran across this piece in the The New York Times, titled, “Journal’s Paper on ESP Expected to Prompt Outrage”.

The article refers to this paper by Daryl Bem, titled “Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect”, apparently accepted for publication in The Journal of Personality and Social Psychology. Bem’s paper is hilarious – a self-evident satire, was my reaction even from the summary provided by the New York Times. Regular readers, if any are left after my very irregular blogging, will know that I perhaps stumble across satire more than could be accounted for by its actual presence in the world, but still, here’s the Times on Bem’s method:

In an interview, Dr. Bem, the author of the original paper and one of the most prominent research psychologists of his generation, said he intended each experiment to mimic a well-known classic study, “only time-reversed.”

In one classic memory experiment, for example, participants study 48 words and then divide a subset of 24 of them into categories, like food or animal. The act of categorizing reinforces memory, and on subsequent tests people are more likely to remember the words they practiced than those they did not.

In his version, Dr. Bem gave 100 college students a memory test before they did the categorizing — and found they were significantly more likely to remember words that they practiced later. “The results show that practicing a set of words after the recall test does, in fact, reach back in time to facilitate the recall of those words,” the paper concludes.

In another experiment, Dr. Bem had subjects choose which of two curtains on a computer screen hid a photograph; the other curtain hid nothing but a blank screen.

A software program randomly posted a picture behind one curtain or the other — but only after the participant made a choice. Still, the participants beat chance, by 53 percent to 50 percent, at least when the photos being posted were erotic ones. They did not do better than chance on negative or neutral photos.

“What I showed was that unselected subjects could sense the erotic photos,” Dr. Bem said, “but my guess is that if you use more talented people, who are better at this, they could find any of the photos.”

That’s just brilliant!! Just fucking brilliant!!

But in case there were any doubt, the “rebuttal” piece to be published alongside Bem’s original is titled: “Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi”. So there’s the real point of Bem’s piece, nicely laid out in the form of a critique of what is meant to be a self-evidently methodologically flawed original.

Now it’s late here – very late – and so my satire/sarcasm detector is, I’ll confess, a bit on the fritz in relation to the Times article itself. Is the author in on the joke? The piece does quote someone proposing that it might all be a hoax. I just can’t tell whether the voicing of the article is meant to align with this quote, or whether the author is intending to report this straight.

Regardless, the comments to the article (I’ve looked at only a few – there are hundreds) suggest that much of the readership is taking the study at face value – and then either criticising it, or supporting it, depending on the commenter’s attitude to science, psychic phenomena, etc. Periodically commenters pop up saying this is a joke/hoax/morality tale/satirical plea for greater methodological rigour. I’m too tired to trawl through the comments to have a sense of where the balance falls – what the ratio of “it’s a hoax” to “it’s a serious piece” comments would be. My suspicion would be that more commenters will assume the original study is legit than not. Which raises the question: if the NYT reporter is in on the joke, was this the wisest way to play along with the gag? Within an academic community specialising in relevant forms of research, there’s enough context – enough shared professional sensibility – for a satire to have a fighting chance of being recognised as such. Once the satire is removed from that space, far fewer assumptions can be made about what perspectives and background knowledge readers will share – which makes it much more likely the satirical point will be missed…

And one final question – which I would try to answer myself if I were less tired, but my eyes won’t focus on any more reading tonight… Some commenters who are explaining that the original piece is satire, are suggesting that the journal has been Sokalled. Really? I would hope it’s a one-two punch proposed to the journal, with the “rebuttal” and the original essentially submitted together?

Apologies to everyone who has been asking serious social theoretic questions in the comments recently – I hadn’t intended to post until I had proper time and thought-space to respond… Hopefully back on topic soon…

Must. Sleep. Now.

One response to “Insomnia [Updated After Some Sleep...]

  1. john doyle January 22, 2011 at 12:05 am

    I wonder if it’s possible to know the results of a study prior to collecting and analyzing the data. Oh wait, I guess Bem demonstrated that too.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: