Rough Theory

Theory In The Rough

Category Archives: Fieldwork

Wearing the Juice: A Case Study in Research Implosion

[Ed. 9 September: Now that events are unfolding a bit more slowly, and people have had a chance, for the most part, to learn about the basic facts, I've moved my on-the-fly updates to the bottom of the post, so that the original text is easier to find. I will try to update all the broken links next week.]

Original Post

A couple of people have sent me the link to this debacle of two researchers attempting to study what they call the “Cognitive Neuroscience of Fan Fiction” (further historical background here and here, collated links there, and information about the original research (which somehow doesn’t get around to mentioning that the research is designed – not for academic publication – but for a popular book whose working title is Rule 34: What Netporn Teaches Us About the Brain) in the researchers’ background information).

As someone looking on from outside the fan communities directly involved in this mess, the whole thing unfolds something like a live action version of the phenomenon Justin Kruger and David Dunning discuss in their “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments” (1999, Journal of Personality and Social Psychology, Vol. 77, No. 6., p. 1121-1134).

Kruger and Dunning are interested in whether, below a certain level basic competence, it becomes very difficult for people to improve their skills – because they are, in fact, too incompetent to be able to tell the difference between competence and incompetence in the first place. They take as their point of departure the story of hapless bank robber McArthur Wheeler who – some of you will remember from my previous post on this article – robbed two banks in broad daylight without any disguise and, when arrested almost immediately based on the bank security footage, burst out: “But I wore the juice!” Mr. Wheeler was evidently under the impression that, by rubbing lemon juice on his face, he could conceal himself from security cameras (Kruger and Dunning 1999: 1122).

Assuming this mess is not some sort of elaborate research-themed performance art, or the result of a revenge-fuelled identity theft, researchers Ogi Ogas and partner Sai Chaitanya Gaddam are trying their best to demonstrate to the world that they are something like the academic research equivalent to Wheeler. They have blundered into an online community whose members write and read, among other things, erotically-themed fan fiction, and have presented community members with a poorly-designed questionnaire (now taken down, but for a while being modified on the fly as people lined up with complaints about the research design – participants have posted screenshots and a text version of the survey after its initial modifications – note that a number of the final option responses and some other warnings and qualifications seem to have been added in response to criticisms of the survey in its original form – the modifications are often palpably different in style from the original text).

Among many other problems, the questionnaire asks respondents to provide sensitive information about sexual habits, desires and fantasies, in a setting where the questionnaire could be accessed by minors, without – as far as I can tell – having vetted the research design with their university’s IRB (the researchers are currently being hounded across several websites with demands to answer the question of whether they did, in fact, submit the project for ethics review – while answering other questions, they have steadfastly ignored this one: quick suggestion that, if the researchers don’t mean to imply the answer is ‘no’, then they should probably address this question very explicitly, very soon). [Side note: there's a nice critical discussion of the limitations of IRB's that's been sparked by this whole mess: here.]

In the ongoing discussions now sprawled across a number of sites, the authors continue to dig this initial hole deeper by using terms regarded as offensive by members of the community (and, in one case, defending this because these are the terms that are standard in the sex industry – as Marx might say: !!!), by blithely demonstrating their own participation in widely-criticised assumptions about sexuality and presuppositions about gender, by demonstrating ignorance of basic facts about the community that could be gleaned from a quick skim of community sites, and by insisting on knocking back well-reasoned and absolutely on-target critiques by arguing that they are not doing “social research” and are not actually interested in the community anyway, other than as an example of a much more general phenomenon (these last, the researchers seem to believe, get them off the hook on ethical and basic research design requirements).

I’m not going to write my own critique of this mess: the community has already done this, eloquently, thoroughly – and, given the circumstances, with admirable patience. I am always warning my students when I teach research methods that something like this can happen – that this is why I’m so harsh on their research designs. Welcome to my new case study. I’m serious. I’m thinking of assigning parts of this trainwreck when I teach research methods next term.

I’m posting on this mainly because I’m wondering why the researchers have not apologised far more abjectly for having blundered into a community so ill-prepared – and possibly having ignored basic legal requirements and professional ethical standards governing their research. I am wondering if they are simply failing to register how devastating are the critiques being made of their work – perhaps because they are assuming these critiques have arisen defensively, due to strong affective attachments and loyalties within this particular community – or perhaps because they have “othered” this community so much that they aren’t sufficiently open to how badly they are being schooled here. Sai Gaddam’s university website suggests a potential vulnerability in this regard – let me quote from the source (apologies: I owe a poster in the original discussion a hat-tip for drawing attention to this, but unfortunately I’ve lost track of the comment – if you want to make yourself known, I’ll add a link):

My research interests have evolved over the years I have spent in the Ph.D program, but my derision for my subjects remains a constant. Well, not really, but this quote does make me smile.

The individual I chose as my principal subject for the experiments … was an old toothless man, with a thin face, whose features, without being absolutely ugly, approached ordinary triviality, and whose facial expression was in perfect agreement with his inoffensive character and his restricted intelligence.

The Mechanism of Human Facial Expression — Guillaume de Boulogne

So, for what it’s worth: I don’t belong to this community, but the criticisms being made of your ill-conceived research are excellent. Listen to them. You have tried wearing the juice. They’ve seen through it. It wasn’t the disguise you hoped it might be.


[Ed. 7 September: Still no time to update the broken links below, but wanted to point to the discussion at metafilter, for those interested. ETA: and Neuroanthropology weighs in! - Twice!]

[Ed. 4 September: If people aren't aware, Ben Goldacre from Bad Science has referenced SurveyFail on Twitter, linking here and also to Alison Macleod's fantastic overview at The Human Element. Rushing at the moment - apologies for not responding yet to comments.]

[Ed. 4 September: Another day, a few more broken links. Ogi Ogas and Sai Gaddam seem to have had their websites removed from Boston University - not surprising, given the report that they are not affiliated with the university for purposes of this research project. Gaddam's blog has also been made private. The links I have below off their names therefore no longer point anywhere. Again: my schedule's too hectic to fix this right now, so just noting the problem. Some limited information about Ogas is included in his Wikipedia page, as a backup link... If the old Boston University pages end up being included in any of the screencaps collections currently being collated online, I'll restore links to those once I have time.

For folks interested in legs, this post has been picked up at Josh Jasper's blog at Publisher's Weekly, as well as at Alison Macleod's the human element. Macleod's blog has a very clear overview of how the whole thing unfolded, as well, for folks new to this whole mess and trying to get a sense of what happened.

Broken link clean-ups still days in the future, I'm afraid...]

[Ed. 3 Sept: Folks, just a note that the researchers have taken down their site - after an amazingly offensive final blowup that, honestly, must be seen to be believed... This will break a lot of the links I've posted below. I'll try to clean these up later, but for the time being, there's are a number of good summaries of the whole incident - now christened SurveyFail - see especially Yonmei's post at, as well as a report of a response from the IRB at their university, which has disclaimed any affiliation with the project and asked the researchers not to use their uni emails or web addresses in conjunction with this activity. (My favorite part of the linked IRB discussion was the report that, when the IRB office was contacted directly: "Their exact words 'I had a feeling it would be about that.'") Links cleanup might have to wait a couple of days - schedule is awful at the moment...]

One Way

Gungahlin town centre clock, one way sign, and construction barriers.I spent some time today with a group of people working – loosely – on issues relating to heritage, neighbourhood character, and “place making” in a community facing massive demographic change. One of the persons present had been involved in the creation of the ACT Cultural Map, and presented some highlights from that project as grist for discussion. The presentation highlighted a number of features from the Gungahlin town centre design – a greenfield development that, according to the presentation, recruited a local artist to create designs based on stories collected during community consultations. Developers have begun to incorporate these designs into new structures in a variety of ways – from patterns on manhole covers, to distinctive bus shelter designs, to etchings on glass doorways in the town centre – to create a distinctive sense of place while commemorating elements of the area’s history. Much of the presentation centred on visual images of the design elements created through this process.

This kind of commemoration always has a strange, haunted character for me, as it effectively celebrates what has been destroyed by the development process, and tries to build a sense of the distinctiveness of the new community by pointing to what is no longer there – as though the new community is expected to coalesce around what it has displaced. The discussion today centred on images of various design elements – themselves generally quite attractive, and spoken about, initially, just in terms of their visual appeal and distinctiveness. The mood in the room was playful, excited about the possibility of creating similarly unique visual elements in new communities locally, and the discussion revolved around the aesthetic merit of the designs, viewed as communal art.

At one point, however, the content of the artwork suddenly broke through what had, until that point, been essentially a discussion of form, and there was an almost tactile wrenching and reorientation of the mood in the room. The shift took place as the presenter displayed an image of the grates used around the base of new street trees, and the group puzzled over what the grates – which at first glance just looked attractively functional – were meant to represent. The presenter, excited and enthusiatic, explained:

They’re tree roots! Do you see? Because beautiful old trees were cut down – and their roots were everywhere, knotted together – and they’re gone now…

The presenter suddenly paused, thrown out of the presentation by registering – as the rest of us also were – the fundamental strangeness of surrounding these spindly new trees, all planted in their isolated and orderly formation, with artwork representing the mesh of mature root systems from trees that had grown old together, intertwined, and had then been destroyed to make way for the development process. No one voiced or telegraphed any criticism – the mood in the room was poignant, not critical. The presenter paused for some time, not really knowing what to say. Then quietly, almost reverent:

Well… at least we’ve got the memory of them…

I’ve committed to writing a conference paper loosely organised around the issue of how we understand the concept of “community” in a dynamic social context. Tentatively, the paper will discuss the “problem” of post-traditional communities as a foundational issue for classical sociology, make a few gestures at contemporary planning theory discussions on “community”, and then explore the ways in which some of these concepts play out in a couple of case studies from my field research. I may periodically toss up fieldnotes of this sort, as I try to work my way into what, exactly, I plan to write – the draft paper will eventually make its way onto the site. Happy as always to receive feedback on the theoretical or empirical dimensions of the piece.

[Note: image of the Gungahlin town centre clock modified from the one Cfitzart posted to Wikipedia. The original image - and therefore this one - is posted under the terms of a GNU Free Documentation License.]

Dubious Text

So my talk for the “Dubious Ethnography” panel is out of the way – one down, one to go. I went through a particularly intense crisis of confidence about the whole thing yesterday, when the talk remained unwritten at 6 p.m., after an entire day filled with nothing but endless interruptions. It also didn’t seem promising that I have an intense sore throat and the beginnings of what feels like an ear infection – and, as I explained to the audience this morning, not being able to speak or hear seemed an unpromising beginning for a discussion…

In the end, though, I did enjoy giving the talk – and received some very good questions. Interestingly, the most positive and the most negative reactions related to my discussion of epistemology and critical judgment – which is somewhat amusing, as people generally just fall asleep when I discuss epistemology. Maybe I’m onto something with this narrative thing… ;-P

Some members of the audience really liked the notion of trying to understand the reasonableness of various positions in a local political conflict, while also trying to examine all of those positions critically for what they don’t quite grasp with reference to a more overarching and comprehensive vision of that context. One questioner in particular, though, was very unhappy with this proposal, really pressed me to declare a side – and then was unconvinced when I tried to explain that my main quarrel was not really with anything that was unfolding in the community where I research, but rather with certain frameworks with in the academic literature: that my main “side” was a critique of those academic positions.

I was challenged further to explain how this was an ethical position – don’t we ultimately all have to take sides with reference to what we are studying? Is it ethical to analyse the weaknesses in all competing positions without choosing a particular position we most strongly prefer? I suspect this is really, at base, not the universal and theoretical issue the questioner takes it to be, but more like an empirical and contingent question: depending on the conflict, it might be possible or impossible, ethical or unethical, to choose a side. My main purpose at the moment (not in this brief talk, which would be completely inadequate, but in the thesis) is to make plausible the notion that we can ground judgments in a recognition that some kinds of mistakes can be made by otherwise quite reasonable and moral people, who have seized upon a piece of their social context, confused that piece for the whole – and act as though everyone else has done the same… The context will then determine whether these judgments drive in favour of a form of political movement actually playing itself out on the ground in a particular dispute. I don’t think my answer was adequate – I’ll have to work on explaining what I mean.

Anyone who’d like a copy of the talk can email, with the caveat that, as always, the written version is not quite what I actually said – I tend to watch audiences, dwell on things that seem to get people nodding in agreement, and skip lightly over things that seem to get people nodding off… I’ll leave readers to guess which sections of the text fell into which categories…

Now I have to collect my thoughts for tomorrow’s talk – which, for local readers, will be delivered as part of the Environment & Planning Lunchtime Seminar series, in 8.7.6, at 12:30 (attendance is free; BYO food…).

Seeking Safe Haven

It's total war!  Everyone must fight or work!Fieldwork routinely leads to these priceless stories, many of which are nevertheless too tangential to make it into the dissertation. One of my favourites relates to one family’s story of their experience of the panic caused by the Japanese attacks on Australia during WWII. My informant reports that a hysteria swept through the local community, who feared that their small rural holdings would soon be overrun by invading forces. My informant’s father, convinced that Doreen was soon to fall, ordered his family to pack all of their belongings and flee to the safe haven of… Strathewen. Nonlocal readers probably won’t understand why this story is so priceless: Strathewen is just down the road – some 18 km away from Doreen: it’s unclear why relocating there would have provided any greater safety…

One odd side effect of collecting these sorts of stories from older community members is the palpable afterglow of gratitude toward the US for its timely entry into the war – an afterglow that extends to encompass one somewhat awkward American researcher, trundling around with a digital recorder to capture this kind of oral history… There is a strong, sustained sense that the US cares deeply about Australian security – a belief that overrides even some often intensely critical opinions about the current US administration.

[Note: Image from Australia Under Attack, 1942-1943 - this site posts some fantastic artwork and documentary material from this period, and is well worth a browse.]


Get every new post delivered to your Inbox.