[Ed. 9 September: Now that events are unfolding a bit more slowly, and people have had a chance, for the most part, to learn about the basic facts, I’ve moved my on-the-fly updates to the bottom of the post, so that the original text is easier to find. I will try to update all the broken links next week.]
Original Post
A couple of people have sent me the link to this debacle of two researchers attempting to study what they call the “Cognitive Neuroscience of Fan Fiction” (further historical background here and here, collated links there, and information about the original research (which somehow doesn’t get around to mentioning that the research is designed – not for academic publication – but for a popular book whose working title is Rule 34: What Netporn Teaches Us About the Brain) in the researchers’ background information).
As someone looking on from outside the fan communities directly involved in this mess, the whole thing unfolds something like a live action version of the phenomenon Justin Kruger and David Dunning discuss in their “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments” (1999, Journal of Personality and Social Psychology, Vol. 77, No. 6., p. 1121-1134).
Kruger and Dunning are interested in whether, below a certain level basic competence, it becomes very difficult for people to improve their skills – because they are, in fact, too incompetent to be able to tell the difference between competence and incompetence in the first place. They take as their point of departure the story of hapless bank robber McArthur Wheeler who – some of you will remember from my previous post on this article – robbed two banks in broad daylight without any disguise and, when arrested almost immediately based on the bank security footage, burst out: “But I wore the juice!” Mr. Wheeler was evidently under the impression that, by rubbing lemon juice on his face, he could conceal himself from security cameras (Kruger and Dunning 1999: 1122).
Assuming this mess is not some sort of elaborate research-themed performance art, or the result of a revenge-fuelled identity theft, researchers Ogi Ogas and partner Sai Chaitanya Gaddam are trying their best to demonstrate to the world that they are something like the academic research equivalent to Wheeler. They have blundered into an online community whose members write and read, among other things, erotically-themed fan fiction, and have presented community members with a poorly–designed questionnaire (now taken down, but for a while being modified on the fly as people lined up with complaints about the research design – participants have posted screenshots and a text version of the survey after its initial modifications – note that a number of the final option responses and some other warnings and qualifications seem to have been added in response to criticisms of the survey in its original form – the modifications are often palpably different in style from the original text).
Among many other problems, the questionnaire asks respondents to provide sensitive information about sexual habits, desires and fantasies, in a setting where the questionnaire could be accessed by minors, without – as far as I can tell – having vetted the research design with their university’s IRB (the researchers are currently being hounded across several websites with demands to answer the question of whether they did, in fact, submit the project for ethics review – while answering other questions, they have steadfastly ignored this one: quick suggestion that, if the researchers don’t mean to imply the answer is ‘no’, then they should probably address this question very explicitly, very soon). [Side note: there’s a nice critical discussion of the limitations of IRB’s that’s been sparked by this whole mess: here.]
In the ongoing discussions now sprawled across a number of sites, the authors continue to dig this initial hole deeper by using terms regarded as offensive by members of the community (and, in one case, defending this because these are the terms that are standard in the sex industry – as Marx might say: !!!), by blithely demonstrating their own participation in widely–criticised assumptions about sexuality and presuppositions about gender, by demonstrating ignorance of basic facts about the community that could be gleaned from a quick skim of community sites, and by insisting on knocking back well-reasoned and absolutely on-target critiques by arguing that they are not doing “social research” and are not actually interested in the community anyway, other than as an example of a much more general phenomenon (these last, the researchers seem to believe, get them off the hook on ethical and basic research design requirements).
I’m not going to write my own critique of this mess: the community has already done this, eloquently, thoroughly – and, given the circumstances, with admirable patience. I am always warning my students when I teach research methods that something like this can happen – that this is why I’m so harsh on their research designs. Welcome to my new case study. I’m serious. I’m thinking of assigning parts of this trainwreck when I teach research methods next term.
I’m posting on this mainly because I’m wondering why the researchers have not apologised far more abjectly for having blundered into a community so ill-prepared – and possibly having ignored basic legal requirements and professional ethical standards governing their research. I am wondering if they are simply failing to register how devastating are the critiques being made of their work – perhaps because they are assuming these critiques have arisen defensively, due to strong affective attachments and loyalties within this particular community – or perhaps because they have “othered” this community so much that they aren’t sufficiently open to how badly they are being schooled here. Sai Gaddam’s university website suggests a potential vulnerability in this regard – let me quote from the source (apologies: I owe a poster in the original discussion a hat-tip for drawing attention to this, but unfortunately I’ve lost track of the comment – if you want to make yourself known, I’ll add a link):
My research interests have evolved over the years I have spent in the Ph.D program, but my derision for my subjects remains a constant. Well, not really, but this quote does make me smile.
The individual I chose as my principal subject for the experiments … was an old toothless man, with a thin face, whose features, without being absolutely ugly, approached ordinary triviality, and whose facial expression was in perfect agreement with his inoffensive character and his restricted intelligence.
The Mechanism of Human Facial Expression — Guillaume de Boulogne
So, for what it’s worth: I don’t belong to this community, but the criticisms being made of your ill-conceived research are excellent. Listen to them. You have tried wearing the juice. They’ve seen through it. It wasn’t the disguise you hoped it might be.
Updates:
[Ed. 7 September: Still no time to update the broken links below, but wanted to point to the discussion at metafilter, for those interested. ETA: and Neuroanthropology weighs in! – Twice!]
[Ed. 4 September: If people aren’t aware, Ben Goldacre from Bad Science has referenced SurveyFail on Twitter, linking here and also to Alison Macleod’s fantastic overview at The Human Element. Rushing at the moment – apologies for not responding yet to comments.]
[Ed. 4 September: Another day, a few more broken links. Ogi Ogas and Sai Gaddam seem to have had their websites removed from Boston University – not surprising, given the report that they are not affiliated with the university for purposes of this research project. Gaddam’s blog has also been made private. The links I have below off their names therefore no longer point anywhere. Again: my schedule’s too hectic to fix this right now, so just noting the problem. Some limited information about Ogas is included in his Wikipedia page, as a backup link… If the old Boston University pages end up being included in any of the screencaps collections currently being collated online, I’ll restore links to those once I have time.
For folks interested in legs, this post has been picked up at Josh Jasper’s blog at Publisher’s Weekly, as well as at Alison Macleod’s the human element. Macleod’s blog has a very clear overview of how the whole thing unfolded, as well, for folks new to this whole mess and trying to get a sense of what happened.
Broken link clean-ups still days in the future, I’m afraid…]
[Ed. 3 Sept: Folks, just a note that the researchers have taken down their site – after an amazingly offensive final blowup that, honestly, must be seen to be believed… This will break a lot of the links I’ve posted below. I’ll try to clean these up later, but for the time being, there’s are a number of good summaries of the whole incident – now christened SurveyFail – see especially Yonmei’s post at Feministsf.net, as well as a report of a response from the IRB at their university, which has disclaimed any affiliation with the project and asked the researchers not to use their uni emails or web addresses in conjunction with this activity. (My favorite part of the linked IRB discussion was the report that, when the IRB office was contacted directly: “Their exact words ‘I had a feeling it would be about that.'”) Links cleanup might have to wait a couple of days – schedule is awful at the moment…]
Like this:
Like Loading...
Related
Rough Theory 
I wore the juice is one of the best lines I’ve ever heard. It is the best excuse, and I am now going to use it for all tight spots. Thanks. You’ve changed my whole life.
Thank you for your contributions; it was heartening to have somebody with scholarship and expertise in the field weigh in.
Long-term, rm over at LJ is collecting screencaps of Dr. Ogas’s postings; within a couple of weeks, I expect much of what he said will be available in that form. http://rm.livejournal.com/1701259.html
People are carefully capturing all of what was said in the shaggirl posting comments, in particular.
Hey roger – it’s a priceless story, isn’t it? 🙂
Hi Jonquil – thanks for posting (and sorry you were held in moderation – it’s an anti-spam thing, and should only happen the first time you post).
There were plenty of people in the original discussion with scholarship and expertise in the field – including people with qualifications in psychology, the biological sciences, and neuroscience whose expertise is far more on the mark than mine. The reason I piled on as well was because there had been some suggestion that this mass of well-thought-out critiques could somehow be dismissed as some kind of overreaction of fandom to outsiders or academics as such. I figured the one thing I could add was that the folks who were saying that they were angry as academics were having an absolutely plausible reaction: anyone looking in with any sort of interactive research training would react critically to this.
Thank you for the screencaps link as well – I’ll update the post here when my schedule is calmer in a few days, so that it’s easier for people to find examples of what I’m describing (although the SurveyFail entry linked above gives a better account than anything I’ve done here).
It’s awesome to see someone outside the community recognize how incredibly smart and effective that community’s critiques are. I agree, Ogas and his colleague don’t realize quite how well they were just schooled!
I am quite possibly the person who first posted that lovely quote from Sai, but I don’t require attribution.
As jonquil and Liz said, above, it is quite nice to see someone from outside the media fandom community recognize how intelligent and blistering the critique has been. Cheers!
This post has been included in a linkspam roundup
Unfortunately (http://shaggirl.livejournal.com/190980.html) it appears that the pair are convinced that the juice is working just fine – or that at the very least they’re going to get away with the loot.
That’s fantastic. As an IRB member and social scientist I watched the whole thing unfold with horrified fascination. I didn’t speak up because I keep my professional and fan lives completely separate, but I kept thinking, “Oh my God, these people are the reason we can’t have nice things.” Because for every ten thousand inoffensive, minimal-risk pieces of social science research and evaluation that have undergone a tangle of red tape and misery to get through IRB, there’s one set of asshats like this fucking it up for the rest of us by calling their ignorant trolling “research.”
Hi Liz – unfortunately, as Sam has pointed out, I think that knowledge will be slow, if ever, in coming…
par avion – thank you – happy to hat tip, but can just leave it here in the comments if that’s better? He has lots of other questionnable stuff on his blog… 😦
Sam – yeah, have just been over there and commented for what it’s worth 😦
anon – yes – it’s one of the reasons I posted – I don’t have to juggle identities in the community, and know it can be complicated when you do need to. But I had exactly that reaction: that this is what IRBs are, unfortunately, for.
In this case, though, it’s sounding as though the issue is more that they misrepresented the university affiliation – so, rather than flouting the IRB process, that process probably doesn’t apply to them. It’s just that they gained the trust of a community – and in particular the trust of the folks who initially supported the “research” – by implying the project had some sort of academic standing that, apparently, it doesn’t have…
Normally where the IRB process leaves off, professional ethics standards pick up – and would apply if professional standing means anything. I haven’t poked around to see whether they are members of any professional association that would have standards for their conduct – even if they are publishing a popular book?
“Oh my God, these people are the reason we can’t have nice things.”
Exactly. Or more aptly put: These people are the reason *I* have to fill out more paperwork.
I am not certain who their dissertation advisers were (while I’m sure they’re not involved), they would be the first people to contact, as they’re generally listed as recommendations for wherever they try to get a job next (Though, I’m not sure they’re going to try and get a job in academia after this, they may just go into pop! psych literature).
Do not miss “neededalj”, who IS a cognitive neuroscientist, take on what the doctors were really trying to do, and why it is horrifically unrelated to science. The survey, she strongly suggests, was never the end goal.
Hi Jonquil – I’ve been worried about this sort of thing as well – either that they might have been after answers to a tiny subset of questions, or that they were in fact after the mobilisation against them (although to be honest I’m not convinced they’re skillful enough to orchestrate a reaction in any useful way). Again, if they had been undergoing a proper academic study, the IRB process would have scrutinised very, very closely any form of research that doesn’t declare openly to participants what the research is for – informed consent requires it. It’s of course possible to get studies done that involve deceit – but it’s very hard, and there have to be clear benefits and no alternative means of collecting the same information.
Hi Chameleon – yes – I mean it’s entirely possible they’ve already written off any notion of academic careers in the narrow sense, but there still may be professional associations that are valuable to them. Also, I thought Sai’s webpage indicated he’s doing a postdoc, so there should be some uni affiliation there, even if not to BU.
Thank you for posting this.
I really really hope that the “data” they collected from the survey is thrown out and never used. If they carry on with their “research” after all the crap we’ve exposed and try to pass off that data as legitimate, if they attempt to get this book published, I won’t be the only one breaking down their publisher’s door.
I feel personally violated by this. Like someone has barged in on a very private and intimate part of my life and exposed it to the world. There are reasons why fandom tries to stay under the radar… And for them to just come in and mislead us… there are no words.
If they do try to use their data as legitimate, is there anything we can do to stop it? Okay, so we schooled them, they’re completely illegitimate and they seem to have backed off… but now what? Even if they’re not in conjunction with UB or any academic boards, the info they collected could still be extremely damaging.
Thank you so much for the idea of using this as an example of the bad consequences of having a bad relationship with the people you hope to study. I teach research methods to grad students in a social science, and until recently also directed our large graduate program, and I do hear a lot of complaining about IRB requirements. But, yeah, this is why we can’t have nice things.
There are only so many times I can tell our students about the Tuskegee experiment – this is going to be a whole lot more fun.
Pingback: The curious case of the game show neuroscientists, or how NOT to research an online community « The Human Element
Hi Lola – Yes – I had the same thought. 🙂 Most of my students find things like Tuskegee – or even Milgram – just too distant from anything that can picture themselves doing. Whereas I suspect many of them could more easily visualise wandering into an online community and all hell breaking loose… Particularly if they’re involved in any online community themselves and a sense of how communities can mobilise. (This was an exceptionally articulate and credentialled community, admittedly, but nevertheless – this fail is closer to things that are intuitive for most students.)
Hi Stitches – Yes 😦 I don’t have the legal background to know what sort of formal recourse is available in this sort of situation. When researchers are clearly bound by IRB requirements, it’s much easier to know what to do (for me – presumably others have a much better sense of what’s possible for more journalistic works, or for neuroscientists working in the field outside of traditional academic contexts – what sorts of ethical standards apply, and what one does when these are violated).
The failure to protect minors from potentially traumatising imagery and/or violations of duty of care to minors might give some avenue for objecting – but I’m unclear what the legal framework for raising these would be in the US (or, for that matter, in Australia, in a situation like this).
Privacy laws might potentially offer some recourse – these might apply, for example, to collection of information by the book publisher? Not sure whether Ogas and Gaddam could be considered agents of the publisher? Or whether privacy laws apply?
If they are involved with a professional association, that association might provide a means of recourse. Gaddam was listing himself as being a postdoctoral researcher on his website, last I could access it (Boston University has removed their web pages – which makes sense, given that it has claimed they are not affiliated with the university other than as graduates). If Gaddam is a postdoc, then he might be bound by some other university’s IRB requirements, even if it’s not BU. And of course there’s always the possibility of placing moral pressure on the publisher (Dutton), even if formal legal pressure is not available – I don’t know enough about the publisher to know whether they’re the sorts of folks who would respond to this, or whether they’d enjoy being able to describe this as a “controversial” text…
There’s obviously extensive discussions on fansites about this issue (haven’t been able to be online much since I posted the original comments above, so I’ve not been able to track this closely) – but people there are collectively more likely to have a sense of what’s possible…
N Pepperell–
Though I didn’t screencap, Sai Gaddam’s webpage said he was doing his postdoc at a corporation rather than at a university, so he is not bound by IRB regulations either. (It might have been Hewlett Packard, but I’m really not sure.)
Thank you for weighing in on this debacle; it’s nice to see outsider perspectives.
If/when you boil Ogi Ogas’s mind-bogglingly varied and egregious mistakes into lesson materials, would you be willing to share them on this blog? I would be very interested in seeing how you turn this into a case study/lesson plan.
Great summary. I wrote this up for my community, who are professional market researchers/social researchers as well as academics. I think it’s important to say that anyone who knows about good survey design would have come down on this like a ton of bricks, because most of us actually have some standards…
If I were making it into a case study, the other element I’d highlight is the nature of the research question itself and how on earth the researchers’ goals were ever going to be met by their chosen methodology. Unless it was always going to be pop science handwaving.
If readers are looking for other examples useful in teaching research ethics in the social sciences, Lisa Wynn’s site at Macquarie is an interactive training module that might be of interest.
Hey R Tushnet – Many thanks for the reference. I should also have been less shorthand when I talked about the issue above: the issue, of course, isn’t that there is a shortage of cases – some cases are “classics” that show up everywhere, but there are any number of less prominent examples that can be taught. It can be difficult, though, to get students to “grok” why – in the phrase used above – they “can’t have nice things” – why there are so many restrictions on their work, which they don’t perceive to involve major risks. This particular situation sort of dramatically illustrates the potentially high risks to other people – but also to the researcher who starts data collection unprepared.
In many ways, the sorts of mistakes made on the questionnaire aren’t all that dissimilar to the mistakes a novice research in one of my courses might make – but this is precisely why they are taking a course, so they can make those mistakes in class, where they won’t do any real-world harm. Looking for ways to get the long “sandbox” period of research training to make sense to students – communicating that this isn’t some arbitrary or bureaucratic thing – can be challenging.
Sorry – this is less a response to you, than that your comment gave me a chance to articulate some things that have been running through my mind since the exchange above, but that I haven’t had time to write down.
Hi Alison – Your post was really fantastic (it’s now been picked up by Ben Goldacre, as you no doubt know). I had wanted to summarise the history when I originally sat down to write this post, in a way that would make this more intelligible to people who haven’t been following it unfold, but my schedule at the moment is insane, and ultimately I decided it was better to get something up, than to wait until I had time to do a proper overview. When I saw your post this morning, I was glad I hadn’t attempted it, as you’ve done a much better job than I would have. Thank you.
Hi neededalj – I’ve been meaning to post links to your fantastic posts on neuroscience, research and this mess (as well as sabrina-il’s on psychology) – apologies – I’m drowning under other obligations and lagging badly…
But yes: happy to share when I’ve put together materials for my students (this will likely be over the Australian summer, when I can take a breath and rethink it from a pedagogical point of view). I’ve noticed on Ogas’ Wikipedia talk page that this post and elf’s work have been mentioned, but that the editors don’t want to modify the article to mention this until something has been formally published. This makes me wonder whether, in putting something together for students, I should also put something together for a methods journal. My worry would be that the last thing anyone wants, after this mess, is another outside researcher fumbling around writing about the situation. So perhaps best left alone… But certainly happy to share the teaching version…
Hi starlady38 – Yeah, I have a memory of Hewlett Packard as well. If it’s conducted as a post-doc, however, there would still probably be formal supervision of some kind, and there might well be a university involvement. But yes: it sounded private sector.
“My worry would be that the last thing anyone wants, after this mess, is another outside researcher fumbling around writing about the situation. ”
Honestly, if you want to do it, I’m sure you could explain your motivations and put up a formal proposal on LJ requesting permission from the individual participants and explain that you aren’t studying our psyches but the Dr.s behaviors. You’ve earned a *lot* of credit within the community. If you did something like say “If you’re comfortable having me mention your contribution, please let me know”, I bet more than one participant will smile so wide you can see her razor fangs from space and say “Go ahead!” (If you do this, let me know, and I’ll boost the signal.)
There will, of course, be some people saying no on principle — but many, many of us are grateful to you and would love to help.
RAZOR FANGS FROM SPACE.
I would love to see these cretins written up as an object lesson. You seem respectful, thoughtful, and knowledgeable enough to listen to and talk about the community in which this took place without Othering us or condescending to us, which is a sadly rare state of affairs outside the community itself. Go you, and I will signal boost for you as well.
Just a quick note to furnish an answer to the questions up-thread about Sai Gaddam’s post-doc, this is what was written in his (now deleted, but still google-cached) B.U. website:
“I am a recent graduate from the Department of Cognitive and Neural Systems at Boston University. My research was focused on the modeling of physiologically plausible neural models of vision and memory. I applied these models to problems in image understanding and machine learning.
In my current stint as a SYNAPSE post-doctoral researcher, I collaborate with Hewlett-Packard to extend and develop these models on hardware using nanoscale (very very tiny) memristor devices.”
I (and many others) would love to see someone write this up for a methods journal or other publication.
I have a lot of screencaps and pages pulled from google’s cache. Sai’s BU website said:
I am a recent graduate from the Department of Cognitive and Neural Systems at Boston University. My research was focused on the modeling of physiologically plausible neural models of vision and memory. I applied these models to problems in image understanding and machine learning.
In my current stint as a SYNAPSE post-doctoral researcher, I collaborate with Hewlett-Packard to extend and develop these models on hardware using nanoscale (very very tiny) memristor devices.
Google cache link for research/post doc page.
Yahoo cache link to Sai’s MLLinks page. And the Google cache for same.
Thanks for the feedback – when I saw this unfolding, it hit all my buttons – bad science and bad methodology with no regard for respondents’ welfare, and that’s even before you get to the offensive parts.
I did wonder whether it was a meta-experiment on the acceptability of academic qualifications and psychology-speak. Don’t know if anyone has screencaps of the original FAQs (the ones with Mona Lisa filtered), but the tone of that struck me as deliberate obfuscation. In the end, I don’t think it’s that.
I am still horribly fascinated by what they intend to do with the far right, and it disturbs me that their modus operandi involves researching ‘unsympathetic’ subgroups – in other words, my hunch is that they didn’t expect people to have the power/confidence to complain, or that external groups would be sympathetic to that complaint.
I much appreciate this post–as a science grad student (albeit in a field only related by the fact that I use statistics), I’ve been getting really frustrated by people who think the outrage is an anti-science/anti-academic backlash, and dismiss all the fan scientists & academics who’ve spoken up because we don’t use our real names. So it’s great to see a post like this from someone outside the community who does work with human subjects.
Alison, I’ve been wondering someone like that, too, but given the book deal, it seems unlikely–would they be able to get a publisher to announce a false book topic? Wouldn’t it have been easier to not announce the book at all until they were done with the survey?
I don’t know. I’m skeptical that they’re doing what they say they’re doing, but I’m not sure how far from it they can go without moving outside the realm of the book topic.
(Thank you for your excellent post, too, btw.)
@N Pepperell: Add me to the list of people who would be very happy to see this whole mess written up in a journal article. The more Ogi’s mistakes are chronicled in public, searchable spaces (as long as fandom is treated with respect) the better, IMO. I look forward to seeing your lesson materials once you have a chance to get around to it.
@Alison: I do think that Ogi and Sai were deliberately obfuscating, but not in a way that meant they were researching something else. There are now some comments on my posts that talk about this, and I believe that they thought by throwing around a lot of science terminology they could intimidate people into thinking they knew what they were doing and not critiquing them on their own turf. Clearly, they were wrong.
Everything in they wrote up, including the official FAQ, took the concepts they were attempting to use down to their (flawed) basics, but then dressed those basics up in technobabble. I never, in anything they wrote, saw them get into the nitty-gritty of what they were *actually* going to do in a task-oriented way.
Pingback: links for 2009-09-04 « Embololalia
This is my best guess as what they were after:
They left enough information pointing to what they wanted to show. Much of the survey was a ruse as was the intentional misleading introduction they gave about the Mona Lisa, etc…. And they were not interested in fandom’s response at large nor were they were they interested in attraction or erotica for the general female fandom population. What were they interested in? As Ogi mentioned in his response to my question of Why Slash (available in some of the links you have above), his response was telling.
My question: A simple question: why are you focusing on slash in your survey and not just relationships in general for fanfic without regard to who the pairings are? Have you found a difference between those who primarily read and write slash compared to the general population? We’re not privy to your research with men but what you’ve mentioned here, which is quite intersting, it doesn’t sound like it’s so specialized. Okay that was more than one question.
His response: Thank you for your questions!
Well, slash is kind of the female equivalent of the straight male interest in transsexuals. That is, the opposite of what culture would predict. So it probably reflects a more direct subcortical effect. Also, there’s already data out there about romance novels we can use, which probably overlaps with relationships in fan fic, but we do have a few questions that aren’t specific to slash. Maybe we’ll have more in the next round.
This is where he showed his hand and then brought the topic over to his LJ when he got slammed for it. He wasn’t interested in most of the information he was gathering, but there were targeted questions in his survey that would get him the info he wanted no matter how poorly the rest of the survey was written, as long as those questions were answered he was happy. He was really only interested in heterosexual females that wrote and read slash, specifically hard-core kinky slash.
And my guess is he was expecting to see a big difference between heterosexual females who didn’t write slash and those who did on the hidden in the weeds question of feeling guilty about sex.
His email to another member points in this direction (also in the links). He defends is comparison of women liking slash to straight men liking transsexuals because “some deep sense of pleasure or satisfaction ultimately rooted in subcortical circuits” compels us to seek out slash/transsexuals despite fearing exposure to society at large.
And in his mind, slash went against the Cultural norm and would be equivalent of heterosexual men who viewed transsexual persons. The obvious problem being that it was his Cultural norm.
So basically he was just looking at females who wrote and read culturally accepted material and comparing them to females who wrote slash (not culturally accepted). He would do the same with males but would insert those who viewed transsexual persons. He then expected the two later groups to be similar in their physical responses. And, unfortunately, if slash writers-readers did not rate themselves high on the guilt factor or a few other questions, but the physiological data did correspond with what he expected, then he could say we were delusional.
And if things happen to work out as he predicts, then he can show that different types of netporn (Rule 34) has an effect on the brain by lighting up different areas of the brain in comparison to more “normal” folks. Internet is bad……
Somewhere along the way he will be surveying very conservative folks to get their reactions. I’m not sure where exactly this fits in.
Yes, the survey was horribly designed and yes both of them were very insensitive to the situation they were causing.
Again, this is my guess as to what they are attempting to do.
Hey folks – thank you for your comments and discussion, and apologies again for letting comments through without immediately replying – I’m online only sporadically at the moment. I may not have time to reply very adequately, but let me make a start…
Beth and par avion – Aren’t caches and screencaps a wonderful thing? 🙂 I’m wondering at the phrasing above – “collaborating” with Hewlett Packard. This probably just means doing the postdoc there, but could conceivably mean that there is a partnership between a university and Hewlett Packard, and so it’s best described as a collaboration, rather than “I’m doing an internship with HP”, for example… This explanation of SyNAPSE describes the relationships between private, public, and university sectors in the following way:
This suggests that Boston University could be involved with this aspect of Gaddam’s current work. But I would think that a respectable private organisation would care about the research bona fides of their staff, even if BU is not associated…
I don’t have time to dig further at the second, but it might easily be possible to dig up better information than I’ve found just now. (Apologies – my work is very heavy at the moment…)
Jonquil, anon, par avion and neededalj – I would be very interested in writing this up – just sensitive to the complexities of the situation… I suspect that a number of other teachers of method would be interested in this as a case study, for the reasons discussed above – it’s a good cautionary tale for how badly things can go wrong…
Thank you for the offers to signal boost – but I’m wondering how best to approach the issue… It seems a bit similar to what’s just happened, for me to create my own site and ask for feedback on my research idea… ;-P It might be that it’s best raised in a forum that already exists in the community itself – as a working idea that can be discussed as a possibility. I’m not sure and am open to suggestions…
If it looks like it has support, I’d then need (ahem!) to go through the IRB process here (assuming the research would involve, not just reading documentary materials, but speaking with people involved). So it would take some time to get underway… But I’m open to suggestions as to the best way to get some feedback on the time – also open to suggestions on timing – I don’t want to distract from other things going on around this right now…
@N Pepperell: I would be willing to host a discussion of the research possibilities with the fandom community on my livejournal. I created it solely for the purposes of entering this discussion, and I intend for all of the material there to be related to the discussion of SurveyFail and any similar matters that come up.
Can you see my email when I comment? If you would like to discuss this further, let me know.
Alison, Carmarthen, neededalj, and romaine24 – Yeah, I find it hard to tell where to draw the line between deceit and just sheer inability to grok the context, resulting in an inability to communicate clearly what they were trying to do. Certain things fall firmly on the deceit side of the continuum: implying (and sometimes claiming outright) a Boston University affiliation for this project, not declaring commercial interests in the project or explaining clearly how the results would be published – and probably a research design involving distracting people with a mass of questions, when only a few were actually relevant (the few they kept mentioning they were keeping constant across the various sites they claimed to study – like the “do you believe in true love?” question). Of course, it’s possible to get IRB approval for a research design involving the latter technique – it’s just that there would then be a level of oversight and cautious research design that is not reflected here…
Certain things might be deceit, but might also just be a matter of not knowing how to get the point across. Of course, this also could have been minimised by a proper IRB process, where you would need to have other people vet your plain language statement and consent procedures – people who would be checking the statement against the actual aims of the research, to see whether what you’re telling research participants expresses what the project is about.
At least one of their core questions seems to relate to areas of the brain involved in “transgressive” activities. They seem arbitrarily to have decided – based on their own notions of what is socially or culturally sanctioned – that male production and consumption of materials related to transsexuals would be equivalent, in terms of transgressiveness, to female production and consumption of materials related to male/male sex.
Putting my social theory, rather than my methods, hat on for a moment: as many many (many – very very many) people pointed out in the outpouring that followed this astonishing comparison: they seem to be assuming that “society” or “culture” is a monovocal thing, such that, once they’ve identified what is “transgressive” from the standpoint of a particular part of social/cultural experience, then any behavior that “transgresses” that social/cultural dimension must somehow be “anti”-social – must come from some space other than the social. I think this is how they are assuming they can draw conclusions about the brain – because, the “reasoning” seems to be, what else could it possibly be, once the only dimension of the social they are willing to acknowledge has been ruled out. They think they have eliminated social explanations, leaving no choice but to leap into the deep structure of the brain. (I think I could make this case more convincingly with a trawl back through their comments, but I haven’t time right now – apologies…)
Of course the absolute hordes of people asking “Whose culture?” were pointing out the obvious: that no one’s culture or social experience boils down to one set of experiences or perspectives alone – that “social experience” is itself multifaceted, lumpy, and contradictory – with the added irony that they were literally standing in a culture whose norms and communal practice suggested the possibility to practice – socially – communally – what they were trying to study, without constructing this as a transgressive practice.
The researchers seemed to take things like pseudonymity as evidence of shame or internalisation of the notion that the community is engaging in transgressive conduct. That’s one option. Another is simple realpolitik: there can be real world consequences from any sort of online activity, due precisely to the conflicting expectations and requirements of the different layers of our social experiences. This doesn’t mean that one layer of social experience (because it has hard power, for example) gets to be called “social”, while everything else we do must be an expression of our “reptilian brain”… Among all the other fails, there’s just a basic fail, here, in thinking through the potential complexity and variance within what often gets talked about as the “same” social context. This isn’t even getting into the whole issue of the international scale of this particular community… (it just becomes endless, the flaws in this, doesn’t it?)
I’ve been secretly wishing the folks at Neuroanthropology would weigh in on this… Just for a perspective that works on both neuroscience and culture in a much more nuanced way…
But sorry – have meandered a bit. One thing I started to write at neededalj’s site earlier, and then realised I couldn’t post anonymously, was that I’m quite worried about a whole other potential for ethical violation – related to how often in various exchanges they trumpeted how exciting the net is, as a datamining opportunity for private information people would not usually disclose on this kind of mass scale. I’m concerned about whether, now that this has fallen over, they might simply try to take what they’re after by mining (with their own assumptions and interpretations) the content produced by the community? Perhaps with the intention even still of using their questionnaire data to “support” various interpretations they have of what this content means?
The issue of internet content is complex due to the strange character of much of the material being publicly accessible, but obviously written for and by a community of a sort that historically would have been private.
It can be okay to do, say, structured observations of public spaces, or research into public meetings, without consent – in the assumption that the activities are taking place in a public setting and that the participants are either anonymous or deliberately engaging in acts they intend to have witnessed publicly. (There are limitations on this, but I’m not trying to discuss the issue in fully adequate detail here.)
Certain kinds of activities can take place in public spaces, however, and still be regarded as private activities for purposes of research ethics. If you decide to make a phone call from a public telephone booth, for example, this is an activity in a public setting – other people can potentially overhear what you say. But you are engaging in a private act – you aren’t intending for everyone walking past to listen in. A researcher therefore can’t camp out behind a public phone booth, lurking and making notes on the conversations, without consent, and argue that it was okay because it wasn’t a phone booth in your house.
The internet – like all other public spaces – involves both sorts of publicness. There are some sites where the point is to make a public noise – where there’s a town meeting sort of space that is attempting to reach a broader public sphere. But there are lots and lots of basically private interactions, as well, that just happen to be taking place in this sprawling public space of the net. As far as I’m concerned, datamining in one of those private spaces is sort of like eavesdropping on those public phone booth conversations.
And I’m quite worried that – aside from everything that’s been explicitly disclosed as part of this research projects – there is also a background intention of doing that sort of eavesdropping. You don’t generally describe analysing questionnaire results as “datamining” – and yet they routinely talk about the wonders of the internet as a source to datamine. I’m very nervous about what this means – about how they might intend to analyse, say, the images in the community’s literature, or the discussions on forums…
Yeah… think I’m gonna go be queasy somewhere for a while… 😦
Hi neededalj – sorry – our comments crossed in the posting. But yes – that could be a good idea, since you’re posts have been examining the research implications in such detail. We can see what others think, and then we can email? (my own email address is linked from my name in the original post)
@N Pepperell: Oops, do I not have anonymous commenting turned on? I’d never used lj before from a production standpoint and literally made up the whole journal so fast I didn’t stop to look at any of the settings…I sat on my inbox waiting for the confirmation email so I could start commenting on ogi’s lj in another futile attempt to get through to him about his…research failings.
I’ve been so focused on the neuro/scientific side of his problems I haven’t let myself think about the ethical implications on a larger scale. It’s not a pleasant set of ideas.
Ha. Cross posted again. Yes, absolutely. Once other people weigh we can definitely email.
Alison @26: I’ve got screencaps of the FAQs but even zipped they come to 3.4 MB. I’ve emailed them to rm at livejournal.com who’s collating all screencaps but if you need them contact me via my dreamwidth account (legionseagle.dreamwidth.org) and I shall arrange to send them.
Pingback: Sex, Lies and IRB Tape: Netporn to SurveyFail « Neuroanthropology
^^ points up ^^ You got your wish! I was just coming over to tell you.
You should mirror those links before they expire from the Google Cache.
Hello!
I’m just lurking around reading this strange and fascinating saga… but am delurking to urge you to PLEASE do analyse and write this up; and please do archive it all before it disappears. It would be so great to get a clearer discussion and explanation of what this is about and why it happened, written by people who understand both the neuroscience and the social science.
Hey folks – sorry for the tardiness in responding – high workload at the moment. On the issue of grabbing things from Google cache – I have some things backed up here, and others are also collecting and posting collections of screencaps. On writing an article, hopefully soon I’ll impose on neededalj to host a preliminary discussion, mainly with the intent of exploring whether this would be a good idea, and potential concerns it might cause. More on that very soon…
Pingback: Nature/Nurture: Slash To The Rescue « Neuroanthropology
Pingback: The Random Planet · Blog Archive » Good things all ’round.
Pingback: Surveyfail: No One Expects the Fannish Inquisition « Tara on the 505
Pingback: Genreville » Blog Archive » How Not to Engage/Enrage Fandom in One Simple Lesson
Not sure if you decided to use this case as an example or not but I’d thought I’d let you know the book will be released on April 28, 2011.
http://www.amazon.com/Billion-Wicked-Thoughts-Largest-Experiment/dp/0525952098/ref=sr_1_1?ie=UTF8&s=books&qid=1286845706&sr=1-1
A Billion Wicked Thoughts: What the World’s Largest Experiment Reveals about Human Desire
Sai Gaddam (Author), Ogi Ogas (Author)
We will all be interested in what their definition of Experiment means. 🙂
Hi romaine24 – I’ve used it as an example in teaching my research courses this past year – and had been looking periodically to see whether the book were actually going to be published, but hadn’t seen this. I had originally thought about putting together an article on the issue as a case study in what can happen when researchers wander unaware into online (or, for that matter, any…) communities, but have been caught up in other unexpected obligations arising from a new job – it’s also something I’d want to approach quite carefully, since a lot of people felt understandably burned by lending their trust to researchers over this… I was a bit worried about coming across as one more person trying to mine the experiences of the people involved for my own extrinsic reasons… But this really was an amazing debacle, and it really should be written up… Is the book available for review purposes yet?
Hello again,
I haven’t seen anything about its availability beforehand. However, it’s being listed under Psychology/Human Sexuality so it might be possible for an academic (you) to get a preview.
I found it interesting that Ogi Ogas was not the lead author as he was the front person involved with those in fandom.
Fandom is actually a very friendly place, but as its not well known in the general public, folks get their fur up when trying to be labelled or grouped by traditional society means. It will be interesting to see what they have to say and I do wonder if they think we’ve forgotten about them…we haven’t.
Good luck and I’m so glad to hear that you did use this debacle as a learning point. I would have enjoyed it in my Experimental Design courses (way back when).
Congratulations on your new job.
It might be available for review – there’s one reference (Acrobat file): Dutton 2010 Hot List.pdf. It would be nice to have a response before it’s even out.
Hey there – Apologies for the delay responding – limited time online at the moment. There’s a longer promotional bit on the book here, amongst other places. It doesn’t sound like the argument has become more complex with time – e.g.:
I’ll poke around and see if it’s possible to obtain a review copy – once the book is out, this generally isn’t difficult. In advance, I’m less sure…
Pingback: true, but not for you « Letters from Titan
Pingback: Learning new patterns « beyondthetext
Pingback: Good things all ’round. – The Random Planet