December 1, 2006
Posted by on
I’ve been watching with some amusement the evolution of Scott Eric Kaufman’s meme experiment, which I mentioned here the other day (anyone who hasn’t yet linked back will… I don’t know… have some horrible chain-lettery thing happen to you very soon…). For those following from the sidelines, Scott has posted an update on the quantitative success of his experiment. Personally, I’ve been following the unintended qualitative dimensions of the project – specifically, the number of people who’ve evidently decided to help out because, well, they’re so damned irritated about how badly they think the whole thing has been designed.
Sarapen opened with a thoughtful and even-tempered critique (including a literature review, even), but not all participants were as kind. Scrolling through the comments on Scott’s blog, you see a very large number of methodology criticisms, pointing Scott to things he hasn’t controlled for, noting problems in capturing the relevant data, criticising what he believes the data will show, complaining that he hasn’t sufficiently defined his hypothesis prior to the experiment, accusing him of stacking the experimental deck, arguing that the experiment can’t possibly be expected to follow the course of “wild” memes, contesting the finer points of whether “meme” was the right word for this – and, my personal favourite:
Based on my experience with technorati, when they pick up a link can be highly variable and not well-correlated with the actual time that link is created (to the point of being off by days). Your methodology is already crap just on technical grounds, even before taking into account all the objections above. Try using a web bug or something like it next time.
I suggested to Scott that he put all of this criticism to productive use at the MLA conference – prove the value of internet academic discussion, by challenging his panel audience to see whether they can come up with as many reasons that his methodology is “crap”. ;-P Since making this comment, though, I’ve begun to wonder: perhaps we’re looking at the birth of a new kind of PhD student performance art – the methodology slam. Someone stands up on some obscure corner of the net, calls out their research methodology, and asks a friend to tell a friend… Perhaps the results can be submitted as part of the portfolio to the committee approving candidature – a new criterion before you can call yourself ABD…