Rough Theory

Theory In The Rough

Category Archives: Methodology

Beta Blocker

I know that L Magee has been working very hard recently on the development and debugging of the ontology matching software that will provide the empirical data for a dissertation on the Semantic Web. Being the kind, considerate friend that I am, and wanting to ensure that LM stays always in a state of good cheer through this intensive work period, I must draw LM’s (and everyone else’s) attention to how the beta is already garnering scholarly repute. From orange.’s interference: yes, LM, you have now been immortalised as:

LMagee, contributor to Rough Theory and author of the strangest beta I ve ever seen

The Theory Chapter Reloaded

I know this is becoming a bit of a regular rant… but I was thinking again this afternoon about how common it is for methods courses and textbooks to start with some kind of introductory “theory chapter”, which generally informs students that, before they begin any kind of research design process, they must:

(1) know their epistemological and ontological stance; and

(2) be able to position themselves in relation to a wide range of theoretical debates.

This is so common that I’m beginning to get a bit worried about how counter-intuitive I find it to be. I mean, I love discussions of epistemology and ontology – probably a bit too much ;-P – and I’m quite happy to position myself away in theoretical debates of all sorts. But I think it’s fairly safe to say that I would never start a research design course or text with these issues. I think it’s also safe to say that these are not issues that arose – in this form, at least – early in my own engagement with either research or philosophy. Am I that much of an outlier?

Amusingly enough, my main objection to this approach is itself ontological: there’s something about formulating the issue in this way – as though the researcher is some kind of disembodied consciousness, floating around in The Matrix, saying, “I need Theory – lots of Theory!” – and then out roll the shelves of high-powered concepts from the aether, from which the disembodied consciousness then selects whatever approach makes it feel most secure. What about the relationship of the theory to the object of analysis? What about the relationship of all of this to some underlying question? How do students make sense of and understand their theoretical choices, when this is how theory is presented to them?

Then there’s the pedagogical issue: maybe I overcompensate, but I tend to assume that most students – most peers, for that matter – won’t be as interested in abstract theoretical discussions as I am… Unless forced to start with these issues because the students are confronting them in assigned texts, I tend to sidle my way up to terms like “epistemology” and “ontology”, because I think it takes a bit of intellectual grounding for students to be able to understand why someone would care about what appear, on their face, to be rather abstract concerns. My experience has been that students find the concepts terrifyingly fuzzy – and that their fear isn’t assuaged by the tendency of “theory chapters” in methodology texts to rush past a definition of these concepts, and into long lists of competing ontological and epistemological stances one could conceivably adopt – all lined up in a row, in neat boxes – sometimes with light bulbs flashing beside them – as though people make a common practice of dealing with significant ontological and epistemological questions by trundling their conceptual carts down the theoretical aisles in some vast grocery store of human knowledge…

I know I’ve said this before – recently enough that I shouldn’t still be ranting about this topic – but my impulse is to start with something much more grounded – much more solidly within students’ experiential frame: with what students are curious about, where their passions lie. From here, they can begin to ask questions – and those questions will then, eventually, give them the basis for finding ontological and epistemological questions meaningful – and for translating their interests into something that might fall within the boundaries of academic research.

I realise that textbooks don’t have the flexibility I have in the classroom, to build a discussion around students’ questions and dreams… But still… Wouldn’t it be possible, at least in principle, for a text to talk about curiosity as the origin point for a research process? To sketch some examples (which surely wouldn’t be any more misleading that the text box versions of theoretical positions these texts already supply) of how particular researchers found their way to problems, which then teased and thwarted them into methodological strategies – and then to unpack the concepts of epistemology and ontology from there?

Praise by Design

Via Marginal Revolution, an interesting overview article on studies of the effects of praising children for their innate abilities (intelligence, uncultivated skill, etc.) vs. praising them for the effort they direct into a task. I’ve posted occasionally here and elsewhere on my criticism of the “self esteem” claims that run through popularised literature and professional practice in the education sector (at least when and where I worked in the field, which was admittedly a long time ago, in a nation far, far away…) – where a belief that adults need to instill a “high sense of self-esteem” in children functioned as a sort of unchallenged “urban legend” regulating educational practice in what I often considered to be counter-productive ways.

Out of context, my position will probably sound a bit odd: the short version is that my pragmatic experience of consulting on educational program design caused me, over time, to conclude that a principal challenge when developing effective educational strategies, particularly for children who are struggling, is actually to communicate that the need to expend considerable effort when learning is quite normal and to be expected, and should not be interpreted as a sign of failure: that some level of struggle – even quite a high level of struggle – in the process of learning is a feature, not a bug, and that one can come out the other side of this struggle with the mastery of a new skill. My perception of most strategies oriented to short-circuiting this process and trying to instill “high self esteem” directly, rather than trying to provide students with repeated, supportive opportunities to experience how they could successfully overcome frustrating situations, was that they encouraged precisely the opposite of their intended effect – convincing students either that challenging themselves was unnecessary, or that the experience of being challenged itself was a sign of failure. This summary article sounds as though there’s now a reasonable body of research that supports this position…

Since I don’t tend to write on my educational work here, though, I’m not actually posting the link to bask in how it confirms my predispositions… ;-P (And, in any event, I haven’t had the opportunity to backtrack to the studies themselves, to see how much confidence I place in their findings…) I was, however, intrigued by some of the methodologies reported in the summary article, and thought the piece might therefore be of interest to some students from a research design point of view. Even in the limited detail that comes through in the summary, there are some very clever ideas for organising a research process.

I have to admit, I’m a bit ethically leery about the first method discussed in detail: children were given a fairly easy task to complete, and at the end of the task were given a single line of praise – told either that they were “smart” or that they must have “worked hard”. The “smart” students, according to this study, then trended toward risk aversion and, after exposure to a deliberate failure via a task purpose-designed to be too difficult for them to do, their performance on the original easy task actually went down, while the “hard working” students trended toward more challenging tasks, and their performance improved. All well and good, and the conclusions are certainly interesting, but there’s something about the process of providing the kids with a specific set of conceptual tools for processing their success or failure on a task, and then deliberately setting them up to fail, so you can stand back and observe that, yes, in fact, they used exactly the conceptual tools you gave them… This technique gives me – I think the technical term would be – the heebie jeebies… But this reaction doesn’t remove interest from the thinking behind the research design…

In any event, this and other studies are recounted in more detail in the article and, I suspect, backtracking the studies could be quite useful as a source for creative research design concepts.

The Present Twilight

So I haven’t written much substantive lately – and this post unfortunately won’t break that trend. ;-P Prosaic work responsibilities are bearing down on me and, for at least the next several weeks, I simply won’t have time to dig in to serious questions. Which is frustrating, because I feel at the moment like I’m absolutely seething with ideas that are searching for expression and form. And writing – structured, sustained, in-depth writing, rather than the sorts of scattershot sketches I can dash off in between other things – is the only way I know to show myself what I’m thinking – to discover what force, if any, these still-inchoate ideas might possess… Read more of this post

Method Acting

PhD students in my school have a sort of progress review every six months, resulting in a written record that goes on the student’s file. These reviews vary from program to program (and from supervisor to supervisor), a diversity that interferes with the normal process whereby students gossip with one another and share tips about how best to prepare for such things… ;-P Last night, I received the following email from a colleague puzzling through the requirements for their upcoming review:

In preparing for my “candidature confirmation” – I notice that “methodologies” is distinguished from “methods”. I have not found any meaningful distinction between the two, other than methodology should be the *study* of methods. But I’m confused about methodology in the plural – I thought this was synonymous with method itself (like multimedia is synomymous with media). Am I missing something?

As someone who teaches “methodology”, I suppose I should know the answer to this question, but I confess I’m at a bit of a loss… My stab at the question was that “methodologies” might be the “to do list” component of method – what you’re planning to do, in the order in which you’re planning to do it – while “methodology” might be the logic that explains why your “to do list” can actually provide an adequate answer to the research question you’ve asked. But this is just a passing speculation on my part… Anyone have a more grounded answer (or a more entertaining speculation) for the question?


Someone sent me an email link to Richard Hamming’s (1986) “You and Your Research”, which I have read previously, but not for some time. The piece analyses why a few scientists manage to make significant contributions to their field, while the rest of us… not so much… ;-P Read more of this post

Methodology Slam

I’ve been watching with some amusement the evolution of Scott Eric Kaufman’s meme experiment, which I mentioned here the other day (anyone who hasn’t yet linked back will… I don’t know… have some horrible chain-lettery thing happen to you very soon…). For those following from the sidelines, Scott has posted an update on the quantitative success of his experiment. Personally, I’ve been following the unintended qualitative dimensions of the project – specifically, the number of people who’ve evidently decided to help out because, well, they’re so damned irritated about how badly they think the whole thing has been designed.

Sarapen opened with a thoughtful and even-tempered critique (including a literature review, even), but not all participants were as kind. Scrolling through the comments on Scott’s blog, you see a very large number of methodology criticisms, pointing Scott to things he hasn’t controlled for, noting problems in capturing the relevant data, criticising what he believes the data will show, complaining that he hasn’t sufficiently defined his hypothesis prior to the experiment, accusing him of stacking the experimental deck, arguing that the experiment can’t possibly be expected to follow the course of “wild” memes, contesting the finer points of whether “meme” was the right word for this – and, my personal favourite:

Based on my experience with technorati, when they pick up a link can be highly variable and not well-correlated with the actual time that link is created (to the point of being off by days). Your methodology is already crap just on technical grounds, even before taking into account all the objections above. Try using a web bug or something like it next time.

I suggested to Scott that he put all of this criticism to productive use at the MLA conference – prove the value of internet academic discussion, by challenging his panel audience to see whether they can come up with as many reasons that his methodology is “crap”. ;-P Since making this comment, though, I’ve begun to wonder: perhaps we’re looking at the birth of a new kind of PhD student performance art – the methodology slam. Someone stands up on some obscure corner of the net, calls out their research methodology, and asks a friend to tell a friend… Perhaps the results can be submitted as part of the portfolio to the committee approving candidature – a new criterion before you can call yourself ABD…


There seems be this unusual theory floating around the school of social science that I might be the best person to coordinate our quantitative research methods course – a “common course architecture” course aimed at second-year undergraduates from various programs. I’m finding this theory a bit hard to believe, personally, but others seem not to share my scepticism. Read more of this post

Dubious Text

So my talk for the “Dubious Ethnography” panel is out of the way – one down, one to go. I went through a particularly intense crisis of confidence about the whole thing yesterday, when the talk remained unwritten at 6 p.m., after an entire day filled with nothing but endless interruptions. It also didn’t seem promising that I have an intense sore throat and the beginnings of what feels like an ear infection – and, as I explained to the audience this morning, not being able to speak or hear seemed an unpromising beginning for a discussion…

In the end, though, I did enjoy giving the talk – and received some very good questions. Interestingly, the most positive and the most negative reactions related to my discussion of epistemology and critical judgment – which is somewhat amusing, as people generally just fall asleep when I discuss epistemology. Maybe I’m onto something with this narrative thing… ;-P

Some members of the audience really liked the notion of trying to understand the reasonableness of various positions in a local political conflict, while also trying to examine all of those positions critically for what they don’t quite grasp with reference to a more overarching and comprehensive vision of that context. One questioner in particular, though, was very unhappy with this proposal, really pressed me to declare a side – and then was unconvinced when I tried to explain that my main quarrel was not really with anything that was unfolding in the community where I research, but rather with certain frameworks with in the academic literature: that my main “side” was a critique of those academic positions.

I was challenged further to explain how this was an ethical position – don’t we ultimately all have to take sides with reference to what we are studying? Is it ethical to analyse the weaknesses in all competing positions without choosing a particular position we most strongly prefer? I suspect this is really, at base, not the universal and theoretical issue the questioner takes it to be, but more like an empirical and contingent question: depending on the conflict, it might be possible or impossible, ethical or unethical, to choose a side. My main purpose at the moment (not in this brief talk, which would be completely inadequate, but in the thesis) is to make plausible the notion that we can ground judgments in a recognition that some kinds of mistakes can be made by otherwise quite reasonable and moral people, who have seized upon a piece of their social context, confused that piece for the whole – and act as though everyone else has done the same… The context will then determine whether these judgments drive in favour of a form of political movement actually playing itself out on the ground in a particular dispute. I don’t think my answer was adequate – I’ll have to work on explaining what I mean.

Anyone who’d like a copy of the talk can email, with the caveat that, as always, the written version is not quite what I actually said – I tend to watch audiences, dwell on things that seem to get people nodding in agreement, and skip lightly over things that seem to get people nodding off… I’ll leave readers to guess which sections of the text fell into which categories…

Now I have to collect my thoughts for tomorrow’s talk – which, for local readers, will be delivered as part of the Environment & Planning Lunchtime Seminar series, in 8.7.6, at 12:30 (attendance is free; BYO food…).

Arrested Development

So I’m in the process of trying to organise the material I’ve collected thus far, with the goal of focussing my remaining empirical research to fill in gaps in arguments I actually plan to make in writings related to my research grant (as opposed to my standard mode of operation, which is randomly to pursue whatever interesting material happens across my path when I’m in the field…). I particularly need to make some targeted decisions about what to do with some interesting tangents that have come up during interviews and observations that were primarily designed to capture other things: which tangential material should I leave to one side? Which material should I try to make more robust through some more rigorous research targeted to the tangent? Which topics should be addressed on a theoretical level via secondary materials, with perhaps the occasional illustrative use of field material for… local colour? Read more of this post