Rough Theory

Seeking reasons to procrastinate in the face of the mound of marking that sits on my table, I’ve been spending a leisurely Saturday morning reading various studies of cognitive bias. (Note to self: this is probably not the best way to prepare for marking first-year undergraduate work…)

In the process, I stumbled across a very entertaining article by Justin Kruger and David Dunning, titled “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments” (1999, Journal of Personality and Social Psychology, Vol. 77, No. 6., p. 1121-1134). The article begins with the story of the hapless McArthur Wheeler, who in 1995 robbed two Pittsburg banks in broad daylight with no disguise. Arrested within an hour of the broadcast of the bank security footage, Wheeler expressed shock that he was identifiable on the security tape, protesting “But I wore the juice!” Apparently Wheeler believed that rubbing lemon juice on his face would render him invisible to security cameras… (Kruger and Dunning 1999: 1122).

While Wheeler most likely would meet anyone’s definition of “incompetent”, what Kruger and Dunning are primarily interested in are… er… relative incompetents – folks like you and me, who may be quite skilled in some areas, but are likely not so skilled in others. In those areas where we aren’t so competent, Kruger and Dunning ask, do we know that this is the case? Their hypothesis is that, below a certain level – they isolate out the bottom 25% in the specific skills (logical reasoning, grammar, humour) they test – we may be so poorly skilled that we actually don’t know enough to realise how far off the mark we are – we may, in fact, not know enough to recognise competent behaviour, so that we can begin to model it to improve our own performance.

The study is worth a read (any lurking methods students might particularly enjoy the discussion of how to test whether someone is in the bottom 25% in terms of their sense of humour…). I make no specific claim about the broader validity of the study’s findings, but some of the specific results have a certain intuive plausibility: The authors cite, for example, findings that study participants in the bottom 25% tend to overestimate their skill level considerably, and that they do not tend to revise this positive self-assessment, even after being provided with samples of higher-quality work to “grade”. Interestingly, the authors also mention that participants in the upper skill levels tend to systematically underestimate their compentence – until they are given an opportunity to view others’ work, which then allows them to revise their self-perceptions in a more accurate direction.

Both of these observations track reasonably well with my teaching experience. Struggling students often view suggestions for improvement as unfair and as impositions of impossible standards; they need assistance to get a very concrete sense that better work really is possible – and that it is realistic to expect them to produce such work. At the other end of the spectrum, extremely talented students are generally acutely aware of how much more is possible in ideal circumstances – and can come to measure their work against a standard of perfection that would make anyone depressed, causing their self-perception to become inappropriately low… This can actually be a bit more difficult to manage, since you wouldn’t want to lower someone’s sensitivity to how their work could be improved, but an unrealistically harsh judgment of one’s own work can also be counter-productive…

I’ve been experimenting recently with types of assessments that provide students with an opportunity to view and edit one another’s work, in part to address these sorts of issues (and also to decentre the teaching process a bit – particularly in advanced courses where it’s quite reasonable to expect at least some students to know more in many areas that I do). I’m not completely happy with these experiments to date, so I’m continuing to tweak, but I think there is value in providing students at various skill levels with an opportunity to see what kind of work is possible – and also what kind of work is common…