There seems be this unusual theory floating around the school of social science that I might be the best person to coordinate our quantitative research methods course – a “common course architecture” course aimed at second-year undergraduates from various programs. I’m finding this theory a bit hard to believe, personally, but others seem not to share my scepticism. As a matter of general practice, other people’s confidence seems to override my uncertainty (not in my mind, but in theirs… I find it remarkably difficult to convince people that I’m not qualified to do something – a difficulty that I sometimes consider a kind of orphaned gift: imagine the wonderful things a proper con artist, for example, could achieve with such a thing… Whereas all it does for me is make me feel guilty about benefitting from what, as far as I can tell, is some kind of academic halo effect…), so I suspect I will end up coordinating this course. Since I am honestly not particularly skilled in this area, and since I also find it unacceptable that students should suffer for my own shortcomings, I sense a vast amount of course preparation work in my near future…
It doesn’t help that the course has changed hands a number of times in recent years (people keep casually blurting out terms like “poisoned chalice” when I ask about the past versions of the course. They then suddenly remember that we’re having this discussion because they are asking me to take over – a situation that generally results in a wince and a very, very guilty look). And then there’s the issue of student reaction to the course. I gather this hasn’t generally been fantastic – although I also gather that many students change their opinions over time, particularly when realising the demand for these sorts of skills when they hit the job market.
And there are some fairly fundamental differences of opinion about the function the course should serve: is it an early version of something like the Research Strategies course – aiming to teach students how to connect research questions with appropriate methodolgies? Is it a statistics class? Is it a software training class, focussing on developing marketable skills with SPSS and other common tools? Why is it offered separately from the qualitative methods course? Do we believe the qual-quant distinction is meaningful and should be perpetuated for some specific reason we should articulate to students? etc.
In its most recent incarnation (in as much as I’ve been able to reconstruct this – it’s proven strangely difficult to obtain a full course guide, exercises, and such for the course – I’ll have to approach one of the current tutors), the course seems to have been heavily skewed toward software training, covered some reasonably basic statistical concepts, and apparently involved a major project in which all students undertook a statistical analysis of the same public policy issue – something related to drug use – a topic that was apparently chosen because someone expected it would resonate with the “youth”… I’ve blogged previously about why I’m not too worried about whether my classes resonate, so I’m not too concerned one way or the other with this particular project…
(As a side note, whenever someone starts worrying about how to “relate” their subject to student subculture – a worry that, in my experience, often leads fairly directly to the expression of some reasonably bizarre fantasies about what students are like – I always remember an incident at a previous job where I could overhear another staff member speaking with their students after class. This was a genuinely excellent teacher – possibly the most skilled I’ve ever known – but from an upper middle class background and intensely nervous about his ability to “relate” across racial and class lines to the inner-city African-American kids he was teaching. On this particular day, this nervousness translated into an absolutely atrocious, cringe-worthy effort to speak in dialect – from memory, the phrase “packin’ some heat” somehow made its way into the conversation – because, you know, surely these kids would have… er… packed some heat at some point in their young lives… ;-P I don’t think I’ve ever heard something so painful in my life – it was like Ricky Gervais meets Mr. Kotter…)
My worry about what I’m hearing about the current incarnation of the course is that, by dictating the project to which students would apply their statistical and software skills, you’ve effectively removed the ability to practice defining a research question, and learning about the relationship between methodologies and questions. You’ve probably also made it easier for students to avoid getting a good “feel” for mathematical reasoning, as my guess would be that a predefined project, likely broken down into predefined “chunks” to make it easier to guide students through it, has a higher risk of students just mechanically moving through the process – fewer opportunities to “get” deeper concepts by being forced to work through complex problems…
So my impulse is somehow to loosen up the course so that students can have some say in defining their own projects. The challenge then becomes how to do this while ensuring that there is some broad equivalence in the amount of statistical and mathematical sophistication required to complete the projects. Maybe I still need to develop a selection of projects? Maybe it would be sufficient to have a very detailed set of assessment criteria, which students would then have to demonstrate that their independent projects meet? Maybe develop a few “default” projects for students who would otherwise drown in a research design process, but leave the option of individually-designed projects open for the students who want it (which, in practice, will probably be only two or three people, though, if I leave this option open…)? I have to think about this…
I also have to think about the structure of the course. It currently involves a one-hour lecture, and then either a two-hour tutorial session or a two-hour laboratory session, depending on the week. What to do with the lectures? My temptation would be to use them to try to help students develop a “feel” for mathematics – to focus on mathematical claims in policy literature, journalism, activist writing – to get them thinking about what is plausible, to have a sense of what is ridiculous… This structure, though, would mean that all of their hands-on training and assessment-specific teaching would go on in the tutorial and lab sessions – a structure I don’t personally mind, but that may lead to highly divergent student experiences, depending on who leads their tutorial sessions (I’ll lead one or two, depending on timetabling issues, but most students in the course will probably do their hands-on work with someone else). I also need to worry about the tutorial staff – I like fluid and relatively unstructured interactions with students, but I am highly conscious that this is a matter of personal teaching style – I can’t design a course to require this from teaching staff.
And the nature of the material means that some concepts will have to be learned relatively linearly, and therefore covered systematically: maybe I can swing my lectures so that the systematic concepts are covered there in a preliminary way, and married with some examples of policy or popular mistakes that derive from misunderstanding the underlying concept – and then the tutorial sessions can provide a more conventional approach to those same concepts? The lectures are also probably the place to discuss the basics of research questions and methodologies – and the various epistemological and ontological issues that underlie the quant/qual divide… Hmm… How many weeks do I have???
The labs, I think, would need in the first instance to provide basic software orientation. Then, by preference, I would think the labs would focus on the actual production of students’ final research projects. If I open these projects up so that students design their own questions, this could become a bit tricky… I’ll have to think carefully about what students are meant to produce from the lab sessions…
Too much to think about – and a lot to learn, myself, while I’m doing it. It reminds me a bit of the first course I coordinated here. The course was in Advanced Australian Politics – which amused me, because earlier in the year they had rejected my application to teach… er… basic Australian Politics, on the grounds – quite reasonable, to me – that I wasn’t qualified… I taught the entire course in a state of utter terror – I was cramming, sometimes up until hours before I walked in to deliver my lecture, in a way that I hadn’t had to do since my own undergraduate exams. I had redesigned the course syllabus before a lot of this cramming took place, so some of the lectures were interesting exercises in trying to justify what I now knew I needed to cover, under a week whose title suggested it was going to talk about something else entirely…
Then there were all the organisational issues… I had to massively change the assessment load after the first class session – I had developed the assignments based on the typical writing load at my previous university, and I literally (I originally spelled this “litterly” – I clearly still believe this course was rubbish…) had students come up to me in tears after the first session. It was only then that I heard from other faculty that there was a “cap” on assessable writing load – which I had… er… somewhat excessively exceeded. So I had to send an email around modifying the course requirements. And, since I hadn’t had time to select appropriate readings before the course began, I had left the syllabus oddly cryptic – “TBA” might have been the most commonly used word… I would wander in and pass out readings I’d only discovered the day before… Give lectures describing events I’d only learned about the morning of… Poll students regularly (responses counted toward the course participation mark) about what they thought they needed to learn – and then go research that myself… It was one of the most terrifying things I’ve ever done in my life…
And then my evaluations came back. How I dreaded them. They were actually good, but what was funniest is what received the highest praise: the precise things that were weakest – my knowledge, and my organisation…
So who knows? Maybe I can charm a set of students into believing that I know something about quantitative methodology, as well…
Like this:
Like Loading...
Related
Rough Theory 
My understanding of this class was that the students found it painful, but relatively easy – in that the work was mostly mechanics that could be rote-learned and/or copied. But that was a few years ago, and I didn’t have to do it myself.
Your biggest problem is that most social science students are petrified of numbers. Rather than seeing the statistical concepts as just another language for expressing some relationship, it is a mess of meaningless figures and mostly meaningless operations. I don’t really see the value in teaching software in either of the research courses – you can run most of these tests in any basic spreadsheet – but my background might make me an outlier in that assessment.
Nor am I sure that teaching the mechanics is actually terribly useful. Social scientists who can run regression tests but don’t know how they work are a menace. Most of it will be forgotten quite quickly anyway. I’d rather see a course that worked through major concepts – averages and variance, correlation and causation, significance etc. – in the lectures and tutorials, focusing on the what and why, and using examples/theoretical readings, and only mentioning in passing that “oh, there is a test for this”. You seem to be leaning in that general direction anyway.
Actually, I am not sure you even can teach mechanics. It is the sort of thing a weekly/fortnightly exercise sheet and instructions for how to go away and do it work best. Perhaps allied to a drop-in lab time for students who just don’t get it?
A lot of the project complications will be data related. If you restrict it to a single (largish) dataset – say ABS statistics for Local Government Areas – you could get students to formulate a question based on available variables, do a small lit. review, choose and justify a statistical method, and then run their analysis. That sounds easier than it probably is.
Yes – I agree with your sense of what the priority should be. My dilemma is that I’ve inherited a course that seems to be centred on walking students through mechanics – as a response to the fear of numbers you’ve mentioned (the same fear that has led this course to roll downhill to me…). Teaching this kind of course as rote procedure helps contain students’ anxiety without, however, actually diminishing the causes for their fear.
My worry is that I’m redesigning blind – I’ve been involved in math curricula designs before, but it’s been a very long time, and I’ve lost the sense of what students don’t know, of where they fall down… So there’s a large risk of pitching a conceptually-based course too high, and provoking some kind of repeat of my advanced politics course, where I had students approaching me in tears about the course requirements… ;-P
I tend to agree with you on mechanics, but am not sure it can be excised from the course this year (I will also be involved in a longer-term course redesign process, which is aimed at rethinking both the quantitative and qualitative methods courses, and this process will provide some scope for rethinking issues like this). My off the cuff recommendation was that they distill out software and mechanics style training into a separate intensive course, as this will probably give a sufficient orientation for students to drill in on their own.
That said, it is a recurrent complaint about the more advanced Research Strategies course that we don’t provide more hands-on software and similar training – that we focus too exclusively on the logic of research design. I don’t personally agree with the complaint – I tend to think that research design is actually very difficult, and am always worried that the one-term Research Strategies course isn’t sufficient to communicate how to do this to a reasonable level of competency – but I have to acknowledge that a decent percentage of even our more advanced students disagree…
Back to the quant course: I’ve received some informal feedback from a few of my first years (who are no doubt looking forward nervously to the course…) that they don’t really understand why they have to do any “methods” training at all, if they have no intention of doing Honours. I realised in the course of these conversations that my “stock” answer from the US doesn’t really apply in Australia – in the US, the undergraduate degree is fairly generalist, and so you could explain to someone that they were being credentialled as generally competent in the social sciences – a credential that meant, among other things, that they could make competent evaluations of social science research. In practice, this is the motivating idea behind our common course architecture courses here – that all students coming out of vaguely social science disciplines need a certain base of common knowledge, whatever their specialisations. This motivating idea, though, seems still to be very alien to most students – whether because it has not been articulated, or because the vocational nature of Australian education weighs so heavily against it that the concept isn’t easily absorbed…
I think I keep finding it so shocking that people wouldn’t want to use their undergraduate experience to… er… learn something… ;-P I seem to fumble unconvincingly in these discussions… I actually do have a response – really, if you plan to work in any professional field, you need to be able to read and assess research. And this is actually much easier to do if you learn the underlying concepts that are generative principles for understanding everything else…
So let’s say I target the lectures toward major concepts – would it be better to leave out the “research design” issue (this course is advertised as a course on research methodology, even if it moonlights as a course on statistics), and focus on core statistical concepts (and perhaps a bit on developing a basic sensitivity to mathematical plausibility)?
I still have the inherited structure with the labs, so I’ll have to work out something to do with them. After writing the original post, I had an opportunity to read through the lab manuals – they’re clearly written and well-organised, but do things like toss out vocabulary very quickly – including some vocabulary, to be honest, that would actually be fairly unusual in the “wild”, and more easily expressed using everyday terms – but are very much oriented to “do this, then do that” and magic occurs at the end… My problem is that we don’t currently offer any other structure for software training other than this course – and until this has been resolved through a more fundamental redesign of the methods courses, the software training component apparently must remain.
And I have to figure out what the main assessment tasks would be – and how to produce adequate supporting materials to guide students through them (again, my major problem is that I don’t have a “feel”, sitting here now, for where students are going to fall down). Your notion of restricting to a dataset is good. Ideally, in a course like this, you want multiple assessments so that students can make mistakes and still recover – then the question becomes whether to offer multiple types of assessable tasks, or to have each task contribute a different portion to a final overarching project (which is, sort of, the approach in Research Strategies)…
Many thanks for responding to this…