About Those Course Evaluations...

Curmie was tempted to write a follow-up to his post about the reactions to the murder of Charlie Kirk, but realizes that there is little to be gained by doing so.  Conservatives will continue to pretend that only liberals perpetrate political violence (tell that to the cops attacked on January 6, to Paul Pelosi, to Josh Shapiro, to Gretchen Whitmer, to Melissa Hortman’s family…), that Charlie Kirk was a “moderate,” that Tyler Robinson was heavily influenced by the radical leftist ideologues in the Engineering Department at Utah State University during his single semester of (online?) attendance.  And on and on.

The vast majority of the readers of this blog, however, already know that paranoia, stupidity, and mendacity are in a death struggle to be the defining characteristic of the likes of Donald Trump, JD Vance, and Dinesh D’Souza.  They also know that reasonable conclusions require evidence, but also that evidence does not cease to be evidence just because, taken by itself, it’s inconclusive.  There’s nothing new there.

So, where to turn for another topic?  Well, Curmie mentioned course evaluations in passing in his piece about FIRE’s less-than-impressive trumpeting of their free speech rankings.  And then, this morning, a Friend of Curmie posted a link to an article in The Atlantic with the portentous title, “How Teacher Evaluations Broke the University.”  And who is Curmie to ignore such a sign from the universe?

Rose Horowitz’s article is more than a little predictable: linking course evaluations to grade inflation and lowered standards.  And, of course, there’s commentary on bias: “Course-evaluation scores are correlated with students’ expected grades. Studies have found that, among other things, students score male professors higher than female ones, rate attractive teachers more highly, and reward instructors who bring in cookies.”  People in Curmie’s (former) line of work have seen literally dozens of variations on the theme over the years.

What’s curious, however, is that there is seldom anything suggested that looks like a potential solution to the problem.  So: Curmie to the rescue.

Let’s start with something basic: course evaluations have two purposes which are sometimes in conflict.  One function is to help faculty determine what aspects of their courses are or are not working.  This is useful.  Curmie made some changes based on suggestions made by students on course evaluations: increasing the number of exams so that there would be less material covered by each one, for example.  Sometimes there’s a question about “why did we have to read X?”  Well, because X is really important, and it’s important that you actually see what it says instead of just having me talk about it… but maybe I’d better spend a little more time explaining to students why that’s true.

One of the biggest weaknesses in student writers is in their failure to link the evidence to the conclusion.  Often, adding a single sentence, or even half a sentence, would improve an essay considerably.  But professors are not immune from making similar errors or omissions, and a brief comment on a course evaluation can indeed improve the quality of instruction in future iterations of a course.  Of course, faculty are, and should be, free to ignore suggestions they consider unhelpful.

The other reason course evaluations exist is the one that’s potentially problematic: the evaluation of faculty, especially regarding retention/tenure/promotion decisions.  No, 19-year-olds, even taken as a group, should not be primarily responsible for whether Professor X has a future at the university.  But that doesn’t mean that students should have no input.  So, what to do?

Suggestion #1: take course evaluations offline.  Back in the day, course evaluations were completed in class.  Curmie often had a colleague teaching in the same time slot down the hall, so with 15 or 20 minutes left on the last day of class, we’d switch rooms: he’d pass out the survey in my class and I would in his.  Sometimes, if the schedule demanded, the department admin would do the honors.  Anyway, the result was that if there were 25 students in a class, we’d get at least 22 or 23 responses.

Shifting to online meant that filling out a course evaluation became a choice, and we’d get the formula Curmie mentioned in the piece linked above: “If there are 20 students in a class—5 loved it, 3 hated it, and a dozen thought it was OK, you’ll get two positive responses, three negative responses, and two ‘meh’ responses.”  Yes, you can incentivize filling out the online form, but Curmie could never figure out how to make that not seem like a bribe for completing the eval or a penalty for not doing what the student might reasonably consider a waste of time.

Suggestion #2: stop pretending to an objectivity that doesn’t exist.  Assigning a numerical score to a question that requires an obviously subjective response is inherently problematic.  Curmie would get rid of those number scores altogether: have students write a paragraph each about what they thought the strengths and weaknesses of the course were.  Prompt them with ideas about relevant topics: knowledge of material, availability outside class, keeping the interest of students, etc., but don’t ask about every item individually.  And insist on specificity.  Don’t insist on an answer to whether a faculty member keeps regular office hours if the student never tried to go to them.  If a student’s only complaint is that the course was too difficult, that’s significant.  If the complaint is about turn-around time for essays or exams, how long did it take?  And so on.

Suggestion #3: actively compare course evaluation scores to grades.  Yes, Curmie thinks that 1-5 scale should go the way of the rotary phone, but it’s unlikely to do so, so here’s what Curmie did when he was called upon to review a colleague’s RTP documents: take the average score on that overall instructor rating and subtract the average grade in the class.  So, for example, a popular professor might get a 4.5 on that 5-point scale, and the average grade in the class might be exactly a B.  4.5-3=1.5.  That would be an excellent score.  Over 1: good.  0-1: OK, maybe.  Less than 0: terrible.  Yes, this is a quick and dirty analysis, and it shouldn’t be used in isolation, but it does at least discourage buying good evaluations with undeservedly good grades.  Our client is the society.  We need people with skill-sets, not just degrees.

Suggestion #4: when considering a major decision—one involving promotion or tenure as opposed to simply retention—the students who really matter aren’t the ones who just finished the course: they’re the ones who took that course a year or more ago.  You took the introductory course from Professor X: how prepared were you for the advanced course?  You’ve now graduated: are you ready for a job/internship/grad school?  You’ve been out five years: tell us how Professor X prepared you (or didn’t) for your current career path.

Suggestion #5: administrators need to grow some cojones.  Years ago, student opinion factored into decision-making only on rare occasions.  Generally speaking, the only time it mattered was when the senior faculty, department chair, and dean were leaning towards a favorable result for a faculty member, but student opinion was overwhelmingly negative (the converse of that didn’t apply: overwhelming support from students never reversed a tenure denial).  But if administrators are now placing too much emphasis on course evaluations, that’s on them, not (or certainly not exclusively) on the assessment device.

Suggestion #6: look for other means of assessment of faculty.  Curmie always got at least good, sometimes excellent, course evaluations.  But what mattered to him, and what should matter to decision-makers, came not from student opinion, but from student success: his former students’ success in more advanced classes, including in grad school; their pass rate on the state content exam for prospective secondary school teachers; their subsequent lives as artists, teachers, administrators… but most of all, as citizens.

There are indeed some problems associated with course evaluations.  There are, as noted above, some things that would lessen the harm while keeping the advantages.  But the real problem is that lazy and feckless administrators don’t have a clue how to process the information they have available to them.  A lot of programs are rather like the Augean stables, and there’s no Hercules in sight.