Suggested idea: Course exams written by someone else other than the instructors/coordinators. Good or Bad?

Question

As you might know already evaluating learning outcomes is very difficult. For example, student feedback is not always reliable; there are studies showing inverse relations between student learning and student ratings (eg. a course that managed to challenge students).

One potential idea is to have the final exam (or even midterms) in the math course written by someone else other than the instructors. Here are some pros:

• The instructors' bias. By having the final exam questions designed by someone else, we substract the frequent issue of (consciously/unconsciously) training the students to pass the final exam. Ultimately the goal of the course is to help the students tackle outside-world challenges using the techniques from the course.
• Fitting grades into a bell curve bias. Another frequent issue is being pressured from the department to adjust the test difficulty so that the grades fit the bell curve. So this often leads to unrealistically difficult tests that are less about testing a student's understanding and more about lowering the average grade.
• Smaller percentage grade and instructor performance: The main goal here is to test student learning in an independent way and thus instructor performance. So we can have the final exam grade worth less. We can allow the instructor to spread the grade weight across the learning modules (more quizzes less big tests) to encourage active learning throughout the course.

Here are some cons:

• some Course flexibility lost. By "offshoring" the final exam, the course material will have to be suited within some strict guidelines. Meaning that instructors will not be able to test new ideas that they presented during the course.
• Too much emphasis on the final exam. This might lead to too much emphasis on the final exam, and less on active learning throughout the course. But that is an existing problem with the current paradigm and if the final exam relative percentage is not too high, that issue can be ameliorated.
• "Who will design the final exam?" The main issue here is to have the designer of the exam be independent of the course and that can be hard since the professor faculty is quite small. It might introduce a lot of unnecessary politics. Ideally, the process could be automated using technology (i.e. some algorithmic way of sampling exam questions from a database)

Any feedback, references or anecdotes are welcomed. Do you see any problems with this idea? Is the existing way (exams designed by the instructors) better than what suggested?

Answer 1

1. Your point about less weighting of the final exam is made as both a con and a pro (not logically consistent).

2. The issue of less flexibility to alter the curriculum is not purely a con, can be a pro. I know this will raise hackles of community of instructors here who enjoy flexibility and see themselves as Robin Williams in Dead Poets. But there can be a real benefit from the student point of view and from the customer (school, military, employer, government) point of view in making sure that specific things are covered. (You don't see the bar or PE exam as instructor-specific!) And teachers may vary in how they lag, accelerate, or even push their own research (that feature can be a bug!)

3. Another benefit can be motivational. Jaime Escalante has written about this. He was in it together with his kids to beat the tests (hopefully not by cheating). And the presence of an external, seriously designed exam was very motivational because it was a somewhat objective bar to clear.

4. It's not just the bell curve--you can do much more serious psychometrics when you have large test populations taking thoughtful independently designed tests (not ad hoc by individual profs for individual sections). Can do things like cross year comparisons. Can really try to estimate what it means to have "A", "B", etc. knowledge of the subject (for example what level/frequency of problem solving). I.e. trying to set a bar, not just bell curve the kids. See point 5 also, however.

5. I don't think this is convenient for most single instructor courses. (I.e. not for majors courses or random humanities electives). But it does make a lot of sense when you have a lot of different sections taking the same thing. Consider the AP exams, for instance. Or at military academies, there were many required courses (non majors) that had essentially the entire class taking them, but in separate sections of ~30 students (like high school). So if you have 35 sections (say 15-20 instructors), than that is no big deal to have one of them (or a separate person) design the exams (not just the final or midterm, but intermediate exams, every 4 weeks in a 16 week semester). Examples of this would be general chemistry (2 sems), physics (2 sems), calc 1-3, ODEs, EE (2 sems), world history (2 sems), naval history, CS, all the prodev courses (navigation, leadership, etc.). Also many of the engineering courses that all engineers take (statics, "engine math", etc.) before going into major specifics.

6. Note that it's pretty normal if the test is independent for the grading to be independent (A grades B's students and visa versa) or more commonly A and B grade problem 1. C and D grade problem 2. Etc. And of course because of the scale of the effort a thoughtful key can be constructed. (AP even has a system where if graders encounter a problem, for example an unexpected solution approach, they can appeal it to an expert.) Note however, for large classes, this sort of systematic grading can be done even if the test was set by the instructor. For example Professor A. B. Sentmind at Enormous State U., teaching freshman chemistry in a 900 seat lecture hall. (Will have an army of TAs working as an organized team to grade the tests.)

7. As far as Crowell's point (see comment below headpost) about instructor cheating, it's actually harder for them to do this when the exam is standardized and grading consolidated. (Not impossible, harder...than if they do everything in isolation.) In addition, the same objection about "instructors might cheat" if you measure them to assess performance applies to students as well. Yet we still try to assess them. Just the chance of some cheating should not stop attempts to measure things. (Should we stop having companies do SEC financial reports because some have cheated in the past? No. There is still a net benefit, even if not perfection, in having standardized reporting as per GAAP.)

Answer 2

I am a high school teacher with three preps this year. Two of them have a "final" that are written by an external party (a Geometry final from the New York State Board of Regents and an AP Calculus test written by College Board). I don't mind either of them. I fully admit that I "teach to the test", in the sense that I trust the test to be well-designed and fully cover the domain of standards that I agreed to meet when I took on the class. In return, the students who pass the class have an extra form of accreditation that says that they have a mastery of the domain that is worth something of value (either an advanced HS diploma or the potential to be placed out of college Calculus).

To address the pros and cons you mention in your post:

• Instructor bias: absolutely a problem. If I hate optimization problems in Calc, I might decide to not teach them. If I adore them, I might spend five weeks on it instead of two. Neither of those decisions would serve my students, who are presumably learning an authentic survey of introductory Calculus so that they can use it in future courses or other career development.
• Fitting grades into a bell curve bias. Not a problem that happens at my pay grade. I suppose I am relieved that there is an external source assessing my students so that I am not only relieved of the high-level design of the syllabus but also of the level of mastery required for "passing".

• Smaller percentage grade and instructor performance: I'm not sure I understand this point. I don't think the design of my unit tests would be impacted in any way if I became responsible for my final exams.

• Some course flexibility lost. This might betray my philosophy of education to some degree, but I am not certain that teachers have a broad enough view of their students' needs to deserve authority to decide what material they should teach in a course. Sure, if you're teaching a doctoral student, then go for broke. But if you're teaching, say, an undergraduate accounting class and spending time not ultimately preparing the students for the accreditation exams or their professional lives, then you're off the reservation. (That may have come across as harsher against our professionalism than I intended. We know how our field is changing, and deserve a voice in the direction of our course development. But if we cannot convince a guiding committee that we are correct, we're on shaky ground when it comes to overriding them.)

• "Who will design the final exam?" If it's a course that is part of a sequence, I might suggest that the exam should be written by next semester's instructor. This is definitely a stakeholder in measuring what the students are taking away from your class.