The more I improve the quality of my course, the worse the student evaluation becomes. Should I provide low quality lectures?

Question

For almost 6 years, I have given courses on different CS topics at my university. This year, I decided to make my lectures and tutorials for a specific master course as perfect as possible, so I and my team (tutors) put an enormous effort (compared to previous years) to provide:

• clear and informative slides,
• explanation on a digital board + the slides (the only course providing recorded slides and board),
• diverse examples,
• the best possible video and audio quality,
• exciting and interesting weekly tasks,
• we even covered the fundamentals of linear algebra and math in general as a bonus because I noticed that my former students had a problem understanding some topics because of this.
• we also made the course hybrid: online, on-campus and recorded (the only course provided with this format at our university).

As I have taught the same course for 4 years, I would say that this year's version is at least twice as good than the first version. I not only base this on what I mentioned earlier but also,

1. I have mastered the topic very well now,
2. I prepared the weekly tasks from real-life examples (this makes them of course much more difficult but better to learn),
3. My courses have become very well organized and transparent (compared to previous years),
4. etc.

However, year after year, the student evaluation becomes worse based on strange reasons (e.g. the topic is difficult, we had to search by ourselves to understand*, etc).

EDIT: One of the common reasons given is that they could understand some topics better when they watched YouTube videos. For the university level (masters degree), I believe I have to explain on a higher level and not like 3Blue1Brown videos.

Consequently, the evaluation of this year is very bad compared to the first one from 4 years ago (which was relatively good).

I am wondering now whether I should go back to the old style, which I personally find bad and not very conducive to learning, but the students seem to like. Or should I continue improving my course regardless of the students' comments? Note that as a junior lecturer, the evaluation would help me to get promoted.

*By providing diverse examples and tasks, of course, they need to search by themselves and that was as intended. I think this is a legitimate learning technique as we cannot cover 100 percent of the topics in the course.

Answer 1

One observation, based on my own habit of completely redoing classes, is new syllabuses suffer severely from lack of tuning. In practice, a "worse" syllabus which I've taught a few times before is as good or better than a new "better" one.

I'd say to put a moratorium on "improving" the class. You've taught it this way once, you've seen what didn't quite work and so on. Use that to make tweaks to smooth it out.

One thing that jumps out -- real-world examples. All students say they want them, but as you wrote, they always involve way too much domain specific knowledge. The actual thing you're trying to teach gets lost in the mess. Cut them and go back to the old "teaching" examples. It's not a complete loss -- when students complain you'll be able to say "yeah, we tried that -- didn't work".

Then just work on general smoothing. If lots of students loved a certain 3rd-party video, go ahead and add a link to it. Think back over test Q's you were sure more people should have known, but didn't. If students were weak on a topic they needed later in the course, expand it a bit and cut one of your darlings. I love teaching recursion, and did it at the end of 1st semester intro to programming, but it's tough to give a good assignment on, students needed more work with arrays, so it was cut.

A longer-term project is to look at previous, or even concurrent, courses in the major. Some things in the old syllabus may have taken advantage of those topics (maybe they all wrote small bash scripts, but have never seen HTML), or meshed better (and even if you wrote the old syllabus, it may have meshed coincidentally).

One of the hardest things for me is to look over my shiny new baby syllabus (and the rest) and realize I had some good ideas, but also a lot of wishful thinking. And thinking it wouldn't need lots of tweaking was pure hubris. I feel like teaching ComSci at least has a metaphor for this -- writing a new program is fun. But you don't have a product until after the drudgery of debugging, adding nice interfaces, and so on.

Answer 2

Are you sure that you are measuring the quality of your course and not something else?

• The last two years are special (corona), maybe that affected evaluations
• 6 years is a time in which the curriculum in the schools can change
• 6 years is a time in which the perception and expectation of students change, maybe there are good lectures on youtube or theattention span has changed
• 6 years is a time in which the composition of your audience may change - maybe CS became more popular

So before drawing conclusions, talk to your colleagues to get a reference value for how critical students may perceive something. Check if the curriculum in school of your audience has changed (if homogeneous enough).

After you actually verified that you can actually read something from this, break it down.

Answer 3

Does your university have a center where you can go to have your syllabus evaluated? For example, at the universities I have taught at, there are Centers for Teaching and Learning (or some variation upon that title), where you can have your syllabus and materials evaluated by administrative staff focused on pedagogy. There might not be anything "wrong" with your course, so I suggest having an external reviewer look at your materials might help.

On a personal note, for an online course I teach, I've ended up adding a lot of different materials to accommodate a variety of learners and learning styles over the years. However, by adding more videos, lecture notes, and activities in addition to the standard lecture and slides, some students have felt overwhelmed by the sheer variety of course content. So while I thought I was making my course more accessible to some, it was becoming less accessible to others by bringing in too many types of content.

Answer 4

Your Premise Is Broken.

If someone explained to you that he had refactored a program and now it ran slower, you would not call that an improvement. Yet you make an analogous claim in your question.

To the extent that student evaluations measure quality (which is a BIG, BIG caveat!!), your course is lower quality* than it was before. You had something that worked, and you "fixed" it until you no longer had something that worked. Your starting point has to recognize that.

You denigrate, "the old style, which I personally find bad and does not help to learn but the students seem to like it." I think you should strongly consider that you may not have a good perspective on what helps your students learn. That can be true without any failing on your part - several of my favorite professors from undergrad were objectively terrible at gauging the level of understanding.

Fundamentally though, if quality means good student evals, your effort was counterproductive. Sorry.

---

*Alright, look: "quality" is totally undefined. You could define quality to mean "maximizing the number of examples that use pineapples", and then you could legitimately have a higher quality course, but your metrics would not really lead you anywhere useful. I'm assuming that you're going with something like student satisfaction as your measure of quality, because otherwise mentioning the student evaluations is a non-sequitur.

Answer 5

I think many answers here miss the most basic thing. The thing is to talk to students and ask them what their problems are when trying to perform on an evaluation test/ what they find difficult. The quality of the course as a course itself should by determined by what the student gets out of it, not of the material or Individual components.

You may have the best material in the world or the best presentation in the world, but if the students don't get it (over repeated testing), then that material has some fault in it preventing it from being useful.

You also have another thing to check, the actual question papers. See what the students are writing, what they can and don't answer , and, if you've talked about these things in class or not.

Also secondly, you don't have to just make it abstract because it's a higher level. People are people whether they are studying in UG or masters. If you can present the topic in a nice and easy to digest way, then even masters student would get more from your lecture than if you didn't.

Two key examples of this I found are Tristan Needham's book on Visual Differential Geometry, and Complex analysis. Sure, it doesn't really teach either of these topic in the conventional sense, but reading these books, the student can develop a real passion for these subject and that can propel them into studying and working through a dry rigorous text.

Answer 6

I believe I have to explain on a higher level and not like 3brown1blue videos.

Why?

Your job is to help students understand.

It's not beneath their dignity - or yours - for you to speak at a "low" enough level for them to easily grasp the concepts.

I'll bet that in earlier iterations of teaching this course, when your own understanding was less solid, your explanations were easier for the novice to follow.

As you increasingly mastered the topic, things that now seemed obvious to you (but aren't obvious to the novice) "fell out" of your explanations, and those tiny omissions made it increasingly difficult for each new crop of novices, i.e. the new sets of students, to follow.

Answer 7

I remember reading that among the studies that compared student evaluations of their teachers with student learning, only one study was really scientific, in that it 1) measured learning by an external, independent test (not by the grade), 2) was longitudinal, i.e. tested students *before * they took the course as well as after, and 3) also measured student performance in later courses, for which the given course was a prerequisite. There was a slight negative correlation between student performance measured this way, and student evaluation of the teacher. In other words, hard work is arduous, but it pays off. I believe there is far too much emphasis on positive student evaluations.

EDIT:

@CuriousFindings suggested I add references. I am not sure if any of these 3 is the article I was trying to remember, but they all seem highly relevant:

Student Evaluations of Teaching Encourages Poor Teaching and Contributes to Grade Inflation: A Theoretical and Empirical Analysis, W. Stroebe, 2020

Why Good Teaching Evaluations May Reward Bad Teaching", W. Stroebe, 2016

"Student evaluations of teaching (mostly) do not measure teaching effectiveness", A. Boring, K. Otoboni and P. B. Stark, 2016

Answer 8

You seems to have done an impressive update to your course, and this is what really matters: you should not adjust the course contents based only on negative evaluations of others.

Students will adjust their expectations to meet the expectations of the instructor, and changing the contents may lead to negative feedback since the students will “naturally” compare the course to the previous year. The key is to assess if your expectations are realistic.

Constructive students feedbacks is one way to assess if your expectations differ from those of the students, but it is not the only way. Presumably you can compare the contents of your course with what is offered elsewhere, and judge if your course is an outlier in terms of contents and workload. You can also discuss your evaluations and mode of delivery with colleagues to see if the criticism is specific to your course or a cultural feature of recent cohorts critical of all courses.

If indeed you judge that the negative comments (which presumably do not originate from a militant subset of disenchanted students) and other indications point to a problem with the contents, then make the adjustment.

Answer 9

If you'd like to better understand student reviews, you'll have to ask them. Ideally, in a conversation with a few of them, rather than a survey that doesn't let you ask clarifying questions.

As a Master student in CS myself, I'll go over your bullet points to give you my thoughts, but keep in mind that I have not experienced your course.

For the university level (master degree), I believe I have to explain on a higher level and not like 3brown1blue videos.

There are some very high-quality videos on youtube. They don't go in-depth as much as the university course should, but they are amazing for understanding the rough idea. In my experience, a lot of lecturers don't give an introduction and overview, they jump straight to the rigorous math and leave me wondering what exactly we are doing, why we are doing it, and whether this is something I should be able to follow or even come up with myself.
Personally, I prefer learning with just youtube and blog posts, plus the university course's exercises + solutions over a live lecture that I can not pause, vastly. All I need for that is clear keywords on the slides so I know what I need to google and how much I need to grasp of it.

clear and informative slides,

which you are covering well. However, other students who prefer a live lecture will gain less from the "informative slides" and would prefer them to be light on unneccessary details. "A good talk has very little text on the slides" vs "Good lecture slides make the actual lecture redundant".

Now one solution is of course to have a lecture and a different script for self-studying. Which is generally nice, except

explanation on a digital board + the slides (the only course providing recorded slides and board)

Make sure that it is clear which parts need to be looked at as necessary and sufficient for the exam. One course I had this semester provided additional video recordings as replies to some student questions, had lecture slides, video recordings where the lecturer stated more than was written on the slides, exercises that seemed to deviate from the lecture, graded projects that were fully off-topic in my opinion and distracted from actually learning anything, and reference material book chapters.

The book chapters were all very good! But reading the reference material from just a single day's lectures would already take me two weeks. And when I asked what was relevant for understanding, and what for the exam, the answer was that all material might contain something the other stuff does not was very unsatisfactory.

So please make sure you aren't overwhelming your students by giving them a lot of "helpful material" that they then have to work through in addition. (Or think that they have to).

we even covered the fundamentals of linear algebra and math in general as a bonus because I noticed that my former students had a problem understanding some topics because of this.

This is great for the struggling students, and boring for the ones that already got it. Perhaps some students are giving you worse reviews because of this. If you can start this kind of lecture with a slide that summarizes what will be covered to allow students to see if they need this lecture or not (and tell them this is the intent), that could help.

diverse examples,

"Do I really have to look at those? I already got the main concept but maybe the exam will ask about this specific example..."

One thing you have not mentioned yet is how clearly you state the goals. I find clear goals very useful, such as:

• What is relevant for the exam
  (Exams are often just barely related to the actually interesting topics. Because the interesting part is harder to actually test.)
• What is just additional info for those interested
• What is not even really meant to be understood? (e.g. some proofs on the slides)
• What are students supposed to learn in
• This course
• This lecture session / This exercise
  the best possible video and audio quality, exciting and interesting weekly tasks,

we also made the course hybrid: online, on-campus and recorded (the only course provided with this format at our university).

I love that! With all the thought you are putting in, you probably got this right. But for completeness: Hybrid can be a terrible experience compared to online-only. Things I've experienced that are negative examples:

• Lecturer used laptop microphone for zoom, and worn microphone in the room. So everyone who was there in person understood perfectly fine, but whenever the lecturer walked one step to the side, we wouldn't hear anything online.
• Chat was not monitored.
  I find one of the biggest advantages of online lectures how easy it is to interact with the lecturer. I can simply unmute myself and ask when something is unclear. Compare that to raising your hand for ten minutes, then yelling "sorry?!" and still not being noticed, in the lecture hall. But even with this pro gone, just having a chat that the prof notices works well. Having a TA monitor the chat instead is alright too, but then no interaction is possible anymore.
• Lecture took place in-person only, exercise session directly afterwards took place online-only.

I master the topic very well now,

That's great, but does this make the course harder as well? It sounds like you did a lot of additions that would be very useful for learning the topics of your course. But sadly, your course is probably less than a third of the total credits students are supposed to take in one semester. Where every lecture already is more effort than it is supposed to be. If I feel like I have to put in nights and weekends to fare somewhat okay in your course, that will negatively influence my review of your course.

Also, is your presentation still aware of the difficulties that people have with the topic when they first encounter it? Perhaps you've mastered your topic too well and assume things are obvious that you yourself also didn't grasp instantly.

we had to search by ourselves to understand

Keep in mind that this can mean multiple things. Maybe you did explain it in a way the did not understand, yes. Or maybe you explained things that were too obvious so it was confusing because it was not clear why you were even explaining in the first place. Or maybe it just took a while to really grasp it and reading it up online worked, but watching the lecture a second time a week later would also have worked... This is looping back to my initial advice: Perhaps you can ask some of the students how it's going.

For example, you could casually do some small-talk with students in the break when there are no more questions. One time a lecturer did this and realized thanks to that that we actually had none of the ~five prerequisite courses he assumed everyone must have taken before his lecture ... he didn't realize that CS students didn't even have the option to take those prerequisite lectures and still his course was one of the recommended three to us.
He also started to summarize in between subtopics "What did we do. Why did we do it. What do we want to do next. How will we do it." after our casual feedback, and that made it a lot easier to follow.

Answer 10

This is touched on by Simon's answer, but the scholarship of teaching and learning (SOTL) is discovering that people are remarkably bad at determining how well they are learning. The vast majority of students will say that they are "a visual learner", by which they mean they want to sit back and watch an entertaining lecture. SOTL, on the other hand, consistently shows that the best way to learn is to roll up your sleeves and struggle through the material. Needing decent course evaluations for your promotion is the big downside of refusing to cater to students wishes.

The best counter to bad evaluations that I can think of is if you can provide evidence to your promotion committee that students are learning better under your revised course than the previous version (providing evidence to your students that they're learning better can also help their attitude). If there is a subsequent course that a large portion of your students take, you could see how their grades from one course translate into grades in the next course. But for a graduate level course (my interpretation of "master"), finding a subsequent course may not be feasible.

If you don't have a subsequent course, you could use performance on the final exam (or specific questions) as a gauge for their learning. The difficulty here is comparing that performance across terms. Even if you repeat a question, if it matches too well with a question from their homework, that will inflate those scores on the final exam.

Answer 11

EDIT: One of their common reasons, is that they could understand some topics better when they watched youtube videos. For the university level (master degree), I believe I have to explain on a higher level and not like 3brown1blue videos.

Based on this student feedback, it might be the case that students are lacking some preliminary or basic knowledge of things you are assuming in the course. It is okay to use your notes to explain things "on a higher level", but you should ensure that students who lack preliminary knowledge have access to resources to learn the basics before they get to your "higher level". Students often lack competence in preliminary topics that are assumed knowledge in a course, even if they have passed a previous course that taught that material (often due to forgetting things, etc.). Often in my own teaching I have found that it is useful to include some preliminary notes/resources to assist students to learn or refresh knowledge of topics that are taken as assumed knowledge in the course.

One thing that would be reasonable to do here is to pick a bunch of online videos that explain preliminary or basic concepts well (perhaps ask the students which videos they found helpful) and give links to those resources to offer students a refresher on those topics. Don't be afraid to incorporate outside resources if these are good quality and freely available; this can be a useful supplement to your own course materials. In particular, you can then frame your own notes and resources in a way that quickly goes on to higher-level concepts. This can allow you the freedom to focus your own notes on important substantive material in the course, while also offering guidance on resources for gaining preliminary knowledge. Even if your students find the videos more helpful than your notes, if they find those videos through links you have provided, they are likely to give you credit for that in their evaluation of the course.

Answer 12

*By providing diverse examples and tasks, of course, they need to search by themselves and it was intended. I think this is a legitimate learning technique as we cannot cover 100 of the topic in the course.

1. Structure the content into a tree by section and then subsections to reduce search time of content.

2. Provide a digital quiz for each section, and subsection to document a distribution of student knowledge. [Any distribution other than a normal is bad to see for a section]

3. Address all sections with a large percentage of quiz mistakes a review in class at a special time.

Answer 13

All the students are not always right. That is what one can say about evaluations. But you should try to do the best you can for them. You are doing a job, but you are also training the next generation of productive citizens. If improving the course keeps you on track, and may help the students in the long run, I suggest continuing to do that. (It is definitely true that students need review on math. Mathematics teachers can tell you that, since we constantly review lower level topics in higher level classes.) Experimenting can also be useful, if you keep the correct framework in mind. Do not change just out of exhaustion. But try to find the time to plan for changing things up, and see how it goes, including how you feel about it.

Answer 14

I think that it is very important to acknowledge both limitations and importance of the student evaluations. There can be reasons for bad evaluation results that you either can't do anything about (such as general frustration about the impact of the Covid situation on the university experience), or where students have certain expectations that I'd proudly disappoint (such as all too detailed information about what will be asked in the exam and how certain potential questions should be answered).

However there is also very worthwhile information in the student feedback that may help you to understand why your singing and dancing latest update of the course doesn't work quite as well for the students as you think it should. It really pays off to read the feedback with an open mind and to develop an opinion about everything that is written (like, as I said before, "the students may have a point but I can't do anything about X" or "feedback Y reveals that the students have an expectation that I think is inappropriate, and next time I will say straight on that they shouldn't expect this", or "I wasn't aware that Z is a problem but according to the students it is, and I should do something about it").

Always have in mind that what the students write is their perception (sometimes by the way there are rather unique perceptions that are not shared by most other students). It is not "true" in any objective sense, and there may be reasons to create "dissonances", i.e., to do something for better learning that leads to some students being unhappy (as written in another answer, students are often not good at assessing how well they're learning, and they may sometimes learn more and better from a course that they like less). On the other hand, ultimately you do your work for the students, so their perceptions are important and shouldn't be discounted without good reasons.

One thing that I often do is a personal feedback survey, which the students can fill in anonymously, only asking as open questions "What do you like about the course?" and "What do you think can be improved?" after maybe 1/3 of the course (the official evaluation comes much later), just to see early on how they think it goes and to maybe change something, or to openly give reasons for why I'm doing something that some students don't like.

One issue that apparently hasn't been mentioned yet is that there are some psychological factors that are very important for making the students feel good, like whether they can perceive your passion for the subject, whether they feel encouraged to ask and contribute in class, whether they feel valued for their contributions etc., or rather negatively, whether you make them feel that you are frustrated by their lack of appreciation for your wonderful course. You can have the best course preparation in the world, still if you make students feel embarrassed and stupid when you see them making mistakes (in class or in exercises), and they're then scared to share their thoughts, this will not go down well (and rightly so).

Answer 15

Dumbing down your classes and assigning less difficult HW is one way to increase your evaluation scores. So, if you're after higher scores, go with the flow. Students don't see courses the same way, and putting in the time and effort is rarely worth it from the evaluations prospective. You'd have to change the system to achieve what you're striving to, which is better professionals.