Is it ethical to assign problem sets and not tell students which subset of problems will be graded?

Question

It is common practice to have a homework strategy as follows: a professor might assign N homework problems but only have some random sub-set of those problems graded. This is motivated by a large class size, and a finite amount of grading time. The argument is that it motivates the students to do all the whole homework set, as they will not know which ones will be graded.

However, as someone who has been assigned to grade in this way, it feels immoral. I have had students who will do 9 out of 10 of the problems, but that one unanswered problem is 1 of the 3 I am grading, so they get a low grade.

I am curious to hear what folks here have to say about such a practice.

Answer 1

While I see no ethical issue here, you could suggest the following alternate scheme.

Once the homework is due, you take a quiz where students are allowed to use their homework notes. In the quiz you ask the students to provide solutions to the (randomly) chosen 3 problems in a limited time.

As a result, the graders only need to grade the 3 chosen problems.

If the students have done the problems before, they can write down the solutions from their own notes. If they can solve the randomly chosen 3 problems without having done any of them before, in the limited time available, then they probably deserve to get a good grade!

Answer 2

While it may average out to be alright, it is very important to look at the other side and consider how a student perceives this grading practice.

Students don't tend to look at the average but on the outliers which are caused by this system of grading things. The attention is shifted to those that lucked out when giving a partially solved assignment and those that hand in a very well solved assignment and barely/poorly pass. Particular attention is shifted towards those that experience one extreme or the other multiple times.

From a student’s perspective, luck becomes a factor in something simple as having their assigned work to be looked at and this is simply frustrating regardless of whether or not it evens out or can be considered fair across a semesters worth of assignments turned in.

Answer 3

I see no ethical issue, particularly if the students are aware of this grading scheme in advance. The defect you observed averages out: the unlucky students this week might take no penalty for partially-completed homework during another week.

Still, if you are allowed to modify the grading scheme, you could consider mitigating the effect (or perceived effect) of luck by giving “completion points” for plausible-looking answers to the other questions.

Answer 4

It is not only ethical, it is an excellent way of assigning homeworks especially if homeworks are necessarily long and the number of students is large.Say, if a homework assignment has 20 problems (normal number for a calculus course) and the class size is 30 students, then you would have to grade 600 solutions.

Another thing is that the homeworks for a class of calculus level consist mostly of practice problems, many similar problems designed to practice certain skill. Usually it is clear whether a student has acquired this skill if you grade one or two of these problems.

Answer 5

With the scheme of picking 3 problems to grade, if have an equal probability to grade any problem, then a student who regularly does 9/10 problems should expect you to grade the one they didn't do about 30% of the time. That is something a student should expect to happen fairly often, if they turn in, say, 10 homeworks over the course of a semester. Even for one homework, that is a fairly large risk, that they are taking by not doing the question.

There may be correlations that make this more likely, for example if students tend not to do the more difficult questions and there's a bias toward grading the more difficult ones.

Given that the policy is announced in advance and grading the one unfinished problem is not expected to be a particularly rare outcome, the policy seems fair to me.

Based on some discussion in the comments, let me add I don't think this policy is ideal or good. It clearly has problems in that there is some noise in the evaluation of the student. But, I can easily imagine situations where some instructors would choose this grading style as the least bad option. In particular, I think this scheme can make sense in a scenario where you have a lot of students, limited resources to grade, and a lot of (mostly straightforward) homework problems and assignments. Then, the noise should be fairly small (at least comparable to other sources of noise inherent in grading such as the variation in harshness/generosity of different graders and the choice of a finite number of problems to put on the homework), and it could be used by the instructor to encourage the students to do all the problems necessary to cover the full range of material, while not overspending resources on grading.

Answer 6

The grading scheme may be fair under certain conditions.

• You need many homework assignments. If it is only biweekly or monthly, then averaging out is unlikely to occur for almost all students.
• The students should always finish approximately the same number of problems on each assignment. Imagine that you have 10 assignments with 10 problems each, for a total of 100 assignments. You grade one random problem of each assignment. Now, one student one student solved 9/10 problems on assignment 1, and all problems on all further assignments. The student thus solved 99 out of 100 problems. Unfortunately the missing one was the problem chosen to be graded on assignment 1, so the student has 9 correct assignments and one missed one, for a total of 9 out of 10. Sounds unfair to me.

Furthermore, you might run into trouble with perceived unfairness.

The larger the number of students, the more likely it is that one will hit a lucky/unlucky streak. This will quickly spread amongst the students and they will perceive the grading as unfair, even though it is fair for the vast majority.

And finally, you might be accused of not really choosing the questions randomly, but selecting only problems which your favorite student has solved. This could be prevented by publishing the selection to be graded beforehand in an encrypted manner and then handing out the encryption key/password after the grading has occurred.

Checking whether this grading system is fair/unfair under different conditions might be a fun exercise to some introductory statistics class.

Personally I would recommend to tell the students beforehand which exercises will be graded and which won't. After all they chose to be there to learn I assume. For me it worked pretty well with pretty much no graded exercises at all. Most of my friends and me did almost all exercises anyways. And the ones who didn't: Well, most of them stayed at university for longer, only to not graduate in the end.

Edit Forgot to mention that we got feedback on all exercises, graded or not. Sometimes individually, sometimes in the form of example solutions. I think that's important so that all solved exercises help the learning process, not just the graded ones where you get feedback in the form of the grade.

Answer 7

It is statistically absolutely unethical in the parameters you described, if you pass/fail students based on average assignment grade. I assume a hypothetical best-case example where you:

1. Teach a motivated class filled with students that do not skip any questions. (The problem only becomes worse if they skip some questions.)

2. You teach a course with weekly assignments, maximizing the chance of "averaging out". I will assume a 12-week course, so 12 assignments.

3. Grade objectively and fairly, where each answer is either correct or wrong to give a % score on each assignment.

I assume that an average homework grade of 60% is needed to pass.

Let's model an adequate student as someone who answers ~70% of the total number of questions correctly, in three scenarios:

1. 12 assignments where 3 out of 10 answered questions get graded (as you mentioned in your post).
2. 12 assignments where 10 out of 10 answered questions get graded (extra workload).
3. 4 assignments where 9 out of 9 answered questions get graded (same workload as #1).

The only randomness I allowed in the model is that the correct/wrong answers are shuffled between tests (or within a test for grading subsampling). That is, there is no randomness in the total number of correct answers given by a student in this model. In scenarios 1 and 3 the total number of correct answers given is 25/36, in scenario 2 it is 84/120.

In both scenario 2 and 3 across 100,000 course simulations the student who answered ~70% of the questions correctly has a 100% chance to pass (that is because the average of averages still has a linear relationship with the total number of correct answers, if not subsampling). In scenario 1 this drops to ~94.4%. If all your students are perfectly adequate, you would unfairly fail 2 students out of a class of 37.

What about a more borderline student, one which answers ~61% (22/36, 73/120) of all questions correctly? What is their probability of passing? Without subsampling it is once again 100%. With the subsampling in scenario one it drops to 56.5%!

My code for the above simulation.

---

Subsampling is fair if and only if you subsample randomly (without bias) and you compute the average grade only over the subsampled questions. It is unethical in my opinion if you compute the average grade over assignments where it is random which questions are graded, as it unnecessarily introduces statistically significant variance in who pass/fail simply based on whether they got graded on the questions they got right, and which they got wrong.

To spell this out crystal clearly: when you use the average assignment grade rather than average correctly answered questions in combination with subsampling, you are throwing away information you already have and unnecessarily replace it with randomness. It's no more ethical than ignoring the existing pass/fail policy and instead flipping a coin to pass/fail students with a borderline grade, as that is essentially what you're doing.

Answer 8

An ethical issue would arise if such randomness were the primary cause of poor grades for a student who had no knowledge of the rules and no way to make up a poor individual grade. It is the overall course grade that matters, not the grade on a single assignment.

But, since this is homework, not the final exam, the student can probably absorb a single hit and still come roaring back. Grades are usually based on a collection of individual marks, even when only a final is graded.

But, a student who is otherwise qualified shouldn't be given a poor overall grade due to any random factors. A fair assessment, overall, needs to be made.

Answer 9

I once went to a pedagogy talk and the faculty presenter (in a STEM discipline) said that, among things, which can improve student results are: assigning homework, collecting homework, grading homework.

You have to think about "what is the purpose of this specific homework?" and "what is the purpose of the grade?" Is it to ensure understanding, correct misunderstanding, deepen understanding, assess understanding, and then what is the level of the work (think Bloom's taxonomy). There are many kinds of homework, and different strategies are okay for different types. For example, in teaching writing a "minimal grading" model for grading, e.g. essays, is often suggested for both pedagogical and time management reasons. It's good for students to write, and it actually is not necessarily helpful for them to get back a document filled with red ink. But if you were giving a multiple choice grammar test obviously it is minimal grading by definition.

One purpose of grading is "riding herd" which is to say to simple establish that the student has done the reading or made a valid attempt at the assignment. Then it's fine to just quickly give check/no check grades.

In your case, it sounds like the grading involves giving substantive feedback in order to help students do better in the future and highlight errors in their thinking. It sounds like you would be giving "partial credit." Thus all of the comments about "getting it right" are not exactly on point because it is not an all or nothing grade.

In that case, it makes sense to randomly pick a few problems to do that for. That work is time consuming and students are unlikely to read all of the feedback on all problems.

Of course if the homework is a high percentage of the grades and are the main summative assessments of student understanding that's a different story.

So, if the main benefit of homework is improved understanding then having students do homework is good. If they need a reward for doing homework, announcing that three problems will be randomly selected for grading on each assignment lets them know they need to at least try all the problems. You could also reward simply attempting all of the problems on a check/no check basis.

Answer 10

Another way to do this, which partly mitigates the 'luck' problem, is as follows: a student can hand in A of the N exercises. Three of them will be graded randomly. If the score on them is x%, the total score on the assignment will be x%*A/N.

This way, a student that solved 7 of the 10 exercises cannot get a low score by having bad luck: if the 7 exercises were indeed correct, the score will simply be 70%. Conversely, a student that solved 3 of the 10 exercises can not get more than 30%. Luck only comes in is if a student tries to game the system by intentionally handing in wrong answers, or if a student thinks they solved an exercise, but they did not.

The disadvantage of this system, is that, while you only have to grade 3 exercises per student, it will not be the same 3 each time. So there is some extra work in devising grading schemes for all exercises etc.

Answer 11

The students are told in advance, so they know what gamble they are taking leaving one or more unanswered. (If they don't that is an academic skill in itself that needs some work!)

So in principle, this is a way to ensure that all students take the material covered by the set equally seriously.

So the practice as such is ethical. There is a problem, however, in that some academics wish for their subject to occupy more of the students' attention that is proportionally reasonable. (The "my subject is the only one worth doing" illness is too prevalent among academics.) They can achieve this via this trick. If I have three contact hours per week (normal for a typical module) and the students have 15 contact hours (excluding TA time, workshops, tutorials etc) then my homework should roughly occupy 1/5 of their homework time. If I inflate this doing the subset trick, then that is unethical.

Also, the prof should decide (perhaps using randomising methods) which questions are going to be marked before seeing the worked answers, and not taking any bribes from students.

Answer 12

Consider that essentially the same thing happens each time a student writes an exam in that only a subset of all the possible exam questions are actually graded, while the remainder don't appear on the exam. The difference is simply the amount of information that everyone has: in the exam scenario the ungraded questions are unknown, while in the homework scenario the ungraded questions are known.

If time constraints are such that only three questions can be graded, then the other option would be to just ask three questions. In this case one could worry about the student who knew 70% of the material, but got 0% because the question selection didn't reflect their knowledge. Such a student would have (likely) done better in the randomized version. Finite time requires that we approximate student knowledge via the random selection of questions from an infinite pool. As such, we must accept that the randomization will be beneficial for some and a detriment to others (and hope that this evens out over the course of a semester).

To me, asking 10 and grading three is ethically equivalent to simply asking three.

Answer 13

Ethical? Not fully sure about it, I do not see a problem with this. Fair or sensible? For sure not.

The function of grading is grading. The function of homework is training.

Bias: This scheme results in better grades for students who do all the homework, even if they would perform worse when presented with identical tasks as better students who - for some reason - don't have the time to do all the homework.

Adding spread: If the grade should be an assessment for how good the student is able to handle problems, then this is going to increase the statistical spread of the assessment.

If you have to use uncertainty about what is being graded as a whip to make students work, something is wrong with your lecture.

Answer 14

Let's just count. We have 10 problems out of which 3 random ones are graded, so the probability of any particular homework to be graded is 0.3 and the direct computation shows that the largest possible variance on a set of 10 problems for the grading score of an individual student is 7/12, attained if the student solved exactly 5 problems, which (50% performance) is the usual cutoff for the F grade. The typical semester course lasts 16 weeks with 2 homework sets per week, which gives 32 homeworks. Assuming the normal approximation, we conclude that one standard deviation is at most $\sqrt{\frac{7}{12}32}=\sqrt{56/3}<4.5$ problems even for the borderline students (everybody else is better off). Now, with the expected score of 48 on the borderline, the probability that a (barely) passing student will get <44 problems is 16%, which may be a bit too much. If you lower the threshold to 39 problems, being out of luck has probability 2.3% for each student, which seems acceptable for moderate size classes. If you have 100+ students, I would go for 3\sigma, i.e., the passing score of 35 problems with probability of being unlucky 0.1% for an individual student. In other words, if you add 14 points to everybody's score in this scenario, the bad luck (getting a lower grade than one deserves due to the random chance) will be essentially wiped out even in a large class. You, of course, may now raise some scores beyond what is deserved, creating another type of "moral problem".

In other words the message is that the idea of such grading is neither ethical, nor unethical by itself. It just has a certain chance of error that you should clearly understand and, if you find it unacceptably high, compensate for before implementing this scheme.

Answer 15

If the assignment is to do 10 problems, out of which 3 will be graded at random, then what might perhaps be seen as unfair is that the student might have done a poor job on some of those 3 problems but an excellent job on the other 7. I can see why it would be frustrating to a student to spend a significant amount of time on one problem only to be graded on the basis of another. In particular, if the problem set consists of problems of widely varying difficulty, then this sounds like a sure recipe for inspiring legitimate frustration among students.

On the other hand, if the assignment is to do 10 problems and the student decides to only do 9 and hope for the best, that's entirely on them. They haven't done the assignment and instead intentionally opted to enter a universe where they had a probability of 2/3 of getting away with doing slightly less work than required for the same credit and a probability of 1/3 of being significantly penalized for this. (Significantly in the context of a single problem set. It probably isn't significant in the context of the entire course.) This is a trade-off that they chose of their own volition. Focusing on what happens in 1/3 of all universes where, some could say, they get a worse outcome than they deserve misses the point that there are also 2/3 of all universes where they get a better outcome than they deserve and that the decision to enter into this gamble was their own.

Of course, if this grading scheme is used as an excuse to burden students with assignments which place unreasonable demands on their time, then this would indeed be unethical. But the same is true if all 10 problems are graded.

Answer 16

It is ethical so long as problem sets are not a substantial part of the final course grade. The pedagogical value can be increased, and concerns about unfairness mitigated, through some minor adjustments.

I use a version of this in some of my courses. The major concern that you raise is one of unfairness due to the randomness of which problems are evaluated.

The ethics of the assignment should be evaluated with an understanding of the difference between formative and summative assessment. Formative assessments are supposed to provide feedback to the student and the instructor about the student's (and the class's) progress in the material. Summative assessments are supposed to provide an evaluation of student mastery at the end of a course module or entire course.

Summative assessments are high stakes (i.e., have a substantial impact on a course grade) and can include things like term papers and examinations. Randomness itself in a graded assignment is not a concern, but would become a concern for a high stakes assessment. An example where I think it is handled well is that, as an undergrad, I often encountered essay exams (usually in humanities or social science courses) where we were given 5 questions to prepare for, but only 3 appeared on a final exam (and sometimes the student would only have to answer a subset, like 2 out of 3). This offers a nice balance between having an overly long exam vs. a short exam where the students study to the test, i.e. ignore material that will not be on the exam. An extreme in which it is handled poorly would be assigning three papers and grading only one of them, where the final paper represented 30-50% of a final course grade. This would be indefensibly arbitrary, and I've never seen anything like it done.

Formative assessments should be low stakes, and some instructors seem to think should be no stakes (i.e., not graded at all). The question is how do we treat problem sets, which are common in STEM courses where practice of methods is necessary to learning.

Problem sets sit in an awkward place. They are often graded, and sometimes form a substantial part of a course grade, yet their place in a course sequence really means they should be treated as a formative assessment:

Reading/Lecture → Problem Sets → Examinations

In some sense, if the student does well on the summative assessments, the formative assessments shouldn't drag their grade down. That is, if a strong student is lackadaisical about homework but aces the exams, giving the student a poor grade due to missing or sloppy homeworks seems both punitive and just plain inaccurate as an assessment of their capabilities. So if their purpose is formative rather than summative, problem sets could be ungraded, and used entirely to provide practice for the student and feedback on how well they are learning the material. Unfortunately, I have found that if the problem sets are ungraded many students won't do them, and subsequently will perform poorly on exams.

Sidebar: Doing the problem sets improves student performance, but inevitably there seems to be a high correlation between the students who don't do the problem sets and those who I suspect would struggle with the material no matter what. I think the causality is that students who are underprepared get discouraged by the problem sets, and then don't do the work they need to in order to master the material! Trying to encourage those students while still making it clear that they have to actually do the work is a constant struggle, and probably not unique to me.

The students have to do the problem sets in order to learn the material, but the grades are really utterly irrelevant to the assignment's pedagogical purpose. But getting the students to engage with the material is very difficult if they are no stakes assignments. Offering low stakes (maybe all the problem sets are no more than 10% of the final grade in total) is probably enough motivation to get them to do the work. Since the purpose is practice rather than summative assessment, assigning more work than will be graded is acceptable. It becomes less acceptable if the impact on the grade is higher. It also becomes less acceptable if nothing is done with the ungraded problems. That is, you shouldn't just grade some of them and completely ignore the ungraded ones. Ungraded problems can be gone over in class, in discussion section, or answers could just be provided for the students to self-check.

Finally, there are a number of ways that the assignment could be changed to increase the pedagogical value.

1. All problem sets can be graded for extra credit. I do this in some of my courses. I have found that it provides sufficient motivation for most of the students to attempt the problem sets, while mitigating student concerns regarding grading of random answers. (If they don't earn the points, it's "just" extra credit, and has no impact if they do well on other assignments.)
2. For your example, points can be awarded for 3 out of 4 possible answers. That way, if a student skips (or just messes up) one question out of the 4, that one will be ignored. Points will be awarded only for best 3.
3. Switch to the flipped classroom model. This is of course a major pedagogical shift. In this case, the problem sets would be done in class. They could be completely ungraded, as the professor and teaching assistants will be observing the student learning directly (both that they are doing the work, and how well they are doing). Or they could be graded, but students will have assistance with them. Or they could be marked "complete" if the student is present and works diligently, rather than for correctness, but the professor could require them to be submitted for a grade by students who are absent or clearly not doing the work while in class.

Any of these things would improve the pedagogical value of the problem sets. But the randomness in and of itself is not an ethical concern so long as the problem sets are not a substantial part of the final course grade.

Answer 17

In lieu of writing more stuff in comments I decided to write up an answer that adresses both the original question and some of the discussion that has emerged following it. I'm writing from a perspective of math and in particular the lower level math courses 300 level ones and below in general (i.e. Calc 1-3 and below in the US). This is definitely not applicable to the European system where this wouldn't be an issue probably.

I will first describe the system I'm talking about in more detail. When I taught we would usually assign roughly 20 problems a week for ~10 weeks of the semester and grade 5 problems of each set. The total grade for a homework set was made up of 5 points for the five problems 2 for completeness and 2 for tidiness. Completeness is self explanatory, and tidiness involved not handing in the homework chewed up and illegible and using correct mathematical terminology (equals signs where there supposed to be equal signs, implication where there were supposed to be implications and similar). The total grade for homework was usually about 20% of the course grade. This meant any given problem was worth about 20/90 % of the total grade. (Much less actually because sadly almost all courses I graded were graded on a curve, don't get me started on that).

I want to answer two things. First, is this immoral (assign N choose K to grade)? Second, is this the best?

Let's start with number one: No, it's not immoral.

The reason we assign and grade homework is mostly because students even at the university level (in the US) can't be trusted to do what they should on their own. If the homework is not graded a significant portion of the students will not do it.So they will not learn and they won't figure out where they have shortcomings. Now the morally best choice would be to treat students as adults assume they do the work and when they find out they are struggling come to office hours or otherwise deal with it. Unfortunately that doesn't seem to work. Thus, trying to provide a service, the system attempts to incentivize doing homework by grading it.

Grading all of the homework is logistically and functionally impossible. It would either mean not assigning enough for it to be worth it, or paying way too much money for grading all of it. Math TA's are usually costly and in short supply since they often have better ways to earn money and the STEM fields are less full. Assigning more homework and announcing which problems will be graded in advance doesn't help since it goes back to many (most) students only doing the graded parts.

I admit you only ensure there's no luck involved if the student does all of the homework correctly. To me that's enough, you have a path to a perfect score that's not based on luck. Also assuming you do all the problems and most of them correctly the chances of getting a significantly bad grade are rather minimal.

In the end you have an optimization problem, to which the answer of randomized grading is a heuristically optimal solution. Thus it's not immoral.

Is it the best? No. I don't believe it is the best.

I do believe given the constraints of the US systems it is probably optimal though. We've already covered why it's not optimal to leave students to their own devices and why not all reasonable homework can be graded. There is another way to deal with the problem of providing feedback to students during the semester though and that's oral exams combined with quizess. Oral examination is (in my opinion) by FAR the most efficient and effective way of both providing feedback and finding out what the student knows. In 10-20 minutes of speaking to a student I get more of an idea of what he knows and doesn't, than I can gain from three exams and 20 homeworks. Unfortunately oral exams are strongly disfavored in the US. I'm guessing it's mostly due to size of class and the dislike of telling students they don't know stuff.

The trouble with oral exams is they are more prone to claims of unfairness or randomness and take more time away from the professor, since exams in the US are usually graded by the TA's, whereas oral exams would likely have to be done by the professor only.

Answer 18

One professor might work this way, and another professor might work another way. Each gives ten homework assignments and grades three. The first one doesn't tell which ones are graded, the second does.

The result is that students either put more effort into your course, which makes it entirely unfair to the other professor. And "putting more effort into your course" isn't a positive. Students will learn a lot from homework they do; they don't learn from the extra work they need to do for optimising the grading. In homework, it is reasonable to focus on the things that the student doesn't know; something that the student knows how to do doesn't help them learning and is therefore wasted effort. Any effort put into your course is effort not put into another course.

On the other hand, the students might vote with their feet and take two other courses instead of yours.