How can I motivate the teaching assistants to grade more strictly?

Question

I am a lecturer in charge of a course with over 250 students and several TAs. The TAs are partly responsible for grading the homework assignments. Since these are assignments in programming, no two submissions are identical, so it is impossible to cover all possible cases in the grading guidelines, and the grading has some subjective element.

I noticed that one TA is consistently stricter than the others. For example, if the guidelines say that "code efficiency" is worth 20 points, then this TA would deduct 15 points when the code is inefficient, while the other TAs would deduct only 5 points for a similar issue. A potential problem here is that it might be unfair to the students in the strict TA's class, but this can be solved by allocating the assignment task "horizontally" (each TA grades all 250 submissions in some of the assignments) rather than "vertically".

But I have a different question: I noticed that students who are graded more harshly, take the feedback comments more seriously, and tend to become better programmers. So, rather than just being "fair", I would like all TAs to grade in a stricter way - for the sake of the students. The problem is, most TAs are not motivated to grade strictly - they gain nothing from it; all they get is having to handle students' complaints and appeals, and risking lower marks in the students' feedback (since the students do not understand that it is in their favor until after they graduate).

The TAs are not lazy - they do put a lot of effort in teaching and helping students; they just don't like to be the "bad guys" who give low grades. How can I motivate them to give stricter grades?

CONCLUSION: Thanks a lot to all repliers. In addition to the excellent answers, two things that I did were:

• I assigned myself to one of the TA sections (where the assignments are graded), in order to get a view of the grading task from the perspective of a TA. It was a very interesting and important experience, and helped me refine the rubric.
• I introduced the use of a static analysis tool (specifically: clang-tidy, for C++) as part of the automatic grading. It was way more strict than both me and the TAs in detecting readability and code-quality issues. Students learned a lot just from trying to make clang-tidy run on their code without warnings.

Answer 1

A few things to add:

• Be sure the TA's know you'll back them and will be the bad guy. The students know the TA's are using your guidelines. If a TA uses their judgement but was too harsh, you'll let both them and the student know it was your fault for not being more clear.

• Emphasize the benefits of consistency to the TA's. Remind them students compare scores. Let them know it's OK to ask another TA how they grade something (or you). Let them know if they give too many points, it's causing problems for the other TA's.

• Remind them they had to work hard to pass this class (assuming they did). Students tend to be protective of their majors, especially TA's, and want to maintain standards.

• Get TA buy-in for the grading criteria. This is similar to the last bullet. Suppose it's -50% for not using functions, even if it works. Remind them this is the "learn to use functions" assignment and it said they were required, and you went over functions in class all last week. 50% off is generous.

I managed to go 5 years never having heard the word Rubric, then it was 6 months more before I realized it's the exact same thing as a grading key. I try to be somewhat detailed over a range:

Style:

• -5: didn't try. nonsense var names, random indents, looks like garbage
• -3: barely tried, and only in some places
• -0: actually tried but still looks bad.

"Efficiency" seems way too vague. I try to list specific things they need to do:

Efficiency:

• -5: no array loops, just lots of IF's.
• -3: No nested if's
• -0: at least 2 useful nested if's (even if others could be)

But (and I know this isn't what you asked) in a regular coding class they're often just trying to learn the new stuff and make it work. "Good style" is often too much to ask "Efficiency" can be even scarier and more confusing.

I've never done any training exercises with TA's. Just gone over grading at the first meeting. Then discuss the upcoming assignment, and how to grade the one coming due at every other meeting.

Answer 2

You have a bigger problem than encouraging stricter grading. You need to provide consistent grading. Otherwise your scheme is fundamentally unfair.

For starters you don't have an option to fail to provide a proper rubric. If you aren't doing that, then you are failing the students. If it is a lot of work to do it, then you have a large task, but a required one.

You can make the rubric as strict as you like (though I don't really like the concept of narrow interpretations), but it has to be clear to your TAs and it has to be reasonable to your students.

One way to assure some consistency is to have more than one TA involved with each student's work. They need to agree with each other or appeal to you for a judgement. If they enter student grades into a spreadsheet you can easily see the differences and can also use it for further TA training as needed, such as, for example when one TA is consistently "too" lenient.

For an exercise with lots of parts, it might be possible to have each TA responsible for only one part. This tends to work for final exams where students answer questions, but less well for programming assignments.

Another way to achieve a good rubric is to have yourself or a small team of advanced TAs scan the student work without grading it and use what they learn to refine the rubric to assure consistency. It is probably a mistake to use this trick to create the rubric in the first place, but it gives you an idea of where the students are going wrong and need correction. An overall view. Once that is in place, the actual grading can occur.

Another trick, though not very easily done in pandemic times or with a large pool of graders, is to bring everyone together in real time to grade all the papers. This could possibly be done online (zoom) and you could be present to answer questions and make decisions.

But, again, consistency is a requirement. The rubric needs to be complete to assure that. The "strictness" is a secondary concern, but could be improved (your idea, not mine) with a proper rubric that everyone finds clear.

---

Moreover, if you try to grade things on a fixed scale that are fundamentally "fuzzy" then you have an impossible task. If you can define "efficiency" in your example, then fine. But if it is a fuzzy concept then almost every rubric is likely to leave the grading to intuition. Give precise grades on the things that are precise. But for other things, judgement and a bit of compassion are probably needed.

In CS, some things are clear, of course. If a student uses bubble sort on a large array it is clearly inefficient. But selection sort is more efficient than quick sort at a certain scale, which is why library versions of quick sort normally drop back to selection sort for small sections of the initial set.

But judgments about "proper factoring" of code are judgmental. If your feeling is "I can't define it, but I know it when I see it." then it is nearly impossible to provide a rubric that will be used consistently by a group of TAs.

Answer 3

From my experience, the best way to ensure consistency is by setting simple, clear-cut rubrics.

This can be done via a moderation exercise: have all the teaching staff mark 10 scripts together and see where the disagreements lie.

Alternatively, if you have access to moderation tools like Gradescope, then this obviates the need to meet in person.

Bottom line - just be clear about your expectations. Explain to the TAs the purpose of grading - either harsh or lenient, and the need for consistency.

Answer 4

Several suggestions, dealing with several aspects of the question.

Spend some time training the TAs. At the start of the semester, have everyone grade the same set of sample submissions from a previous semester. Meet in a group to discuss what you and they think matters when correcting inefficiencies and inelegancies. If you can reach consensus, fine. If not, make your own requirements clear to all. Perhaps repeat this exercise with the TAs after the first assignment.

Consider two separate marks for each assignment, one for correctness and one for style. Perhaps be strict on the style scale but weight that mark less.

Weigh programming assignments at the end of the semester more than those at the start, and make sure the students know this. That should mitigate the effect of strict standards at the start and teach them what they need to do to do better later.

Answer 5

For programming assignments, there is nothing better than a suite of automated tests that look at accuracy, performance, and code smells.

You can have tiers of tests:

• Tier I - basic tests
• Tier II - advanced/edge-case tests
• Tier III - performance, code smells tests

Based on the tier, the points/grade is determined.

Of course, you can use TAs to skim through the solutions to adjust the points for clever solutions and attempts to play the tests.

Answer 6

What I and the other TAs did on last semester's course to make sure we graded all students as evenly as possible:

• Before the assignment was released to the students, we went through the rubric to try to poke holes in it. We wanted to make sure that what the students were told to do in the assignment, what the assignment told them they would be graded on, and the actual rubric we would use, were all in agreement. If they weren't consistent, we'd take it back to the lecturer and propose a refinement.
• When results came in, we took a couple of submissions and graded them collectively, to calibrate between the TAs how we would apply the items in the rubric in practice. After the first assignment we had a good idea of who were strong and weak programmers so we'd pick a presumed-strong and presumed-weak submission to calibrate at both ends of the scale.
• When grading an assignment, we'd keep notes on what points were scored for each item the rubric. So you could look up why exactly a student got the grade they got, and what they should improve on a re-sit.
• We also tracked the average mark given by each TA, so that we could do an inspection if one TA's average marks were significantly higher or lower than the others.

This approach worked out quite well.

Answer 7

If you want to reduce variance, you can multi-pass marking.

All deductions come from a combination of a marked part of the code, a reason, and a deduction because of a marked part of the code.

Take such information and hand off the marked part of the code and reason to another TA, and they independently determine the deduction based off your Rubric.

If they significantly disagree, send it to a 3rd TA.

If they still significantly disagree, escalate to you, and improve Rubric.

The first TA is spending most of the time (looking for problems), the second TA only has to apply the Rubric on an identified problem, so this should less than double the marking workload.

Because you are now comparing two (or three) TA's determination of how severe something is, you can bias the average towards the more severe rating. TAs who regularly are the least severe you can review the work of, and encourage to be more severe if they need to be.

You'll note that you can even use such a mechanism to have student-student code review (looking at what other people do wrong or right is of high value when learning). Students are told to use the Rubric to review other student's code, and identify areas that violate the rules. Then TAs can review the student's selection of issues and issue marks that way.

That student-student review can also be used to spot problems that a TA is missing; if TA consistently misses violations of the Rubric that students catch (and a 3rd party TA marks as good catches), then that is a reason to consider talking to that TA about being more strict.

In short, you need visibility into the process. This is expensive to do yourself, so you need to have your large number of Students and TAs provide useful cross-review, consume the resulting data to find exceptions you want to deal with yourself, and then apply corrective action on the exceptions. That should efficiently move the marking to be more consistent, and correctly harsh.

Answer 8

It sounds to me that one of the TAs is not calibrating their marks to reasonable expectations of the student cohort, but to absolute criteria, e.g. x-points lost for each of a list of potential deficiencies. This can be unfair because the deficiencies are often correlated, rather than independent, which leads to a bimodal distribution of scores - they all either got it basically right, or basically wrong. In the U.K. system, where we have first-class, upper- and lower-second class etc., I'd ask the TA to look at the score they have given for the student and ask themselves whether that was consistent with the implied degree classification. For instance if there were 20 marks for efficiency and a marker awarded only 5, then that is saying the work is a borderline fail, according to that quality. Phrased that way, they might see that the score is incompatible with their subjective assessment of the work, rather than the objective "tick-box" score. If another marker could have given it 15 marks, it clearly isn't in fail territory!

... of course it could be the other marker is too lenient, but again it is a calibration issue, they should ask themselves (in the U.K. system), whether the work was "first class" from an efficiency perspective, as that is what the score of 15 (75%) would imply.

How the TAs would moderate their original marks is another matter, but it is a sanity-check of their calibration.

Programming has a large subjective element. The point is not just to write a program that computes the right answer (hopefully efficiently), it needs to be written in a way that is understandable by other human beings, so that it is maintainable by somebody else. Overly prescriptive rubrics can cause more problems than they are worth, because sometimes students can come up with good solutions that don't fit your rubric, and they shouldn't be penalised. Students also enjoy a degree of freedom to be creative in programming assignments, and I think that makes them better programmers in the long run.

Answer 9

Assuming "You" is is the university: Pay your TAs more to increase motivation. You cannot expect good quality work or higher motivation for bad quality pay. Generous grading is low-effort grading.

The other answers advocating rubrics and training are also correct.