Adding a quiz to an application form

Question

I have thought about setting up a quiz as part of a job application form (see background below for details), where I ask about 5 multiple-choice questions which are trivial to answer (or require a minimal Google search) for people in the right field, no more than 2-3 minutes altogether. These would require a significant amount of effort to answer for people without a matching background for the position, since they would need to Google around for a while until they find the correct answer. The application submission will be possible only to those whose answers are correct, effectively blocking many of the applicants with a background mismatch who do not want to spend this much time to apply for a position they're not qualified for (at least this is my hypothesis). That is, the job application form can only be submitted if the quiz answers are correct, in addition to having completed all other requirements (name, current institution, referees, etc.). Otherwise the form will return an error until all the answers are right.

I would like to know what are possible issues I should consider with this system logistically and, mostly, ethically (I cannot really think of ethical issues here, but that's why I'm asking, to make sure). I would also like to know whether I should expect some of the relevant candidates (i.e., those with a matching background) to get annoyed by this system.

---

Background

Every time I advertise a position (at the postdoc or PhD student levels) I get lots of applicants whose background doesn't match the requirements (with wide ranges of mismatch, from wrong subfield to wrong field). Some times this is obvious from the beginning, some times I need to read the application materials in more detail. These applicants often just copy-paste their application materials from some previous call, which generally means they put little effort into it.

I have a system in place to try and minimize the amount of spam/wrong background applications I get. Essentially, there are text fields where the applicants need to insert information, like a description of how they contributed to up to 5 publications. I use this system because 1) the effort is bigger than just attaching a precompiled PDF and deters the laziest applicants and 2) it is easy to quickly spot the most uninterested/unqualified applicants because they tend to leave the 5-publication fields empty.

Still, many bad applications make it through the system, and I end up spending lots of time sorting through applications with approximately zero chance of success. If I spend 10-15 minutes per bad application, going through it plus sending a rejection email to the applicant, it's still a significant amount of time (let's say I have to go through 20-30 of these, that means between half and a full day of work wasted on this, which I could have spent helping my group members with their research).

I want to avoid all of this wasted effort, since the situation is unfair to everyone involved, including those candidates with a matching background whose applications I need to review in less detail than I would like because those spam applications are sucking time away from it (I would say, for any given opening, I usually receive only about 30-40% of serious applications, meaning all strong and weak candidates with the correct background).

Answer 1

Frankly, if I were a good candidate for your position, I might find it insulting and look elsewhere.

Note that if you try to automate the hiring process in some way, even partially, you will get both false positives and false negatives. The false negatives are something you really want to avoid if you are looking for good candidates. And those who look elsewhere are a lost opportunity.

If you are going to spend a few years working with whomever you hire then spending time in selection is reasonable. Make it seem friendly, not unfriendly.

You could, perhaps, depending on rules in place, even get others in the lab help with the initial sorting.

Answer 2

Advertising a position is a bureaucratic process with serious regulatory, legal, and university policy implications. If I were you, I would absolutely refrain from implementing any creative, nonstandard idea like what you’re describing without running it by your university’s HR department and getting them to approve it.

Even if HR approves it, as others have said, it may simply be a poor recruitment strategy. Mainly, I think it would be a way of signaling to prospective applicants that you are the kind of boss who considers his own time too precious to spend on the tedious but necessary work of filtering out unqualified candidates for the position you are advertising, but would happily waste the time of people lower than him on the career ladder.

If you only want to recruit people who have low enough levels of self esteem to consider working for such a boss, then sure, your strategy sounds great…

Answer 3

I think one important question you have to ask yourself is would this system have prevented any of the actual people you hired (and are happy with) from getting their position / applying for their position? Maybe you could even make the effort and ask them.

And while I get your frustration with applicants that do not fit the bill, some that may be from a field that is not 100% percent yours might in the end still be the right candidate and even bring in new ideas that a person with the "correct" background would not be able to see because of being to focused on one topic. A system like the one you proposed could be detrimental for applications of such persons.

Answer 4

I partially agree with Buffy's answer, although I would not use the word offended, rather irritated. If they would be super easy for me to answer I would wonder where the catch is (and probably not realize that they might be hard to answer for people outside the field).

Something to consider as well: Multiple (or single choice) questions are hard to pose in an unambiguous way, making them sometimes quite hard to answer. If this would happen, and I would fail for such a reason (assuming I would be a good fit), I would try to submit 2, maybe 3 times and then give up. Others might beat the system by submitting 20 times or just being better at googling stuff than me. From this I would say adding this might only cause good applicants to reconsider and the resilient ones to apply nevertheless. In any case, if you do this I would let people send in their first guess (no rejecting of wrong answers). Then you can look into the applications ranked by number of correctly answered questions.

You mentioned questions about e.g. contributions to papers. Could you maybe just extend this and ask another question like: How would your current learnings from your papers would help you for research in my group/the project you are applying for? Or something like: How would you approach Task X (where X is a task the prospective student would work on), specify some specific methods and how they would help you in doing X successfully.

I would guess such questions immediately show you who made an effort to look into the research of your group and think how they can really apply their skills for you. In addition the answers also already show you a little bit about the candidate and their motivation.

Answer 5

TL;DR: You have to deal with both, the imposters and those suffering from the imposter syndrome. If your quiz shall weed out the former, it will likely deter the latter.

Just as there are many applicants who are overly optimistic regarding their suitability for a job (i.e., the ones you wish to filter out), there are also many applicants who are overly pessimistic. (These pessimists are over-proportionally from disadvantaged groups who have been subtly indoctrinated that they are less capable all their life, e.g., women.) In particular in academia, where jobs are highly specialised and no candidate checks all the boxes, it is a challenge to make pessimists apply even if they are not a perfect match, lest you have to pick a far worse optimist.

If your quiz shall be any effective in weeding out the overly optimistic, it will likely also deter pessimists. For example, a pessimist may have the following thoughts triggered by your quiz:

• “Why waste time and effort on a decent application if it may be filtered out automatically due to some mistake?”

• “I don’t know this from the top of my head; I am certainly not qualified.” (In particular with respect to your remark that may “require a minimal Google search”.)

• “There are almost certainly some questions in here that shall trick the candidate in choosing a naïve but wrong answer. I found all the answers straightforward so I almost certainly did it wrong.”

• “I am not sure about the answer to that one question; I will certainly fail the quiz and I am not qualified anyway.”

Mind that these won’t be false negatives in the sense that they fail the quiz; these will be people who don’t apply at all. (I here assume that applicants know about the quiz before they write the application, which seems unavoidable to me; otherwise you have people failing because of a lack of time, etc.)

Of course, by making your quiz relatively easy, you can ameliorate the aforementioned problems, but this comes at the expense of being worse at weeding out the overly optimistic or a lot of effort to design the quiz – which I don’t expect to be worth it.

Answer 6

If you have a lot of applicants, you could make it a game.

This is something that is sometimes done by cybersecurity national organizations - they create a treasure hunt-like set of chained questions that end up with the Graal: the application page.

Of course, this is very much aligned with the mindset of these people: they love to investigate. If your area of work is similar it could work.

---

Otherwise please don't, you will lose possibly good candidates. Just make sure that in the job ad you have a set of points such as

If you

• know what a zigrob is, and you built at least a type A or B ...
• ... speak Blazarith fluently ...
• ... used a gripth measuring device in a zufl context ...

then we need you! We offer ...

You could ask in the application form for some information about these points (though it is highly unusual) and in any case during the interview you know where to start and cut short if it does not go well.

Answer 7

To me, both the question and the other answers seem to be addressing the wrong problem. I think the real problem is here:

I end up spending lots of time sorting through applications with approximately zero chance of success. If I spend 10-15 minutes per bad application, going through it plus sending a rejection email to the applicant, it's still a significant amount of time (let's say I have to go through 20-30 of these, that means between half and a full day of work wasted on this...

I would suggest that two minutes is enough to spot most obviously bad applications (i.e., the ones you propose filtering out via technical means). Two minutes of concentrated effort is actually a lot -- even if an application has some subtleties, you should be able to understand the situation within two minutes. And remember, competitive applicants will go out of their way to make it clear why they want to work with you -- if you spend two full minutes reviewing an application and it still seems like the application is not even close to competitive, then it's very unlikely that further review would lead to a good hire.

You also mention sending a rejection email to the applicant. While this is a nice courtesy, I would argue that it is not necessary for applicants who manifestly do not meet the basic requirements of the job. People in a different field, for example, are basically spamming you by sending out applications scattershot; there is no need to further waste your time with a rejection e-mail. If rejection emails are required in your part of the world, I would send them as a batch -- one e-mail bcc'd to all these applicants. Of course, I encourage you to continue sending personalized rejections to candidates who made a real effort.

Now your actual question was about implementing a quiz on the application form. I would argue that you've already done that with your requirement that applicants answer individualized questions. Whether this is a good idea is debatable (if a qualified applicant leaves these questions blank, what do you do? If you interview them anyway, the questions are a waste of everyone's time; if you reject them flat out, you risk losing a good hire). But setting this aside, it seems like you've already defined a filter. All that's left to do is to actually use this filter on your side, spending little or no time on applicants that do not provide satisfactory answers to these questions.