How to screen out candidates for faculty jobs who don't know the subject

Question

I teach physics at a community college and have been on a number of hiring committees for tenure-track jobs. Usually we interview about 6 people for such a position, and of those, usually something like 3 of them show a lack of knowledge of the subject in the interview. When I say a lack of knowledge, I mean that they can't do easy freshman stuff. We will ask them a question that is posed as a request for them to pretend we're a class of students and give a short explanation of something, so it's nominally a test of their ability to teach it, but in reality we find candidates who simply don't know it. These are people who have a PhD from an accredited school. (We hardly ever interview people who only have a master's.)

[EDIT] (To clarify, I'm talking about very basic knowledge, such as understanding what Newton's laws mean. We're not asking candidates to recall obscure trivia. Personally, I don't really care whether they know which is Newton's 1st law, 2nd, or 3rd. I'm talking about candidates who actually demonstrate elementary misunderstandings of Newton's laws. Since a lot of academia.SE users are in math, I think a good analogy would be if someone applied to teach math, and that person didn't know the chain rule -- which I have heard from colleagues in math departments at community colleges is also common. Continuing this analogy, the issue is not that they fail to remember the term "chain rule." The issue is that if they're asked to differentiate sin cos x, they can't do it. They do silly things like attempting to use the product rule, as if the sine was multiplied by the cosine. Or they throw up their hands and won't try, even if the committee tries to help them out.)

Are there good strategies for screening these people out at an earlier stage, without needlessly losing too many good candidates from our pool? Here's what we're already doing:

• We require applicants to submit undergraduate transcripts, and we are less likely to interview people who have poor undergraduate grades (such as Ds and Fs in math and physics).

• We prefer applicants who have graduate degrees from more prestigious schools.

• We prefer a candidate who has taught a wide variety of courses to one who has only, e.g., taught mechanics.

Undergraduate grades do seem to correlate with what we see in the interview, but it's also possible that someone started out their undergraduate education with a weak high school background and then overcame that disadvantage. There is also the difficulty of comparing different countries' grading standards. We require graduate transcripts, but I find those hard to extract useful information from.

We are currently only asking them to give the names of references, but not to supply letters of reference along with their applications. Would it help to make them send letters?

You would think that someone with really weak competence would never get into a good graduate program, and therefore we could just not interview people who have degrees from low-prestige programs. However, our pool doesn't usually include a ton of people who have degrees from the best graduate programs, and we have also seen people in the past with degrees from renowned universities who nevertheless displayed major gaps in their knowledge, as well as highly competent people who got their degrees from no-name schools.

Answer 1

Presumably your interview is doing a good job screening out the individuals who you feel "don't know the subject" and you are trying to screen them out prior to the interview. I think a reasonable screening tool could be a a phone interview. You should probably conduct between 10-15 phone interviews to find the 6 candidates you want to interview.

While I say "phone interview", it is most often now a "Skype" interview. These interviews could be as short as 30 minutes and should be no longer than 1 hour scheduled back-to-back such that they all get completed in two long days. The phone interviews I know about have had 3-6 people (presumably the majority of the committee) present, but I think depending on department politics that you could reduce it to a smaller group (maybe even just the committee chair).

While you could simply focus the phone interview on the questions that cause candidates the most trouble during on campus interviews, given 1/2 your candidates have difficulties recalling key concepts of "random" classes, you may want to help them out a little. If you require that applicants submit a sample syllabus for freshman physics and another for an upper level elective, then the phone interview should focus on the teaching statement and sample syllabus.

During the phone interview you should also mention some of the other key classes of you department that they might teach on. If a candidate is truly clueless, this will not matter, but good candidates will realize that they need to brush up on those classes for the on campus interview.

In summary, an inability to talk about a syllabus the candidate has written would be your screening tool and the hints during the phone interview will help you minimize throwing out potentially good candidates.

Answer 2

You think that asking one or two questions on the spot in an interview is a good way to assess an applicant's knowledge of basic concepts in the discipline. However, this sounds like a relatively poor approach to me.

There are basic principles of psychometrics that are relevant here. Basic principles of item response theory are relevant. First, people differ on a distribution in latent ability (i.e., knowledge of the domain). Second, items differ in difficulty. Third, even with these two bits of information, there is a random component. So if you want to measure latent ability in applicants, then you can improve your assessment by having more items. For example, give them a written test with a bunch of items.

More importantly, I think you need to be be careful with assuming that most people can recall basic facts from the undergraduate curriculum. If you teach that curriculum, then you are likely to recall many such details. If you don't teach such material, even if you've learnt the material and would know how to look it up and solve problems when required, it may not be available to immediate working memory. It might be easy to relearn and if you were given the task of teaching a class related to that content, you may still be able to do a good job presenting it.

In summary, you may want to think about the degree which being able to recall undergraduate knowledge at the time of hire is predictive of job performance. I imagine that it might be relevant, but I also think that a whole range of other indicators would be relevant too. So in addition to trying to measure the construct better, I'd recommend that you treat it as one of several indicators of potential job performance.

Answer 3

There's a famous test in the CS world called "FizzBuzz" that companies these days are using to efficiently screen out candidates that managed to get degrees without actually learning anything. FizzBuzz (and its variants) focus on what are considered to be core concepts that any programmer regardless of language should be able to do in a matter of minutes. As reported, the results are astounding. While material on the effectiveness of FizzBuzz in recruiting has never been extensively peer-reviewed (to my knowledge), coming from that world and even being a self-taught programmer I can tell you that anyone worth their salt should be able to complete the test in at least one language, even if the solution is inelegant.

It sounds like you're running into a similar problem in your neck of the woods, and are already onto a similar solution. I'd recommend trying to find that core set of problems that any undergrad should be able to solve one way or another and use that as your screening. Let them use a textbook, maybe hint at a list of possible problems ahead of time as others have mentioned so they aren't caught off-guard, and be willing to help the way you've been doing. But make the first interviews (possibly over Skype as was also suggested) quick and to the point so you don't waste time. Establish either the base knowledge, or at least the ability to pick up on it quickly while under pressure, before proceeding to full, in-person interviews.

Those tricks have worked in my industry, hopefully they can help you in yours. We've seen that it doesn't matter who graduated with what degrees and has what recommendations or grades, the ones who can actually live up to their title and do their job are sometimes the least-decorated.

Edit: These "quick" interviews can be done on Skype in at most 30 minutes, possibly more like 15 depending on the questions. You can knock out a lot of candidates in a short period of time before dedicating serious resources to second-round interviews (which may still be online and only last an hour), assuming you think the process through in the beginning.

Answer 4

Learning about a subject is not so much about being able to reproduce that knowledge at any moment, as much as it is about being able to re-learn more quickly the next time around. At some point all it will take is a quick look at the materials, or a little bit of time to re-think the steps involved. It would be a pity to lose out on good candidates because they can't pull up knowledge immediately - and in a stressful situation.

I would suggest giving them a list of possible topics in advance. But a big list, and not much in advance - so that knowledge still plays a major role. Or, alternatively, giving them the option to briefly consult a textbook on the spot. The ability to understand things deeply should still shine through.

I guess this is the opposite of what you asked for - it's not about how to screen people out, but how to screen them 'in', in case they do know the materials and the particular interview situation is not well suited to assess that.

Answer 5

So you have a set of screening techniques, but they are not enough, and you may need to add another one (most probably a phone screen). You have a concern that doing so will alienate the candidates, which is totally understandable.

What you should do is not only introduce a new screening, but remove the ones which don't work. If undergrad transcripts show a correlation with the quality of candidates, keep that requirement. If candidates with PhDs don't show significantly higher quality over people with Masters, dump that requirement.

And don't be afraid that non-standard requirements and tests will be perceived as offensive. What is truly offensive (or, I'd rather say annoying) is wasting too much of someone's time over too small of a probability that you'll actually hire them. Asking to fill an hour-long form before even talking over the phone is offensive, asking to derive sin(cos(x)) is not.

Answer 6

To distill some of the other previous comments, I would suggest having a low bar that you expect any viable candidate could do over the phone. The important part is making it easy enough that you don't screen out nervous competent candidates, but it looks like this isn't an issue for your specific situation.

You might frame it in terms of "an initial phone screen"; it also provides an additional opportunity to get to know the candidates.

Also note this doesn't have to be direct questioning, but can be included in other questions (such as "How would you teach Newton's laws").