Your post touches on several well-worn philosophical phenomena. I'll try to give an overview of them in the hopes it addresses your question, or failing that helps you decide what would. (In particular, I'll try as much to articulate these concerns as to answer them.) Wherever it seems I'm making sweeping or otherwise overly concise statements, that just reflects the further reading required to explore the phenomenon at hand.
You might alternatively be interested in how psychometrics reifies the quantification of psychological concepts (intelligence being the first historically), in which case this is technically the wrong SE, but I can suggest some reading material. But I'll assume hereafter the choice of the adjective intelligent isn't the main point of the question; you could just as easily be asking about personality traits.
most people in fact have their own
The Platonic Socrates frequently undermined people's claim to know what a word meant by challenging them to define it, then providing apparent counterexamples to either necessity or sufficiency. Yet we still seem to know when to use specific descriptions, and in particular when to disagree with them and even call some such claims lies. In short, this is by understanding whole statements, questions etc. as emerging from words' interactions; one understands a term insofar as one understands such examples.
if we assume intelligence to be a person's IQ, then we would have to know her IQ, which we assume we don't.
I don't know this person. My Bayesian prior for her IQ is the population's distribution of it. Well, I know people have commented on her intelligence, which gives me a Bayesian posterior. And if I knew why they said it, it would adjust again (in one direction if the reasons sounded especially forgiving, or in another if they didn't). So even if her IQ has never been measured, I have a kind of Bayesian knowledge, which with enough experience leads to a prior very tightly distributed around a high mean (if the evidence is suggestive of a high IQ, that is).
So far, I sound like a mathematician. A philosopher might object that intellectual dispositions are only so well-inferred in the light of past observations insofar as induction or something adjacent to it succeeds. In short, maybe she'll seem like an idiot tomorrow.
Speaking of dispositions, how well do we know a person's mental traits from their behaviours? That's a big topic in the philosophy of mind, with some even historically arguing the behaviours are all there are to the mental traits.
you want to avoid a philosophical discussion about intelligence and you want to be polite and avoid an awkward situation or an abrupt ending of the conversation.
In other words, how do you avoid voicing skeptical objections to the knowledge claim at hand? If you've familiarized yourself with answers to skepticism you find satisfactory (at least when you're not in philosophy mode), you'll have no such objections to voice. You probably don't struggle to politely hold your tongue when someone makes any old claim that assumes our senses aren't pure hallucination.
