There's a couple things to this question and I think we can treat them separately from each other. Your principle question, really, is "I want to make my riddle difficult" - the rest of it is specific questioning about a specific approach you are considering for achieving that. So let's look at that first question first.
Difficulty is, well, difficult. It's a subject that comes up with regularity, most recently here: How to make a riddle tricky without too many clues to make it easy but enough to avoid "too broad"?
The accepted answer to that question has a lot of thoughts on puzzle difficulty and how to make a puzzle not merely more difficult, but better. Rather than reiterate everything that was said there I'll just point you to that answer.
For your second question, and the crux of your "dilemma" - how do I make more abstract clues, so my riddle is harder, without making my riddle too broad?
A couple of canned thoughts, that I mention with some regularity in comments when people have gotten this wrong:
It's not sufficient that your intended answer fits all parts of the puzzle. If many answers could fit, then the puzzle is under-specified. A well-crafted puzzle will give enough information to rule in the intended solution while ruling out everything else.
[Some] riddles often allow good interpretations and answers that were not intended, as is happening here. If there is some reason (beyond "that wasn't what I was thinking of") why the existing answers do not solve this riddle, it's not apparent at all what part of the riddle actually invalidates those responses. Because it seems they should be at least as valid as any other answer you might have in mind, this may be "too broad"—you may need to update the riddle to make sure invalid responses are demonstrably invalid.
The earlier answer I referenced, from point 4 (When is a riddle really "too broad"?) onward, delves deeper here to what you need to do, and why, to make sure you don't end up with something too underdefined. (See also the other answer to the same question, which discusses avoiding making it overdefined as well, as that tends to lower the difficulty.)
But there's little guidance here on how to do that, as there are a number of good ways to do so and how a specific puzzle crafter chooses to achieve the goals are likely to differ greatly from someone else's choices, depending on their skill in using different puzzle elements. @Gareth McCaughan's answer here offers some excellent observations for how to make your riddle be more than just a riddle, so there are different paths a solver can work at to move toward a solution even in the face of more abstract clues; that's certainly a great way to go. Other riddles often end up adding hints that give additional clues toward the solution as time goes on, but as noted elsewhere these tend to feel like clumsy patches to the puzzle's difficulty; ideally, offering enough clues that converge to the same solution from different angles as part of the initial puzzle is the best way to go.
One suggestion that I can't recommend enough: ask a few people to try to solve your puzzle before you post it here! That will give you some great feedback on what people find immediately obvious, what people have to think about, and what puzzle elements they never figure out. Such feedback will help you avoid making things either too trivial, or impossibly inaccessible to anyone not reading your mind. Once you've tried this with a few puzzles and start to get a feel for what works well and what doesn't, it becomes somewhat easier to guess how a new puzzle will fare, but until you've gotten a feel for it, test solvers are very useful!
