You can always "veto an aspect being compelled". You don't need an X-Card, but using safety tools can help.

If I could set myself in the ideal place to handle your scenario, I would be playing with a group who were all willing to use Script Change, a safety tool that generally provides more state information than the X-Card. When I wrote down the Aspect I was worried would be controversial, I'd say as much when introducing the character, and draw the Frame Advance symbol on the index card with the Aspect so other people had a reminder to take it slow when using it against me.

There are two important things here: the ongoing table conversation about Aspects and the proper use of safety tools.

The Rulebook Is Incomplete: Aspects At Table

People can tend to put a lot of emphasis on having Aspects with funny or memorable turns of phrase in them, because that's fun and memorable, but every Aspect stands for a universe behind itself that even the best and pithiest of phrases can only hint at. Every written example of a Fate Aspect is missing this essential component, but that's to be expected, because it only really matters when you're at table trying to use Aspects other people have written and they're capable of reacting to you, something a book can never do. As such, always keep in mind that you're not writing an Aspect for the entertainment of a stranger on a different continent two years later; you're writing an Aspect for use at table. It's there to remind you and everybody else of that entire universe behind itself, but as the author of the Aspect you're not likely to tell everyone else the entire thing up front, and anyway it's hardly fair to expect them to remember it.

One thing that can help is to leave the downside in the universe behind; phrase the aspect in such a way that while it's understood to refer to your character's downside, it doesn't state it openly, making it less available to people who are just scanning the play surface looking for relevant Aspects. Keep in mind that the default assumption of Fate is that (SRD) all characters are competent: "they are the right people for the job, and they get involved in a crisis because they have a good chance of being able to resolve it for the better". Plenty of stories feature people who can acquit themselves well in an action scene teaming up with people who can't to accomplish something neither of them could do alone, so when you write this Aspect, a theme to consider is "why you're there anyway". Maybe you're More Of A Desk Operative but your analysis skills are needed in the field, so you go. Not having regular contact with the underworld can actually be a plus when you're trying to sell a cover identity, too. Or maybe you're a Scarred Veteran of the Secret War and, yes, you can only walk with a cane, but when it comes to the shadow players and their shadow stages, you've got a depth of knowledge few people can match. It can help, but ultimately it's not possible to write in such a way as to guarantee you can't be misunderstood.

This is all to say that, at table, the operational limits of an Aspect are always a topic for conversation, and the handful of Fate Points you start a session with are not your only weapon in this conversation; heck, they're not even your best weapon. When somebody tries to compel your Aspect there are more options possible than you having to take the compel or paying a Fate Point to stop it; it's entirely possible that the compel was wrongly offered in the first place because somebody else's concept of how your Aspect should operate is different from yours.

The conversation about the limits of your Aspect is, necessarily, ongoing, and requires the good faith of everyone involved that it's serious and worthy of time. Because if you pull good faith out of that? Well, the players each have their handful of Fate Points and the GM has unlimited ammo. How's that going to end? So have the conversation. With luck, other people will start to get a sense of the universe behind the phrasing and you won't need to have it as much as time goes on.

A good set of safety tools can help communicate clearly during that conversation, so here's:

Safety Tools: X-Card Considered Harmful

The X-Card doesn't have a lot of affordances to it, and there's a common misuse of it out there -- I attribute it mostly to imperfect teaching -- which goes about like so:

The X-Card is the nuclear option. Whenever something's happening that troubles you, touch the X-Card and we'll blank out the scene and move on, no questions asked.

This isn't how the card's creators envisioned it being used - at a bare minimum they usually talk about the card's use as a two-step process where you tap it to signal caution or take it to take control. Ultimately you have to be willing to blank out the scene and move on no questions asked if that's what the person who took the card is most comfortable with doing, but that's not the only use of the card.

Using the card in this way results in someone who wants to express discomfort losing all agency over how their concerns are handled because they spoke up about them, and that hasn't sat well with a lot of people. When I'm in a larger organized space where the X-Card is institutional, I try to teach it in a more nuanced matter, but as a matter of personal preference I favor Script Change because the vocabulary of video manipulation offers a lot more affordances to talk about how you want play to proceed. Also it meshes well with Fate's usual cinematic angle on things.

But regardless of what tool you use, the purpose of its use is to signal very clearly to everyone involved: this is no longer a conversation that serves a gaming purpose. This is a conversation about the limits of my safety and comfort.

You shouldn't be worried about using a safety tool. You shouldn't be concerned that your use of the safety tool won't be taken seriously when the entire purpose of a safety tool is to signal that you should be taken seriously. There is no secret double-serious mode to unlock. The safety tool is maximum serious.

Maybe you're worried about finding out that the people you game with aren't willing to take you seriously? But as painful as that might be, well, better to find out and move on than wait for disaster. Not gaming is better than bad gaming.

Oh hey, that reminds me of the secret third part to this answer that was always here:

The Move-On Gambit Reversed: The Backfire

I'm not going to say "don't ever put something problematic in an Aspect". One of the best Aspects I ever heard about was in a supers Fate game where one superheroine's Trouble was "Backwards in High Heels", which ultimately called back to a pithy newspaper comic observation about Ginger Rogers that seems so trenchant the popular consciousness has put it in a lot of wrong mouths over the years. All Fate characters are capable of proactive dramatic action, and sometimes that action is against or in spite of a blatant societal double standard about your own accomplishments.

But a problematic Aspect can be kind of a land mine, especially if the thing that makes the Aspect so problematic are its cliched or dismissive expressions in popular media.

This is because the use of Aspects at table is often very improvisational. The GM might prep a scenario with thoughtful use of character Aspects as the driver, but in many cases the use of Aspects isn't so premeditated. The game's guidance on appropriate use of Fate Points to hit Aspects should be viewed through an improvisational lens; you're not looking for reasons to shut it down, you're just making sure someone's doing something people can believe in and keep the improv going.

But the problem with improv is that when you're not being conscious and deliberate about things, you can wind up just hitting the poison frothing at the top of your brain, the cliched and dismissive expressions that popular media put there.

Confronting your own brain poison is ultimately a healthy course of action that makes you into a more considerate person, but it also takes work. Not everybody is going to show up for game night expecting, willing, or even able to put in that work.

All of which is just to say: you can't force anybody to play a game with you. It's legitimate for you to want to explore an Aspect which might turn out dramatic and cool or might be problematic, but it's also legitimate for other people to not want to do that. No matter how you frame the aspect, how the discussions go, or what tools you use, it might just be unworkable for someone else and there's nothing more to be done about it. At least you'll have dealt with it openly.