I'm a trans woman, and have been (still am, potentially :/) the kind of person you describe in your post.
First thing to understand is that you literally can't stop them being nervous or uncomfortable. It's just a fact that shopping for gender affirming clothing is scary, because of the way society treats gender nonconformity (GNC). Especially for people assigned male at birth (AMAB).
But, there's a few things you can do to make it a little easier. These are some I found from my experiences.
Advertise inclusivity
This is the biggest thing. Hire visibly queer workers. Throw up a sign in your shop window showing yourself as a safe space for LGBTQIA+ individuals. Mark your fitting rooms and restrooms explicitly as gender neutral and/or trans friendly. Reach out to local LGBT groups and tell them that you're accepting.
I can't tell you how long I spent looking for a place that I felt accepted. Where I wasn't worried about getting kicked out by security (and I lived in a place where that is illegal! It still worried me). Before I started my medical transition, I ended up regularly going to a mall that was like 40 minutes away, instead of the 3 or so closer options, because the day I happened to go looking for makeup, there was a trans woman working in one store, and a drag queen working in another.
Don't ask how they're doing
This one is maybe counter-intuitive, and other trans/GNC people might have different opinions here. It also mostly applies to visibly nervous people. But, let the customer talk to you if they need something. Don't go up to them to ask if you can help find anything, or anything like that. And, somewhat obviously, don't watch them as they are shopping.
If you are required to talk to customers, then try to do so in a way that the customer can be non-committal and easily back out of the conversation. You can probably get away with "Anything I can help you find?", which lets the customer respond with "Just looking around" pretty easily. Or, I liked Juliana's suggested wording, "Hi, I'm Nicki. If there is anything I can help you with, I'll be by the counter." The goal here is to keep the onus of conversation off the customer, because social anxiety, on top of everything else, is a lot to deal with.
When I was first shopping, it would mostly be around 3 am at Walmart, so I could avoid anyone seeing me and then use the self checkouts. When I got a little bolder and started looking in the daylight, I was always terrified of people coming up to me an offering to help - even in good faith! It's just scary to have to respond when you're doing something that feels like a taboo, even if you know you're doing nothing wrong. Most of the time, after someone asked, I would just leave, because it made me too nervous. There's a lot more at play here than just my trans status, to be fair, but just the same, it would have been much easier to never have anyone talk to me.
Signpost how to use fitting rooms
This is a weird one, but one thing I found was that every store has different fitting room protocols, and figuring out what to do at each was a big hurdle. So, put up some signal indicating those protocols - I'm thinking about the couple of stores which had name plates that obviously needed filled out by a worker, vs a sign that just says go on back. What to do with clothes afterwards - leave them in the fitting room, or on a rack, or on a counter? All hard questions to answer, especially when you're too terrified to actually ask.
Conclusion
Ultimately, you will notice that there weren't a lot of strictly individual interpersonal skills here. Most of what I've described requires the store itself to be on board, and to make broader changes than 1 individual interaction. That's required, unfortunately - transphobia (which applies here regardless of whether the customer is actually trans), is a systemic issue, and requires more than just individual action to fight. Making inclusive spaces is not just about not rejecting people, but finding ways to be actively accepting, which is definitely more complex.
