You are an expert.
Don't downplay that. You've been around and answered enough on that tag that you are actually fully qualified to receive one of the many mod powers for that tag.
Further, there's still plenty of oversight. Mods, other users, and even you can re-open a question you've closed. Leave a comment when you close one, and let the author reply if they want to demonstrate that their question is notably different, if you feel the need to, but don't feel that the dupehammer has no oversight, and isn't community justified.
Lastly, so, so, so many questions need to be closed as duplicate. By demanding that your vote be followed by 4 other people before action is taken, you're wasting time and limited votes. Let them use their votes on other questions - some of which won't even be in your tag.
Don't give in to impostor syndrome. You are not only qualified, but requested, by the community no less, to participate more fully and more dramatically in improving our site.
Please don't take this lightly. You can refuse - we're not going to make you, but we're also not going to allow you to simply hide your expertise and provide a crutch that allows you to pretend you don't deserve or can't handle this task.
We trust you. We need you. Please reconsider your position on the dupehammer.