Talk with your DM about this concern
Before you get too worked up about the risks with the item, first talk with your DM and table about whether or not this is something to worry about.
As you stated, this started with Yakk's answer that said:
The only risk is ... what if you lose the drum?
Now, Yakk isn't wrong, but that's really the case for every magic item/special item/meaningful item/mcguffin that's out there. And the only time that's really a concern is if the DM is of the type that takes things away - or if you made a paranoid character that has this level of security as part of their persona.
The other answers here will absolutely give you good solutions for how to keep it more safely. There may even be answers that offer character builds like a Pact of Blade Warlock to help resolve the problem.
But the bigger problem is in whether or not this is a recurring issue at a table. I was once at a table where it was, and I can tell you that as a player, it's no fun. It's a good tool in the toolkit to create a specific feeling for an encounter, but it's too easy to be overused and just leave your players feeling frustrated that they can't even do their fun stuff when they finally get a chance to play.
Which also leads to the issue that if this is a DM driven decision, then that DM will find a way around your solution as well. If their goal is stripping of items, then items will be stripped.