Yes; you also have to use a free hand
This is covered under Componentsthe rules on material components, on p. 203 of the PHB:
Casting some spells requires particular objects, specified in parentheses in the component entry. [...] A spellcaster must have a hand free to access these components, but it can be the same hand that he or she uses to perform somatic components
So you could wear it around your neck on a string, as long as you have a free hand to manipulate it when you cast the spell.
