It was in a Star Trek book but not a recent one. Definitely not based on any of the TV episodes or movies.
- They are exploring a strange world. I know this doesn't narrow it down.
- Kirk and some other crew are transported away and transformed into giant crabs. The rest don't know what's going on. They have a mixture of alien memories and human memories.
- Kirk is able to hang on to his human self, but another one, I think Chekhov (???) was more crab-like and aggressive. They end up fighting. I think there was also a female crew member who was crabified and they were fighting over her.
- The carapace changes color when they feel different emotions. Kirk Manages to communicate with Spock by getting angry and happy so that his shell pulses in morse code.
I can't remember how they fixed it but they did turn back into people.
As well as identifying the story I would like to remember why they became crabs. It was something like the species was dying out and they needed more crabs?
Its not on https://memory-alpha.fandom.com/wiki/Crab but probably I just don't know the species or planet name to look up.