Are you looking for Piranesi by Susana Clarke?
It begins with the main character living in a fantastic world, and as the story progresses he starts remembering things, and searches for the truth behind them. He does eventually find that his memory had been altered. I don't want to give too much away, because it's the sort of book you're trying to figure out as you go alongside the character.