Asking won't damage your career, but I'd be surprised if it helped.
You might not want to invoke the interpersonal as a reason, though. This is not generally going to be considered justification for working from home, if the company wouldn't otherwise allow you to do so. It may be justification for moving you to another assignment where you're working with different people.