No, a pastor in the US cannot be forced to perform interracial or same-sex weddings against his or her religious convictions.
We need to carefully distinguish between similar sounding scenarios, because different laws and principles apply:
1. A pastor exercising his freedom of religion in his own church walls or with his own congregation.
Freedom of Religion wins here.
This is a private religious ceremony and can be subject to any restrictions the pastor and the church want to put on it. There is no "public accommodation" in a church.
A church can require a couple be church members, or be baptized into the faith, or attend pre-marital counseling. The church can refuse to marry someone previously divorced, or a same-sex couple, or even an interracial couple.
They could certainly face social blowback for those decisions, but it's not against the law, and the government can't force them to perform a religious wedding for someone they don't approve.
This wouldn't apply to a government employee (a justice of the peace), see #3 below.
2. A company offering a service to the public.
This is a "public accommodation", so federal anti-discrimination laws win here.
If you offer your reception hall to the public to be rented out for events (wedding receptions, etc), Federal civil rights laws say that you can't discriminate against someone based on their race, religion, or gender.
This is the "wedding cake" scenario: a bakery that offers their services to the public can't refuse to make a wedding cake for a same-sex couple. (They can, interestingly, refuse to write something on the cake they object to, since that is overruled by 1st Amendment "freedom of speech" considerations.)
3. A government employee is asking for religious accommodation.
This is the Kim Davis scenario, and is a little more complicated.
So on one hand, the US Supreme Court found that the government itself cannot discriminate against same-sex marriage applicants, but at the same time, a government employee is within their rights to request a "religious accommodation" based on their sincere religious beliefs.
The key, though, is that the employer (the government in this case) only has to allow for "reasonable" accommodations of that employee's religious objections. (And yes, the definition of "reasonable" is the basis for many a lawsuit.)
Let's take a simpler example of a religious accommodation: a US Marine objects to working in the kitchen on days when pork is served, because of her faith. It is entirely reasonable to only assign her mess duty on other days, or to reassign her to other grunt duties.
Kim Davis, on the other hand, refused any offer of reasonable compromise. If she objected to personally issuing marriage certificates for same-sex couples, it would have been "reasonable" to simply allow someone else in her office to issue them instead. She refused this option. She was ordered to do so by the court, and jailed for contempt when she refused. She lost all appeals.
4. A private company employee asking for religious accommodation.
This is also potentially complicated, since the ideas of a "public accommodation" and an employee's religious freedom can come into conflict.
Take a pharmacy that sells Plan B ("morning after") pills. The store sells them and has no restrictions on who can buy them, but what happens if one particular pharmacist has a religious objection to selling them?
"Public accommodation" means that the store (as a whole) can't refuse to sell them, but a "reasonable accommodation" would be having someone else on duty handle the transaction. If there is no one else on duty, state laws differ: some states disallow religious objections in this case, other states handle it differently.
In Conclusion
It can be difficult at times to distinguish between these scenarios, and which principles might apply. But the pastor's freedom of religion definitely wins out in your original scenario, public accommodation isn't a factor for private religious organizations.