There are different ways to look at what constraints Modi is facing, as well as possible motives to discriminate against minorities.
First the constraint side:
International condemnation
At the current level of nastiness, Modi's India is unlikely to suffer all that that much. Countries don't typically get involved in other countries' internal affairs (unless they have a self-interest in the matter). There are exceptions, such as the punishment that the EU has inflicted on Hungary. But that is in the context of the EU, which has budgetary strings to pull. The next level up, for India, is the UN. Safe to say it won't get punished there (China tends to veto internal affair stuff, see Myanmar).
If you ramp it up somewhat (say as in Modi's 2002 Gujarat riot tenure) then India does run into risks. Recall South Africa, it managed to stay off sanction for a long while, because it painted itself as resisting Communism regionally. But eventually the public pressure got too much and trade was sharply curtailed.
India makes a fair bit of money from IT outsourcing and services. For the bigger Western companies, those contracts are highly visible. Imagine the fallout from Nike vendors' child labor, just at a higher scale, much more easily traceable. They would divest. And India would become toxic to do business with. There is a huge India diaspora abroad, quite available to ramp up domestic Western outrage if need be.
Even a small level of uncoupling would hurt India - it can't really hook up with China and Russia is not worth hooking up with. And this is at a time in which India is trying to woo manufacturing companies like Apple. Even as many other regional powers are trying to benefit from China-Western tensions by going up the manufacturing value chain: it would risk being left behind, something it really seems to want to avoid.
I'll add another aspect under this bracket too: national self-image. India has long considered itself as a leader of developing countries and the non-aligned movement: they like to see themselves as the good guys.
While Hindu intolerance is certainly a driver for some of its population, it is hard to see most Indians taking kindly to becoming reputational pariahs on the international scene. At their current point, Modi's policies aren't that problematic, but truly amping up discrimination would cost them, especially if other countries started to chime up. Contrast with 2 UN veto members: one has built its self-image on grievances and runs fake elections. The other never bothered with elections in the first place. (You could add that a 3rd veto member runs elections but is generally blissfully unconcerned about its international reputation).
Domestic unrest risks
As others have mentioned, more excesses in minority discrimination may encourage separatism in border provinces. It may trigger riots and insurgency movements. No, the 14% of Muslims couldn't "break away", being too distributed (and to what, one might ask, the 1947 precedent has hardly been shown as best practices).
Again economic and political risks. As well, as another answer states, military and security drawbacks from unstable border areas. This makes it an unproductive policy for Modi to pursue.
Supreme Court and Constitution
I can't speak to the specifics of the Indian Supreme Court and its capacity, or willingness, to act as a watchdog over minority rights in Modi's context.
We are certainly seeing Israel's current government achieving some success in neutering its Supreme Court. The US SCOTUS is also displaying some worrying signs.
But in Western democracies - and India is a democracy, a big part of the answer to this question would be that written, or customary, constitutions and supreme courts are expected to nullify government coercions upon minorities. That may very well be the case in India as well.
Counterpoint: why would Modi escalate?
Well, first and foremost, Hindu-side, this is a religious thing. Religions aren't inherently bad, and can be said to be positive forces. But they are also hijackable and rabid adherents can become irrational. It may truly be Modi being nasty from his own, and his party's, religious convictions. That's the hardest risk to quantify and the biggest reason not to be too complacent.
Political advantage. Modi does well from promoting "defense of Hindus". Might very well be a good tactic to win the next big elections.
Assuming he's a rational actor, he would probably get most of the electoral benefits by keeping the ethnic strife on a low simmer rather than bringing it to a boil. Given enough economic disruptions, as noted in my constraints, support could turn to rejection at the polls.
Conclusion:
On balance, I find this whole notion of events in India escalating much beyond where they are, to something like "Romas in Europe, Inuits in Canada, etc..." rather alarmist and over the top. By the time it got anywhere near, and that includes official laws enforcing discrimination, international public opinion in India's trading and technology partners would be imposing punishing consequences.
Modi is somewhat aware of that - he did end up in the doghouse after the Gujarat riots, for a while. And Putin's misadventures are a salutary reminder of those risks as well: India has nowhere near the resiliency of China, nor is it as tightly integrated as a supplier into Western economies.
p.s. Keep in mind though: this assessment based on rationality is assuming Modi - or some equivalent strong leader - remains in control. A situation where constant dogwhistles whip up an ever more intolerant Hindu mob mentality and minority oppression starts to build up pressure regardless of political or economic cost is also not entirely impossible. It doesn't even have to be a situation the majority of Hindus wants, only one where a big enough group imposes intolerance, by violence if need be, upon other Hindus. Look at the French revolution for example.
