Suppose, a country 'X''s civil airline wants to fly to another country 'Z'. There are three routes. However, the shortest route is 1000km which is through country 'Y'.
Can the country 'Y' block its airspace for civil airlines of 'X' without giving any reason?
If NO, can 'X' go to an international court for this? If YES, what would the penalty for the country 'Y' be?
Yes, overflight is a privilege, not a right. A denial might cause the victims to retaliate in a similar manner, by restricting overflights or other things which are not a right. These are for example no right to immigrate, no right to trade, no right to transfer money through foreign banks.
But most sovereign countries are connected by a web of agreements, treaties, and so on. Breaking one without explanation would lead to retaliation in other areas. So when the US closed their airspace after 9/11, there was general understanding. When EU countries closed their airspace to the Bolivian president some years later, there was more irritation because the excuse looked flimsy.
It depends - if country 'Y' is one of the 134 parties to the International Air Services Transit Agreement, then the first article of that agreement grants scheduled air flights the privilege to fly across its territory without landing, and to land for non-traffic purposes.
Otherwise the answer is a little more complicated - with regard to scheduled international flights for purposes such as commercial transit of passengers, according to article 6 of the Convention on International Civil Aviation, to which the 193 members of the UN are parties:
No scheduled international air service may be operated over or into the territory of a contracting State, except with the special permission or other authorization of that State, and in accordance with the terms of such permission or authorization.
If the flight was not of this type, however, article 5 of the convention grants aircraft the right of overflight and to make non-traffic stops without needing to require prior permission - subject to the terms of the convention:
Each contracting State agrees that all aircraft of the other contracting States, being aircraft not engaged in scheduled international air services shall have the right, subject to the observance of the terms of this Convention, to make flights into or in transit non-stop across its territory and to make stops for non-traffic purposes without the necessity of obtaining prior permission, and subject to the right of the State flown over to require landing. Each contracting State nevertheless reserves the right, for reasons of safety of flight, to require aircraft desiring to proceed over regions which are inaccessible or without adequate air navigation facilities to follow prescribed routes, or to obtain special permission for such flights.
There are a few exemptions - for example Article 7 states that parties have the right to refuse permission to the aircraft of other states to load "passengers, mail and cargo carried for remuneration or hire" if their destination is within the party's territory.
Article 9 of the convention grants contracting states the right to restrict or prohibit flights in the case of emergency so long as the prohibition is applied uniformly "without distinction of nationality". This the provision which was used during, for example, the 9/11 attacks.
So the answer to your question in general terms is yes with regard to scheduled flights, such as commercial airlines, but no in general - if such an infraction took place, the affected state party could bring a dispute to the ICAO Council, which could decide to suspend the infracting state's voting rights in the Council or the ICAO Assembly.
It's generally accepted that national sovereignty extends above a nation's land, and entails a right to exclusivity in one's airspace. Thus, overflights require consent from the overflown country, either to the flight in particular or a general consent through a treaty. The main such treaty is IASTA as mentioned by CDJB, but there have been others. For instance, in negotiating the division of Germany after WWII, the Western allies allowed the Soviet Union's sector to surround Berlin on the condition that each of the Western sector be allowed an air corridor to Berlin. It was these corridors that were used in the Berlin airlift. I believe that there are also provisions in international law for aircraft in distress, but I don't know the details.
Of course, ultimately, a country experiencing unauthorized overflights doesn't have much ability to directly prevent them other than shooting the aircraft down. Shooting down civilian aircraft merely because they are violating your airspace falls into the "Things international law technically allows you to do, but is very bad optics" category. As far as I know, there's been only one case of an airliner being shot down while it was violating recognized airspace: Korean Air Lines Flight 007. This incident was strongly condemned in the West, although part of that was that the intrusion was accidental. Shooting down commercial flights intentionally violating airspace would probably still result in serious reprisals.
Since this is Politics SE and not Law SE, the answer is a trivial "if they have the means (i.e., guns) yes; otherwise, no".
The court question is best asked over at Law.
