The concept of proportionality has existed for a very long time the Code of Hammurabi is famous for "an eye for an eye" that establishes proportionality for punishment of a crime. Applying this concept to international relations is more recent, but less consistent, might makes right has essentially always been the primary international relation rule. The Lieber Code is essentially the precursor to the Geneva conventions that sets a lot a of wartime proportionality rules for the modern era.
The main focus of proportionate response is limiting damage to civilians. From this you can extrapolate a few things, like responding to a single jet violating airspace with a nuclear launch isn't acceptable. The rule is a rough ratio of how many civilians can be killed/injured in achieving a strategic objective, the more strategic value the more collateral is acceptable. Another big focus is protection of those who surrendered/prisoners of war, torturing or killing them is unacceptable. Prisoners should also be released at the end of a war, so if there isn't active conflict imprisoning another country's soldiers would be viewed poorly (likely considered kidnapping or extortion).
It's less documented, but another consideration in proportionality is that more powerful nations need to be more tolerant of less powerful nations. Examples here would be much of the Israel/Palestine conflict, with Israel frequently criticized for responses to near constant threats and attacks. Another example would be the US and North Korea with the Pueblo incident and axe murder incident.