An act of war was an any action or event that could provoke the onset of war.
According to the Klingon Captain Vorok, the Xyrillian ship found shadowing his battle cruiser in 2151, "violated our security and disrupted our systems. Both are considered acts of war against the Empire." (ENT: "Unexpected")
Per the treaty between Earth and the Romulan Star Empire following the conclusion of the Earth-Romulan War during the 2160s, the Romulan Neutral Zone was established. According to Spock, "entry into which by either side would constitute an act of war." (TOS: "Balance of Terror")
Klingon Commander Kang, while on the surface of Beta XII-A and while under the influence of the Beta XII-A entity, believed that Captain James T. Kirk and the USS Enterprise attacked his ship and killed four hundred of his crew. As a result, he claimed the Enterprise and informed Kirk and his landing party that "You are now prisoners of the Klingon Empire against which you have committed a wanton act of war!"
After Kirk beamed the Klingon crew up from the surface of Beta XII-A and left them suspended in transporter, where he had them disarmed following transport, he informed Kang that, in fact, "you're a prisoner of the Federation of Planets against which you may or may not have committed an act of war." (TOS: "Day of the Dove")
Following the brief conflict between the USS Enterprise and Kruge's Bird-of-Prey over Genesis in 2285, Admiral Kirk contacted Kruge and warned him the Klingons' presence there was an act of war, "by violation of treaty between the Federation and the Klingon Empire." (Star Trek III: The Search for Spock; Star Trek IV: The Voyage Home)
Following the arrest of Captain Kirk and Doctor Leonard McCoy in 2293 for the assassination of Chancellor Gorkon, his successor and daughter, Azetbur, said she was willing to meet with the Federation once again under the condition that they "will not extradite the prisoners and you will make no attempt to rescue them in a military operation. We would consider any such attempt an act of war." (Star Trek VI: The Undiscovered Country)
During a 2366 discussion analyzing Admiral Alidar Jarok's claim that the Romulans had a secret base in the Neutral Zone was a lure for the USS Enterprise-D to enter the zone, Lieutenant Commander Data felt that the scenario "would not be an atypical Romulan ploy. In their long history of war, the Romulans have rarely attacked first. They prefer to test their enemy's resolve." Commander William T. Riker firmly believed that Jarok was "a plant to draw us into the Neutral Zone" causing the Federation to "look like the aggressors." Captain Jean-Luc Picard, concurred, adding, "and the Romulans would have a legitimate excuse for responding in force." (TNG: "The Defector")