Frequent invariably fatal phenomena - eg
Kraken / Sea Monsters - large, unpredictable, indefeatableundefeatable, ever vigilant, always fatal. These
(These MAY exist on earth currently but, if so, they are very stealthy and rapid in attack and invariable fatal :-). )Frequent rogue waves - again unpredictable and immense and invariably fatal. These
These occur on earth very occasionally and very occasionally sink even large liners.Your choice ...
If the frequency of either of these was somewhat greater than on earth
(much greater for Kraken :-) ) and if they were invariably catastrophic for boats of any sensibly achievable size then boats above that size would not evolve. Boats well above "sensible size" may be built as a trial, but if these always met with disaster the attempts would be 'discouraged'.