You need to limit the rate at which new connections are established. If you simply use -m limit
you would be limiting the packet rate, regardless of whether it's a new connection packet or a data packet for an existing connection.
According to this question you can use -m state
for that. The whole iptables ruleset would be:
iptables -A OUTPUT -m limit --limit 10/s -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -j REJECT
You may prefer to fine tune this rules a little, the way they are they match anything coming out.
For instance, to only limit tcp connections this way, and leave anything else through:
iptables -A OUTPUT -m limit --limit 10/s -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -p tcp -j REJECT
Note that I only changed the last rule, everything else will be accepted by default or fall down to your other rules.
To go further you should really learn iptables. @sailor already pointed you to a great book on the subject. There are plenty of good resources online as well. Once you get the concepts handled by iptables (chains, tables, rules and policies) you'll find man 8 iptables
has all the answers.
Using a frontend to iptables is also a good option if you don't have the time to get deep into iptables. I use shorewall for most firewalls I manage.