I assume it's a buck converter.
Yes, It is a synchronous buck with on-chip inductor and MOSFETs.
Would it be 8.2 kΩ * 5% = 0.41 kΩ or 8.2 kΩ * 5 = 41 kΩ?
It's self explanatory:
For \$\mathrm{TRIM}\%\$ increase, you should place a series resistor of \$\mathrm{TRIM}\$ times \$\mathrm{8.2k}\$. For example, for 5% higher output voltage, you should place 5 times 8k2 which equals to 41k.
What happens when I increase to more than 5 %?
You MUST NOT exceed +5%, as the datasheet says. But it's not clear but will happen - or at least I couldn't see anything. But it's obvious that the dynamic behaviour will change.
Also, there is increased ripple when 41 kΩ is used in series. Can someone also tell me how to reduce this?
I'm not sure but the behaviour of the converter might have changed, or you did something wrong. Make sure you placed the correct resistor and you're not disturbing the Vo node.
Normally, as can be seen from many different sync buck block diagrams, error amplifier (E/A) gets the output sample (output voltage divider) from its inverting input, and the ref voltage from its non-inverting input. And there's a compensation network across the output of the E/A and the inverting input. At AC the non-inverting input goes to ground (because it's DC) and the E/A forms an inverting amplifier. Therefore the low-side of the output divider doesn't appear in the transfer function.
So, adjusting the output voltage by playing with the high side resistor of the divider (as done with your chip) reflects directly to the transfer function and therefore any slight adjustment will change the dynamic behaviour of the converter. I don't know if the chip does a digital compensation but even if it does the behaviour still might have changed after adjustment. So that might explain why you see excessive ripple.
But still, as always, make sure that you placed to the correct components and the connections are correct.