I know the formula R=V/I, and therefore that resistance is directly proportional to voltage and inversely proportional to current. So if I increase the resistance the voltage increases too right?
We use our mathematical equations to describe, not dictate what happens in the physical world.
Allow me to tackle this in a more abstract way as I've had similar confusions in the past.
Suppose I gave you the following equation: A = B/C.
I then give you some numbers to plug into the equation:
A = 2, B = 4, C = 2.
So we now have 2 = 4/2, which makes sense so far.
Then I tell you that A has increased from 2 to 4 and now you have to determine what changes occurred in B and/or C that account for A becoming 4.
So we are now faced with this problem:
A = 4 = B/C.
Solving this is impossible since we now have 2 unknowns - we don't know what B or C are anymore. B could be 8 and C could still be 2, or B could still be 4 and C is now 1, or B and C are any (infinite) number of fractions whose result is 4.
What a mess...but hold on, what even are A, B and C? And that's exactly the point - all the equation A = B/C does for you is describe the relationship between these mysterious variables I just made up, but it tells you nothing about the underlying physical phenomenon associated with them.
Now let's draw a parallel to R = V/I. Suppose we start with R = 2, V = 4, I = 2.
This gives us R = 2 = 4/2. Now lets increase our resistance to 4. Mathematically we are faced with the same impossible situation above since we don't know what changes in I and/or V resulted in R becoming 4.
Except this time we understand something about the underlying physical phenomenon. We know we have a DC circuit and all we did was replace our 2Ω resistor with a 4Ω one. So let's entertain two possible outcomes of doing this:
Possibility 1 claims that by putting in that 4Ω resistor, our voltage source has become stronger and the circuit is now delivering double the energy (Power = 16W). At this point we are trillionaires since we have magical resistors that create free energy for us.
Unfortunately for us, it's more likely that Possibility 2 has occurred: The 4Ω resistor has resulted in the current being halved and we are now only able to deliver half the energy (Power = 4W).
The main point is that we have two levels of understanding to solve such problems. First is our understanding of DC circuits and the physical phenomena of voltage, resistance, current etc.. And on top of that is our mathematical equations that describe the relationships between those physical phenomena.
We use our mathematical equations to describe, not dictate what happens in the physical world.