You have to distinguish between the distance the man swims, relative to the water around him, and the total distance the man travels, relative to an observer on the river bank. The total distance relative to an observer on the river bank is the distance the man swims measured relative to the water around him combined with the distance the water moves relative to the bank. If the man swims in a straight line the total distance will be the vector sum of the two distances (life gets more complicated if the man doesn't swim in a straight line).
If you're trying to minimise the crossing time, and you don't care where on the other bank you emerge, then you need to minimise the distance the man swims, because the time is this distance divided by the swimming velocity. This is done by swimming in a direction perpendicular to the bank.
You could have different criteria. For example you might want to minimise the total distance travelled. To achieve this the man would have to swim at a different angle that would depend on the speed of the river.
Response to comment:
The diagram below shows what happens as the man swims across the river,
I've drawn the man swimming at some arbitrary angle $\theta$ at a speed $v$. The river is flowing at a speed $V$, and the time the man takes to cross is $t$. The distance swum by the man is $d_m$ and the distance the water moves is $d_r$.
The key point is that the speed the river flows affects where the man emerges on the other side of the river, but it doesn't affect the time to cross. The time to cross is simply the distance swum, $d_m$, divided by the swimming speed, $v$:
$$ t = \frac{d_m}{v} $$
and by trigonometry the distance the man swims is related to the angle $\theta$ by:
$$ d_m = \frac{W}{\sin\theta} $$
so:
$$ t = \frac{W}{v \sin\theta} $$
Both $W$ and $v$ are constants, so to minimise the time you need to maximise $\sin\theta$, and the maximum value of $\sin\theta$ is 1 when $\theta$ = 90º i.e. perpendicular to the bank.
Response to response to comment:
If we take $x$ to be the direction along the river and $y$ the direction across it, the the time taken to cross is just:
$$ t = \frac{w}{U_y} $$
where $U$ is the total velocity and $U_y$ is its $y$ component. Because $U$ is the vector sum of $v$ and $V$, its $y$ component is simply:
$$ U_y = v_y + V_y $$
But the river is flowing in the $x$ direction i.e. $V_y$ is zero, and therefore $U_y$ = $v_y$ i.e. the $y$ component of the total velocity depends only only the man's swimming speed and not on the river speed. This is why the river speed doesn't affect the time to cross.
