"But acceleration is defined by the time derivative of velocity (which is a vector quantity) and thus by the vector difference - or am I wrong?"
You are not wrong. If the velocity changes from $\vec {v_1}$ to $\vec{v_2}$ the change in velocity, $\vec{\Delta v}$, is $\vec {v_2}-\vec{v_1}$. Its value can be found by representing $\vec {v_1}$ and $\vec{v_2}$ by arrows in the directions of the velocities and of lengths proportional to the velocities, with their tails at the same point. $\vec{\Delta v}$ is represented in magnitude and direction by the arrow that must be drawn to go from the head of the $\vec {v_1}$ arrow to the head of the $\vec {v_2}$ arrow. Clearly $\vec{\Delta v}$ won't usually have the same magnitude as $\vec {v_1}$ or $\vec {v_2}$, even if these two have the same magnitude as each other.
The mean acceleration as the velocity changes from $\vec {v_1}$ to $\vec {v_2}$ is defined by
$$\text{mean acceleration}= \frac{\vec{\Delta v}}{\Delta t}$$
The instantaneous acceleration is the limit of the fraction as we consider a smaller and smaller time interval and correspondingly smaller $\vec{\Delta v}$. We write
$$\text{instantaneous acceleration}=\lim \limits_{\Delta t \to 0} \frac{\vec{\Delta v}}{\Delta t}$$