Yes, that is a perfectly acceptable formula for entropy change even outside of thermal equilibrium.
And, as you say, it's related to the irreversibility of heat transfer. The point about ideal thermal reservoirs always exchanging heat reversibly is not necessary to understand this, so let's park it outside for a moment.
Let's imagine a body in an environment. We'll call the temperature of the body $T_b$ and the temperature of the environment $T_e$. If they exchange heat, their respective entropies, $S_b$ and $S_e$, change according to the formula you mentioned:
$dS_b=\cfrac{\delta q}{T_b}$
$dS_e=\cfrac{- \delta q}{T_e}$
Note the sign (the heat flowing into the body is leaving the environment, or vice versa).
Also note, and this is critical, that the magnitude of $\delta q$ is equal in both equations (one receives exactly the same amount of heat as the other loses), but, if $T$ is different, the entropy change won't be the same. The change in entropy (positive or negative) will depend on the temperature of the body or the environment.
Let's have a look at how the total entropy of the system ($S=S_b+S_e$) changes.
If the body and the environment are at thermal equilibrium, in other words, their temperature is the same ($T_b=T_e=T$),
$dS=dS_b+dS_e=\cfrac{\delta q}{T}-\cfrac{\delta q}{T}=0$
In other words, transferring heat between bodies at thermal equilibrium does not change the total entropy of the system.
However, if they are at different temperatures,
$dS=\cfrac{\delta q}{T_1}-\cfrac{\delta q}{T_1}=\delta q \left( \cfrac{1}{T_1}-\cfrac{1}{T_2}\right)$
Now, the sign of the entropy change depends on the relative values of $T_1$ and $T_2$:
If $T_1>T_2 \rightarrow 1/T_1 < 1/T_2 \rightarrow 1/T_1 - 1/T_2 < 0 \rightarrow dS < 0$
If $T_1<T_2 \rightarrow 1/T_1 > 1/T_2 \rightarrow 1/T_1 - 1/T_2 > 0 \rightarrow dS > 0$
Note that I chose the signs of $\delta q$ assuming that the flux of heat went from the environment to the body. So, this means that if the body is hotter than the environment ($T_1 > T_2$), this would imply a reduction in the entropy of the whole system ($dS < 0$) - which the second principle explicitly forbids. And, to the contrary, if the body is colder than the environment, that flux will be spontaneous ($dS > 0$).
So, outside of thermal equilibrium, you still can use that expression to calculate entropy changes - as long as the heat flux goes in the right direction, from hot to cold. It will also apply to heat transfers in equilibrium (and therefore reversible) - although in that case you'll be having some other energy transfer process of equivalent magnitude siphoning that energy off - otherwise the heat would increase the temperature of part of the system, breaking thermal equilibrium (that won't happen).