I've had this idea for a while, and recently I stumbled upon a short paper from N. Gisin that formulated this idea, but I could not find a meaningful discussion on the problem. The paper that I found was https://arxiv.org/pdf/1002.1390.pdf, and Gisin argues that a simple argument can be formulated against nonlocal hidden variable theories (such as Bohmian mechanics).
To summarize the paper, if some nonlocal deterministic hidden variable theory exists - such as Bohmian mechanics, then for a typical EPR-Bohm experiment procedure where spacelike separated Alice and Bob measure entangled particles. Since they are spacelike separated, there will be a reference frame where Alice measures her particle first, and thus
$$p(\alpha | a, b, \lambda) = p(\alpha | a, \lambda)$$
Where $\alpha$ is the measurement result of Alice and $a$, $b$ are the experiment settings of Alice and Bob. However, there must also be a reference frame where Bob measures his particle first - thus giving $p(\beta | a, b, \lambda) = p(\beta | b, \lambda)$. Since these two reference frames are both equally plausible, Gisin argues that any covariant nonlocal hidden variable theory defines a local model in the sense of Bell - which is ruled out by Bell's theorem.
Some ways I think this problem can be avoided are 1) drop determinism and think of a stochastic model (since Bohmian mechanics do exist, dropping determinism altogether seems not to be the solution) 2) drop relativity and assume a universal reference frame (which is obviously not favorable) 3) drop free will (as in superdeterminism) or 4) assume retrocausality is possible (which seems to me - if superdeterminism is not assumed, the most favorable choice).
I'm still not sure if non-retrocausality implies $p(\beta | a, b, \lambda) = p(\beta | b, \lambda)$ - this condition is equivalent to parameter independence (or locality, in deterministic theories). Then does parameter independence hold when there is a clear time order? I.e., if both the settings of Alice affect the outcome of Bob and vise versa, then since either Alice or Bob must have performed the experiment one before the other, then some sort of retrocausality must have happened. This seems to be the problem here.
I also believe this issue can be extended - in special relativity if some particle travels faster than the speed of light, it can experience backward travel in time. Similarly, can nonlocality in quantum mechanics imply some sort of retrocausality?