When does an interaction drop the system into an eigenstate? (i.e. when is a measurement=)
This is an ill-posed question because, first of all, the system $S$ doesn't drop into any state but each observer $O$ has a state about it, as a state $\rho$ is nothing but the coding of past measurements (so it should be named with reference to being dependent on $O$ and $S$ and not only the latter, that is why 'the state lives on the observer-system link' and is not intrinsic to just the system, that would be its algebra of observables). Interactions create correlations during the isolated evolution $\hat{U}(t_1,t_0)$ of the closed system $O\otimes S$ and only those which are stable and robust in time can be regarded as measurements from the point of view of $O$, where the process of decoherence has a major role to play. In this sense, your question is of a phenomenological nature regarding when and how different interactions between observers, systems, aparata... create stable, robust and reliable correlations between the observables of $O$ and $S$ as seen by another observer $O'$, something which depends completely on the nature of the hamiltonian $\hat{H}_{O-S}$, the strength of the couplings and the time scales involved.
I believe the answer by @ACuriousMind is quite to the point. This topic easily turns into philosophical discussions even by experts (that's why he says it is "unsettled"), but I believe some conservative but objective conclusions are unavoidable. You ask for some mathematical description but I believe the difficulty is precisely in epistemological confusions and not formalism. For a more mathematically detailed exposition please refer to the articles and the end of this answer:
INTRODUCTION AND JUSTIFICATION
The world is made of interacting physical systems $S_i$ characterized by their degrees of freedom, i.e. observable properties $\mathcal{A}^{(S_i)}_j$ subject to algebraic relations, in general noncommutative, which characterize their spectra of measurable values and complementarity (in particular, commutator relations are equivalent to specifying Heisenberg's uncertainty relations to pairs of observables and they are also usually enough to deduce their spectral representations). General operator algebras with physically motivated analytic properties are represented by operators $\hat{A}^{(S_i)}_j$ acting on Hilbert spaces. Any physical description must be done from within the world by some system studying other systems, thus fixing a system of reference $\mathcal{O}:=S_0$ specifies an observer which/who will describe the rest (or the subsystem of interest $S$ disregarding the rest as environment) by the correlations it gets by measuring their observables. Observables are then what characterize the connections, the links, between systems since they are 'channels of information' through which systems may affect/know each other via interaction. Unknowing yet how this works, $\mathcal{O}$ interacts with $S$ obtaining values of the spectra of $\hat{A}^{(S)}_j$, but if any two of them do not commute, $[\hat{A},\hat{B}]\neq 0$ the eigenvalues measured are incompatible to be well-defined at the same time
Example: measuring spin $\hat{S}_z$ in a Stern-Gerlach apparatus gives a definite eigenvalue $\pm 1/2$ with $50\%$ chance each, say $+$, this is preserved if another $\hat{S}_z$ is measured in succession by the same or other observer, $100\%$ is again $+$, but if $\hat{S}_x$ is measured afterwards then a new measurement $\hat{S}_z$ reveals that the original information $+$ was lost and again $50\%$ is obtained, therefore incompatible observables do not have common eigenstates and represent properties not well-defined at the same time.
Therefore $O$ measures from $S$ at the same time only at most maximal sets of compatible observables because these ideally have definite sharp eigenvalues at the same time, thus any system $S$ is specified by 'complete windows of interaction' given by all the possible maximally compatible subsets $\{\hat{A}, \hat{B},...\},\{\hat{X},\hat{Y}...\}...$, of its observables algebra. Now, at every interaction time between $O$ and $S$, $O$ obtains through its senses/apparata at most a collection $|a,b...\rangle$ of simultaneously well-defined eigenvalues from a complete set of observables (which is selected depends on the senses/apparata intervening at that and each interaction). So $O$ 'observes' system $S$ at $t_0$ in the state $|\Psi(t_0)\rangle =|a,b,..\rangle$, because the maximally compatible properties defining $S$ have those values at that time. Now, at a later time $O$ interacts again with $S$ but through another compatible set of observables giving a state of measured values $|\Psi(t_1)\rangle =|x,y,..\rangle$. The collection of eigenvalues of the compatible operators form an eigenstate in the Hilbert space representation.
Everything that non-relativistic mechanics is about is to study how to predict the probability that having measured a system in state $|\psi\rangle$ it will be observed at state $|\chi \rangle$ at a later time. That is given by a probability distribution on the space of possible future states for each current state, which in the general noncommutative case is the same as giving a normalized positive linear functional on the observable algebra, and by Gleason's theorem every such functional can be represented by a density operator $\rho$. Indeed, complete eigenstates $|\Psi\rangle$ of some maximal set of compatible observables are in bijective correspondence to density operators which are rank-1 projectors, i.e. $\rho_\psi=|\psi\rangle\langle\psi|$, called pure states (general mixed states are convex combinations of pure states and represent statistical mixtures of our uncertainty which pure state was actually measured). In this way, a state $\rho$ of $S$ is not only the record of the values $O$ observed but a probability disposition for future observations, since Gleason's theorem guarantees that the probability of measuring a future eigenvalue $m$ of observable $\hat{M}$ is given by the expectation value of its projector $\langle \hat{P}_m\rangle =tr(\rho\cdot\hat{P}_m)$, and by spectral decomposition the probabilities and expectations of any observables are obtained, including then transition probabilities between eigenstates $\mathcal{P}(\psi\mapsto\chi)=|\langle\chi |\psi\rangle|^2 =tr(\rho_\chi\cdot\rho_\psi)$.
This is the kinematics of quantum mechanics because no dynamical evolution has been taken into account between observer-system interactions at $t_0$ and $t_1$. When $O$ is not interacting with $S$, the latter is considered isolated or a "closed system" and is left alone to evolve during $\Delta t=t_1-t_0$. Now if $S$ is to be considered the same system at different times, everything which characterizes it must remain invariant, i.e. the observable algebra must be the same at different times, which means the operators representing it must be related by an algebra-automorphism $\mathcal{U}(t_1,t_0)$ tending to the identity when $t_1\rightarrow t_0$, this by Stone-von Neumann theorem is given by a unitary operator $\hat{U}(t_1,t_0)$ transforming the operator representation of the algebra as $$\hat{A}(t_1)=\hat{U}^\dagger (t_1,t_0)\cdot\hat{A}(t_0)\cdot\hat{U}(t_1,t_0)$$ which is nothing else that Schrödinger's equation in Heisenberg's picture. Therefore, if $O$ measured a state $\rho_0$ initially, if at a time $t_1>t_0$ observer $O$ interacts with $S$ to measure observable $\hat{B}(t_1)$ and obtain eigenvalue $b$ with probability $tr(\rho\cdot\hat{P}_b(t_1))$ where the projector evolves by the same unitary transformation by the same reasoning. As soon as $O$ perceives/detects/measures a new value $m$ of any observable $\hat{M}$ of $S$, it must update the information it had stored in $\rho_0$ by the new state $$\rho_1 =\frac{\hat{P}_m(t_1)\cdot\rho_0\cdot\hat{P}_m(t_1)}{tr(\rho_0\cdot\hat{P}_m(t_1))}.$$ This is Lüder's rule and is the noncommutative generalization of Bayes' rule for updating probability distributions by conditioning upon new information, as is justified by Duvenhage's article linked above or in Busch-Lahti articles and their books.
Now, the nature of the evolution operator IS WHAT ENCLOSES THE INTERACTIONS of the subsystems within $S$ as seen by $O$. Why is so? because a priori the automorphism preserving the identity of the system may depend on other systems, and that is what is meant by 'interaction'. Indeed, by Stone's theorem the unitary evolution operator can be recast into its infinitesimal generator the hamiltonian operator: a hermitian operator $\hat{H}_S$, with bounded from below spectra, characteristic of the isolated evolution of the whole system $S$, satisfying $U(t_1,t_0)=\exp (-i\hat{H}_S\Delta t/\hbar)$. The operator $\hat{H}_S$ must depend on the observables of $S$ and those of the other systems interacting with it, for them to intervene in the time evolution. Reasons of symmetry (e.g. homogeneity and isotropy) motivate the terms of the hamiltonian responsible of free evolution (kinetic terms like $\hat{p}^2/2m$) which only involve the observables of the system, so interactions must appear in the form of couplings between different systems observables (like $\hat{\mathbf S}_1\cdot\hat{\mathbf S}_2$ for spin interactions). Given all this, any observer $O$ just needs to know the Hamiltonian of $S$ (or the free Hamiltonian of its subsystems and the interaction couplings) to be able to evolve the observables $\mathcal{A}^{(S)}_j$ in an operator representation via Heisenberg's equation using $\hat{U}(t_1,t_0)$, wich along with the information it already has in a previous measured state $\rho_0$ allows it to establish all the probability distributions of future measurements via the laws established above. But time evolution can be mathematically mapped into state-space instead which just amounts to evolving $\rho(t)$ and fix $\mathcal{A}^S$ giving the usual Schrödinger's equation.
The categorical error in ontology is bestowing reality to intermediate states $\rho(t)$ while $S$ is isolated between measuring times $t_0<t<t_1$: $\rho(t)$ evolves $\rho_0$ into a superposition of eigenstates which generically do not correspond to a pure state of measurements/information of any observer, i.e. Schrödinger's picture intermediate states are not justified to represent physical sates from an operational empiricist interpretation of mechanics. We should suspend judgment about the realist asserting a superposition of all histories and such, because they are not seen by any physical observer and come by from a mathematical convenience outside the original physically motivated meaning of observables and states we started with.
It is clear that the description any observer can make of any system, is intrinsically incomplete because it does not include the $O-S$ interaction. This is unavoidable as observables are meaningful only as "communication channels" between systems. However, quantum mechanics is complete because another observer $O'$ can model by the same theory the measurements made by $O$ by considering the coupled system $O\otimes S$, since now $O'$ has access to both the observables of $O$ and those of $S$ and can study their couplings. Hence, a measurement of $S$ by observer $O$ is nothing but an interaction in the time evolution of the $O\otimes S$ system as seen by another observer $O'$. What $O'$ sees is degrees of freedom of $O$ getting correlated with degrees of freedom of $S$ (e.g. the apparata of $O$ is always measured by $O'$ to be in the same position when $S$ is measured by $O'$ to be in eigenvalue $a$), and those correlations of observables are measurements. The dynamics used by $O'$ through $\hat{U}$ uses $\hat{H}_{O-S}$, something which $O$ could not use, that is why it was not able to describe its own interaction and measurement process. Besides, quantum mechanics is consistent because dynamics guarantees that $O$ and $O'$ get the same values when comparing upon measuring the eigenvalues of the same observable of $S$ if this was left to evolve isolated.
Read Smerlak, Rovelli or Englert for discussions of how to apply the quantum formalism consistently and check out that different observers may have different information on the system but neverthelesss agree. (In short, they agree because either they are decohered and measure compatible observables so they get the same eigenvalues, or they are observers out of contact who are regarded as part of the closed system from the point of view of each other; however if they are to compare measurements they must interact, thus decohering and each observer will see <an eigenvalue of the system> and <the other observer seen that eigenvalue> both with the same probability, grating consistency; cf. Rovelli's articles).
van Kampen’s moral: If you endow the mathematical symbols with more meaning than they [operationally] have, you yourself are responsible for the consequences, and you must not blame quantum mechanics when you get into dire straits…
SUMMARY: the answer to what makes the system project into an eigenstate is precisely the tricky part: the system doesn't drop into anything, only the observer's information about the system "drops" into an eigenstate, because eigenstates are the results of measuring observables of the system by the observer. Your "dropping" is the "collapse of the wavefunction" which is just the noncommutative analogue of the classical Bayesian update of probability distributions after new knowledge is acquired. There is nothing "collapsing" or "dropping" at each measurement because the state is an informational device, a book-keeping of the observer-system past interaction history. The whole point of my answer is that physical interpretation is justified only for Heisenberg's picture: states do not evolve and then collapse, system's observables evolve and you update the info you have at each measurement. Quantum Mechanics just gives the conditional probabilities of questions "having measured a,b.. what is the probability of measuring x,y.. later?". Thus the "state is on the observer-system link" because different observers may have different information of the system due to different past interaction history, so the state is not something intrinsic of the system but relative/relational. However as soon as observers get into contact, dynamics guarantees they will see the same measurements as long as they are of compatible observables and the system was isolated. This is because in the quantum case measuring an observable destroys previous information on a complementary observable. In classical mechanics a common state of the system among observers is possible because it is the commutative limit and all observables have well defined values at the same time for a pure state.
Therefore a measurement is any interaction of the system $S$ with the observer $O$ which establishes a robust, stable correlation between some of their degrees of freedom, but this is a process that can only be described by another observer $O'$ who will see it depend on the couplings between the observables of the compound system $S\otimes O$. So only those interactions in the coupling of the hamiltonian $\hat{H}_{O-S}$ which create correlations robust enough to appear at the experimental sensitivity of $O$ will qualify $O'$ to say that the interaction of $O$ and $S$ was a measurement.
Einstein taught us time and length are relative, Heisenberg taught us that quantum entities do not have absolute definite eigenstates in-between measurements (because if insisting on Schrödinger's picture, the general intermediate superposed state given by unitary evolution, is not justified to be empirically physical in the sense that there is no observer for which it is an eigenstate, as generically there are no physical observables which diagonalize it).
