An interesting aspect not touched on by existing answers at the time of writing.
What was observed about beta decay is that for the same change in the atomic nucleus, the emitted electron (which, being charged, is easily detected) had a wide range of energies. Some electrons come out with high energy, some with low, even though the same initial and final nucleus is involved. So the issue is: how to explain this aspect?
Now it is an interesting feature of energy and momentum processes in general that the decay of one particle into two is a special case, because it has a unique solution. That is, there is only one possible value for the energies of the two decay products. But the decay of one particle into three has a variety of solutions for the energies, depending on how the energy and momentum is divided up between the products. For the first case you have
$$
E_0 = E_1 + E_2 \\
{\bf p}_0 = {\bf p}_1 + {\bf p}_2 \tag{1}
$$
(where 0 labels the initial particle, 1 and 2 the products) and for the second case you have
$$
E_0 = E_1 + E_2 + E_3\\
{\bf p}_0 = {\bf p}_1 + {\bf p}_2 + + {\bf p}_3 \tag{2}
$$
In the case of beta decay the initial particle (0) is a neutron, and the decay products (1,2,3) are proton, electron, and anti-neutrino. But of course this was not known at first. It looked as if you had just the initial nucleus, the final nucleus, and an electron (the electron can also be called a beta particle).
Let's solve equations (1). For example, we might have ${\bf p}_0 = 0$ and then ${\bf p}_1 = - {\bf p}_2$ so $|p_1| = |p_2|$. This fact can be put into the energy equation, using the relativistic relation $E^2 - p^2 c^2 = m^2 c^4$ to relate each momentum to the energy and rest mass of that particle. One finds there is enough information to get a unique solution, and one finds
$$
E_1 = \frac{M_0^2 + m_1^2 - m_2^2}{2 M_0}c^2
$$
In the case of equations (2), on the other hand, there is a range of possible values for $E_1$ depending on the directions of emission and the way the energy and momentum is divided up between particles 1,2 and 3. This is a well known fact to anyone studying particle physics. So when you see data in which an emitted particle (here, an electron) has a range of energies, i.e. some decays give one energy, some give another, then you may suspect that there are two or more further particles in the decay products, not just one (e.g. a neutron decays to three things not two). But when you see a sharp spike in the energy spectrum, with just one energy for some given decay product, then you think "ah-ha! A two-body decay process is involved!"
At the time of the proposal of the neutrino, the options were, roughly: (i) abandon energy or momentum conservation or both; (ii) propose that there is another particle involved which is hard to detect directly (it is uncharged and has low or perhaps zero mass); (iii) propose that the rest masses of otherwise identical nuclei can be different; (iv) other suggestions. Pauli recommended option (ii), but admitted that there was not much other evidence to go on.
Finally, then, to answer the question: the reason to propose the neutrino was massless is that the observed energy data can be used to constrain the masses. In the case of beta decay, some of the decays give essentially all the energy to the electron, so there is none left over for the neutrino, so its mass must be either zero or small compared to that of the electron.
