Helium-4 won't accept a neutron to form helium-5, and it won't accept a proton to form lithium-5: both of those isotopes have ground states which decay by alpha-nucleon fragmentation. But you are incorrect to think that helium-4 is the only low-mass isotope which can't be formed by single-nucleon capture.
As a rule, the excitation spectrum of a "small" quantum system has wider gaps between the energy levels than the spectrum of a "large" system. When you scatter a nucleon from a nucleus $^A_N Z$ (such as $\rm^4_2 He$), the scattering is described by the excitation spectrum of the "compound nucleus" with mass number $A+1$ and whose excitation energy is given by the total energy of the system. Most of my experience is with thermal (milli-eV) neutron beams, where the mega-eV energy scales for nuclear excitations mean that the energy of the compound nucleus is just its neutron separation energy. In the literature these are sometimes called "threshold reactions."
If you put milli-eV neutrons on helium-3, you don't make helium-4. This is mostly because all of the excited states in helium-4 are unbound.
Here's a partial level scheme:
Helium-4 energy levels.
The system $\rm n+{^3He}$ at threshold has the neutron separation energy $S_n$, at the blue line. This threshold energy overlaps with the $0^+$ state at at 20 MeV, thanks to its energy width of 0.5 MeV. The $0^+$ capture state is above the proton separation energy $S_p$, which makes proton emission an allowed decay channel. Furthermore, the electromagnetic interaction can't facilitate a $0^+\to0^+$ transition to the ground state with a single gamma ray, because the photon must carry at least one unit of angular momentum. The $0^-$ state off to the right is also involved, but the analysis is the same for that case.
So neutron capture on $\rm^3He$ can't form $\rm^4He$; instead you get fragmentation into $\rm^1H+{^3H}$ without any photon emission. (Minor caveat at a related question.)
If you put milli-eV neutrons on lithium-6, you don't make lithium-7 for mostly the same reason. Here's a lithium level scheme:
Lithium-7 energy levels (also from NNDC).
Most of the $\rm(n+{^6Li})$ compound nucleus will be in the $5/2^-$ state whose width makes it overlap with the neutron separation energy $S_n$. That state (and for good measure, the $7/2^-$ state below it) is above the threshold energy for $\rm^3H+{^4He}$. In this case a photon decay to the $\rm^7Li$ ground state is allowed in principle, but the strong interaction is so much … stronger … than electromagnetism that the fragmentation is overwhelmingly preferred. Lithium-6 makes excellent cold-neutron shielding, because the neutrons turn into heavy nuclei which stop instantly in the shielding material, without emitting any photons.
You mention $\rm n+{^{13}C}\to{^{14}C}$, which is allowed, but the cross-section is tiny. The more interesting case in the mass-14 system is the compound nucleus $\rm(n+{^{14}N})$, which has an allowed transition to $\rm(p+{^{14}C})$. This is actually where carbon-14 comes from: free neutrons, liberated by cosmic ray-induced spallation, capturing on nitrogen in the atmosphere. Carbon-14 decays by beta emission because its ground-state energy is less than the nucleon separation energy for carbon-13: a consequence of the excitation spectrum getting more dense as the nuclei get larger and more complicated.
