As Adrian suggests, let $y'=z$, to get the second order equation
$$
z''+a\,(z')^2+b\,z^3=0.
$$
Since the independent variable $t$ does not appear explicitly in the equation (I am assuming $a$ and $b$ are constants), we let
$$
z'=p,\quad z''=\frac{dp}{dt}=\frac{dp}{dz}\,\frac{dz}{dt}=p\,\frac{dp}{dz}.
$$
This gives the first order equation
$$
p\,\frac{dp}{dz}+a\,p^2+b\,z^3=0,
$$
which written as
$$
\frac{dp}{dz}=-a\,p-b\,z^3\,p^{-1}
$$
is a Bernoulli equation. To solve it, let $u=p^2$. This gives the linear equation
$$
\frac{du}{dz}=-a\,u-b\,z^3.
$$
I have not done the calculations, but my impression is that you will not be able to get an explicit solution in terms of elementary functions.