The universe cannot have begun in an inflationary phase. Note that it is not necessarily the case that there was a phase that preceded inflation. However, if there was no phase preceding inflation, then the universe had no beginning.
A common way to see this is to note that the cosmic expansion factor grows exponentially during inflation,
$$a\propto \mathrm{e}^{Ht},$$
where $H=\sqrt{8\pi G\rho/3}$ is the Hubble rate, which is constant during inflation. Such an expansion history can be extrapolated indefinitely into the past, with $a$ becoming arbitrarily small but never zero.
That's perhaps not the most physical perspective, since $a$ isn't a well defined quantity in all contexts. Another perspective is that the energy density $\rho$ (which is frame invariant for a fluid that can drive inflation) is constant in time. We can even write the metric during inflation in a way that is manifestly static,
$$\mathrm{d}s^2=-\left(1-\frac{r^2}{H^2}\right)\mathrm{d}t^2+\left(1-\frac{r^2}{H^2}\right)^{-1}\mathrm{d}r^2+r^2\mathrm{d}\Omega^2$$
(see de Sitter space). Again, $H$ is constant in time. In this sense, an inflating universe is a steady-state universe.
Where does $10^{-36}$ seconds come from? If inflation was preceded by a non-inflationary phase, then the duration of that phase is of order $H^{-1}$, where $H$ is the Hubble rate during inflation. If the inflation energy scale is $\sim 10^{15}~\text{GeV}$, then its energy density is $\rho\sim (10^{15}~\text{GeV})^4$, which leads to $H^{-1}\sim 10^{-12}~\text{GeV}^{-1}\sim10^{-36}~\text{s}$.
(I assume $c=\hbar=1$.)