How would one start, by always trying to factor out (might be a long process though if it does not work out)?
I would not recommend it precisely because of the reasons you mention, it doesn't always work out, and even if it does, you then have to do partial fractions which can be a very long and tedious process. The most direct and (in my opinion) easiest way to solve these integrals where you have a quadratic in the denominator is:
- If you have a linear factor in the numerator, split the integrals so that you end up in a situation where you just have constants in the numerator.
- Once you have an integral involving only a quadratic in the denominator, complete the square and make the necessary substitutions to get something of the form $\int\frac{\mathrm{d}s}{s^2 \pm 1}$ or $\int\frac{\mathrm{d}s}{s^2}$, and then solve these as an arctan, logarithm or $u$-substitution in their corresponding case.
Yeah, but how do I do that exactly?
Any integral of the form $\int \frac{\mathrm{d}x}{ax+b}$, i.e. with a linear function in the denominator, can always be solved like this
$$
\int \frac{\mathrm{d}x}{ax+b} \overset{\color{blue}{u = ax+b}}{=} \frac{1}{a} \int\frac{\mathrm{d}u}{u} = \frac{1}{a} \ln|u|+C = \frac{1}{a} \ln|ax+b|+C
$$
Any integral of the form $\int \frac{Ax+B}{ax^2+bx +c}\mathrm{d}x$, i.e. with a linear function in the numerator and a quadratic in the denominator, can also always be solved using substitution:
\begin{align}
\int \frac{Ax+B}{ax^2+bx +c}\mathrm{d}x&= \int \frac{\frac{A}{2a}(2ax+b)+B - \frac{Ab}{2a}}{ax^2+bx +c}\mathrm{d}x\\
& = \frac{A}{2a} \int\frac{2ax +b}{ax^2+bx +c}\mathrm{d}x + \left( B - \frac{Ab}{2a}\right) \int\frac{1}{ax^2 +bx +c}\, \mathrm{d}x\\
& \overset{\color{blue}{u = ax^2 +bx+c}}{=}\frac{A}{2a} \int\frac{1}{u}\mathrm{d}u + \left( B - \frac{Ab}{2a}\right) \int\frac{1}{ax^2 +bx +c}\, \mathrm{d}x
\end{align}
The first integral falls into the linear function in the denominator case we just covered, so that part is solved. For the second integral we do the following:
Integrals of the form $\int\frac{1}{ax^2 +bx +c}\, \mathrm{d}x$ where you don't have a linear term in the numerator, you should always start by completing the square. i.e., rewriting the integrand as follows:
\begin{align}
\int\frac{\mathrm{d}x }{ax^2 +bx +c} &= \frac{1}{a} \int\frac{\mathrm{d}x }{\underbrace{x^2 + \color{purple}{2}\frac{b}{\color{purple}{2}a}x +\color{green}{\left( \frac{b}{2a}\right)^2}}_{\left(x + \frac{b}{2a}\right)^2}+ \frac{c}{a}-\color{green}{\left( \frac{b}{2a}\right)^2}} \\
& \overset{\color{blue}{u =x+ \frac{b}{2a}}}{=} \frac{1 }{a}\int \frac{\mathrm{d}u}{u^2 +\left(\frac{c}{a}-\color{green}{ \frac{b^2}{4a^2}}\right)}
\end{align}
And here you have $3$ cases:
If $\frac{c}{a}- \frac{b^2}{4a^2}>0$ then we can take it's square root as follows:
\begin{align}
& =\frac{1 }{a}\int \frac{\mathrm{d}u}{u^2 + \left(\sqrt{\frac{c}{a}- \frac{b^2}{4a^2}}\right)^2}\\
& \overset{\color{blue}{s \sqrt{\frac{c}{a} - \frac{b^2}{4a^2}}= u}}{=}\frac{1}{a\sqrt{\frac{c}{a} - \frac{b^2}{4a^2}}} \int \frac{\mathrm{d}s}{s^2 +1}
\end{align}
and since you mention you know derivatives of inverse functions, you should immediately recognize that $\int \frac{\mathrm{d}s}{s^2 +1} = \arctan(s)$, so after reversing the previous substitutions you're done.
If $\frac{c}{a}- \frac{b^2}{4a^2}<0$ then we can't take its square root, but we can take the square root of $\frac{b^2}{4a^2} - \frac{c}{a}>0$. So this is what we do:
\begin{align}
& =\frac{1 }{a}\int \frac{\mathrm{d}u}{u^2 \mathbin{\color{red}{-}} \left(\sqrt{ \frac{b^2}{4a^2}- \frac{c}{a}}\right)^2}\\
& \overset{\color{blue}{s \sqrt{ \frac{b^2}{4a^2}-\frac{c}{a}} = u}}{=}\frac{1}{a\sqrt{\frac{b^2}{4a^2}-\frac{c}{a}}} \int \frac{\mathrm{d}s}{s^2 -1}
\end{align}
And once you get to this point, you should remember that there's a very easy factorization of $\frac{1}{s^2 -1} = \frac{1}{(s+1)(s-1)}$, so you can apply partial fractions. This leads you to $\frac{1}{s^2 -1} = \frac{1}{2}\frac{1}{s-1} - \frac{1}{2}\frac{1}{s+1}$ and thus
\begin{align}
& =\frac{1}{a\sqrt{\frac{b^2}{4a^2}-\frac{c}{a}}} \int \frac{\mathrm{d}s}{s^2 -1}\\
& =\frac{1}{a\sqrt{\frac{b^2}{4a^2}-\frac{c}{a}}} \left[\frac{1}{2} \int \frac{\mathrm{d}s}{s -1} - \frac{1}{2} \int \frac{\mathrm{d}s}{s +1}\right]
\end{align}
and each resulting integral is just a linear factor in the denominator, which we already know how to solve from the beginning.
If $\frac{c}{a}- \frac{b^2}{4a^2}=0$ then
\begin{align}
& =\frac{1 }{a}\int \frac{\mathrm{d}u}{u^2 +0}\\
& =\frac{1 }{a}\int u^{-2}\, \mathrm{d}u\\
& =-\frac{1 }{a u} + C
\end{align}
Hope this helps!