I would also partition the problem into two parts like you do - but instead of multiplying the generating functions together, I would just solve the problems involving the loonies and the toonies separately, and then multiply the solutions. As you have already began to get a grasp of, trying to find the coefficient of $x^\text{whatever}$ gets pretty messy pretty quickly, so any attempt to minimize the number of polynomials involved helps.
Now, as far as simplifying goes, in my experience you will generally take the following steps:
- First, get your generating function as a product of polynomials
- Simplify the products/sums using the formulas for a geometric series
- Do as much cleaning up, cancellation, grouping together as you can. If there are terms in denominators, you will usually want to write them with negative exponents instead. So, for example, $1/(1-x^2)^3$ becomes $(1-x^2)^{-3}$
- Now, as applicable, you will "unsimplify" by expanding into geometric sums, the binomial theorem or using the generalized binomial theorem:
$$\begin{align}
(1-x^r)^{-1} &= \sum_{k=0}^\infty x^{rk} &\text{Geometric sum}\\
(1-x^r)^n &= \sum_{k=0}^\infty \binom n k x^{rk} &\text{Binomial theorem,} \; n \in \Bbb Z^+\\
(1-x^r)^{-n} &= \sum_{k=0}^\infty \binom{n+k-1}{k-1} x^{rk} &\text{Generalized binomial theorem,} \; n \in \Bbb Z^+
\end{align}$$
Footnote: NoName mentioned the use of Taylor expansions. That might be reasonable and certainly seems a lot easier than this. I'm just commenting based on what I experienced so far in my combinatorics class this semester, and this is more or less what were taught. So I'm not going to say it's invalid or not: I just wanted to give this heads up as to a possible alternate method.
From there, you use the rules of polynomial multiplication to figure out the coefficient for $x^\text{whatever}$. If $(\text{whatever})$ isn't some particular number, i.e. you're just told to find it for $x^r$ in general, you might have to do it by cases since some of the summations might contribute terms and sometimes might not.
As you might imagine, it becomes a mess very fast for even simpler examples. It might prove fruitful to try very basic ones first to get a feel for the matter.