In some sense it is provable that there is no systematic technique which solves all problems of this type: it is a theorem that there does not exist an algorithm to solve the word problem in group theory.
There are nonetheless many specific techniques that you can try, although it helps to have a large knowledge base to know what technique will work on which group. You seem to have more-or-less picked this group presentation out of a hat, and it's not such an easy example to analyze, so it might be hard to stumble upon a good technique.
My own knowledge base of geometry and topology leads me to the solution I outline here: your group $G$ is representable as a subgroup of the group $\text{Isom}(\mathbb E^2)$ of isometries of the Euclidean plane.
Draw the square $[-1,+1] \times [-1,+1]$. Your element $a$ will act as the glide reflector isometry of $\mathbb E^2$ that takes the lower side $[-1,1] \times \{-1\}$ to the right side $\{1\} \times [-1,+1]$, given by the formula
$$A(x,y) = (y+2,x) \qquad
$$
Your element $b$ will act as the glide reflector that takes the left side $\{-1\} \times [-1,+1]$ to the upper side $[-1,+1] \times \{1\}$ given by the formula
$$B(x,y) = (x+2,y)
$$
You can easily see from this that $A^2=B^2$, namely both are equal to the translation isometry given by $(x,y) \mapsto (x+2,y+2)$. One therefore obtains a homomorphism
$$\phi : F[a,b] / N \to \text{Isom}(\mathbb E^2)
$$
defined by $\phi(a)=A$ and $\phi(b)=B$. In particular, for any element $w \in F[a,b]$, if $w \in N$ then $\phi(w)$ is the identity isometry of $\mathbb E^2$.
Now here comes the hard work, namely to prove the converse:
If $w \in F[a,b]$ and if $\phi(w)$ is the identity isometry of $\mathbb E^2$ then $w \in N$. In other words, $w$ can be written as a relator
$$w = \prod_{i=1}^K v_i r_i v_i^{-1}
$$
for some elements $v_1,\ldots,v_K \in F[a,b]$ and for elements $r_i$ each of which is one of the words $a^2b^2$ or $ab^2a$ or $b^2a^2$ or $ba^2b$ or their inverses $b^{-2}a^{-2}$ or $a^{-1}b^{-2}a^{-1}$ or $b^{-2}a^{-2}$ or $b^{-1}a^{-2}b^{-1}$.
Here's the way I know to do this, expressed rather roughly (it would take a LOT of work to formalize this, roughly speaking a beginning course in geometric group theory). Extend the original square to a tiling of the Euclidean plane by squares of the form $[2m-1,2m+1] \times [2n-1,2n+1]$. Use the given word $w$ to trace out a path in the grid of sides of these squares. The fact that $\phi(w)$ is the identity isometry will imply that this path closes up to form a loop. Use the grid as a guideline for popping (or homotoping) the loop through one square at a time until it becomes the trivial loop. Use that sequence of "popping" steps to build the required relator expression for $w$.
In essence, what one is happening in the above argument is that one is demonstrating that the grid of sides of the square tiling is the Cayley graph of $G$ with respect to the generating set $\{a,b\}$.
It will follow that your group $G$ is isomorphic to the subgroup $\text{image}(\phi) < \text{Isom}(\mathbb E^2)$ that is generated by $A$ and $B$.
Now comes some more work: you can use the theory of Euclidean isometries to analyze the elements of this subgroup. Your two generators are glide reflectors with parallel glide axes of slope $+1$. It follows that every non-identity element of your group is either a Euclidean translation or a Euclidean glide reflection along a glide axis of slope $+1$. Therefore, every nontrivial group element is nontorsion.
Your group also comes up in topology: it happens to be isomorphic to the fundamental group of the Klein bottle. That, in fact, is how I recognized your group. The topology of the Klein bottle is closely connected to Euclidean geometry, and that connection is what led me to the solution I've outlined.