With a PIN, each digit has 10 possibilities, so the total possibilities for N digits is 10^N. With a pattern, each position has at most 8 possibilities (center), or only 5 possibilities (side) if the last one was in the center. Computing the total possibilities is much trickier here, because if you're always returning to the center to get more options then each "return to center" adds no meaningful entropy at all, but it's easy to compute an upper bound: 8^(N/2) * 5^(N/2). In practice a pattern will be worse that that upper bound, though.
For N=4, PIN has 10,000 possibilities; pattern has 1,600 (16% as many).
For N=6, PIN has 1,000,000 possibilities; pattern has 64,000 (6.4% as many).
And remember that the patterns will in practice be worse than that.
Furthermore, it may be easier to deduce the unlock pattern from viewing screen smudges. Sometimes the screen area where the PIN pad / swipe pattern is displayed doesn't get used after unlocking, and if the screen was clean enough before, you can figure out a lot. With a PIN, you might figure out the set of numbers used (which will drastically lower the possibilities), but won't know the order. With a pattern, you might literally be able to deduce the entire pattern at a glance (it's possible to tell where a pattern starts and ends, because the fingerprint at the endpoint is less smudged).
EDIT: Thank you to the people in the comments who pointed out that the Android pattern rules are much more complicated than I assumed (my bad; I don't use the pattern unlock on anything). Worth noting that the actual rules permit far fewer patterns than the upper bound numbers I gave above.
Also, to respond to people pointing out that the maximum guesses barely matter anyway: it may be that you're kind of correct, but for a different reason than you think. The default behavior on Android is (or was as of 2013; I don't know if this has changed) that you have unlimited guesses, just with a 30-second pause every five failed attempts. That's enough to fully brute-force the 4-digit PIN space in under a day, although of course in practice the time is vastly shortened by trying the common PINs first. https://www.theverge.com/2013/7/24/4551962/r2b2-3d-printed-open-source-robot-crack-android-pins (I don't know if the robot could attack swipe patterns, but if so, it would take still less time than PINs, for a given length.) The difference between 20 hours max for a four-digit pin, and 90 days max for a six-digit PIN, is huge... except of course humans are terrible at randomness.
Obviously, if anti-brute-forcing measures are in place (which they usually are on business-managed devices but not necessarily personal ones), a PIN or pattern only needs to be strong enough to stymie the permitted number of attempts. This is how four-digit PINs on credit and debit cards achieve a tolerable level of security. Also, ideally devices would do something similar to the current password recommendations, and disallow PINs (or patterns) too commonly used... but in practice, I'm not going to hold my breath for that.