I can't speak for disciplines other than philosophy, but some of the points below may translate to other fields.
In philosophy I think that there are some departments and some philosophers who would feel that the VAP in math is very impressive, and others who will discount it completely or view it negatively.
If your research area is mathematical (e.g. philosophy of: math, logic, science, or decision theory), then teaching in a math department shows that you know math well enough that mathematicians are willing to hire you.
That's good: It's a sign that you have the technical ability to do work in your area. You could also argue (in interviews, letters of application, etc.) that teaching math to undergraduates will help you teach mathematical philosophy to undergraduates. Those people whose research is in mathematical areas of philosophy are likely to be most impressed.
But it can be bad, too: Some people on hiring committees will worry that you're too mathematical. "Is he/she really a philosopher? Will we be able to understand her/his research? Have interesting philosophical conversations?"
Philosophy hiring committees often have to go through hundreds of applications. People mean well, but they're busy, and there's time pressure, and they sometimes end up making decisions based on trivial features or biases. Your math job can be one of those factors, either a positive one or a negative one.
It may be that the math job would be most beneficial with the highest-ranked departments, but most jobs are not in those departments. I'm not sure about the first point.
If you take the math job, I recommend doing as much as you can to show that you're a "real" philosopher through your publications and conference presentations. You can highlight both the benefits of your math job to a philosophy department, and what you're also doing that's purely philosophical. You should highlight these things in your application letters in way that doesn't sound at all defensive (although some people don't read these letters). It might be even more valuable to have one of the people who writes a letter of recommendation--a philosopher--to highlight these points.
(Finally, any advice of this kind should be taken with a grain of salt. Or a whole shaker. No one really knows.)
