It is important to understand that the only person you owe anything to is you. Yes, technically, your advisor ... advised you, paid your salary probably, etc. At the end of the day, though, this is your life, and any decent advisor will emphasise this: we have jobs to go to, you don't yet.
Remember: life happens. People have kids, fall ill, have falling outs, whatever. (Just an example: I have a postdoc who was supposed to come to work with me, fully funded, but who just decided he cannot make it work on the family side.) As PIs, this is something we have to factor in. Your supervisor will have to rejig a few things, but he should be fine. And, as I was saying, this is not your problem anyway. It is good that you worry about it, but it's his name on the grant and unless you were named a co-PI or something (which you would know because you would have had to give your approval), then your main problem is to finish your PhD.
And it seems your supervisor is okay with you switching. As remarked in the other reply, switching is disruptive in terms of time to completion of your work, so this is not a decision to be taken lightly. I have had 2 PhD students and 4 MSc students who switched over to me from other supervisors. Things I have learned in the process:
The most important thing is to see if the putative new supervisor is okay with your academic program this far. Technically, even if you already satisfy the course requirements, a new supervisor may decide otherwise and request that you take course X, which all of her/his students take as a prereq to working with them..
Also, try to understand how much of your present research work you will be able to use in your thesis. That's where having an amicable switch is particularly useful: one possibility is to keep the previous supervisor as a co-advisor or on the thesis committee, which means that the committee would have a hard time saying no to that part of the work.
And for both of these points, it may be good to make sure you understand departmental dynamics. (I am assuming you mean a new supervisor in the same department.) Make sure there is no long standing feud between the current and putative supervisors. A PhD is not a vaccine against idiocy and there are some pretty spiteful people out there..