We know that work done by Lorentz Force $q(\vec{v}\times\vec{B})$ is $0$ on moving charge in magnetic field as velocity is always perpendicular to the force. This means that kinetic energy remains constant, however, I think that like electric and gravitational fields have a potential associated to the position of an object in them, a magnetic field should too, right? But then moving through a magnetic field will not be able to always maintain constant KE as sum of mechanical energies has to be constant, and "moving" through magnetic field will change its position in the field, hence its "magnetic potential" will change. But that does not happen.
Am I interpreting magnetic field's effects and relation to potential incorrectly? What is the correct interpretation?