A field and a potential difference are really the same thing. When someone says "potential", that's just an abstract way of describing some condition at some point in space. If there's a field, then there are potentials distributed throughout it, or if there are different potentials, then there is a field between them.
In other words, a "potential difference" is a description of how two points in an "electric field" differ. Or, an electric field is some distribution of potential throughout space, and any two points in that field can have different potentials with respect to each other.
As an analogy, you can't have a terrain of valleys and hills, dips and peaks, without also having a concept of "height" to describe the shape of the terrain as you traverse it. As a concept, "height" cannot exist independently of "terrain", and vice versa, as they are both aspects of the same underlying thing.
A potential difference (height) implies and requires an electric field (terrain), and a field implies some variation of potential as you traverse it.
As far as potential energy goes, here's my take:
The presence of an electric field around a charge endows the charge with electrical potential energy, of which potential (in volts) is a measure. A charge that finds itself at a position in the field where potential is 10V, for instance, has 1eV more potential energy than a charge which finds itself at a position where potential is 9V.
The charge experiences a force due to the field, and this force will accelerate the charge, always towards a place of lower potential energy. In the above example, the charge started with 10eV of potential energy, but accelerated under the force exerted by the field, and will have lost 1eV of potential energy once it has been moved to the new position, where potential is only 9V.
When you "apply a potential difference" between two points in space, or in a circuit, you are establishing a field between those points. There is a now a potential gradient between the two points, and electrical potential varies as you move between them. Any charge that finds itself within that field will experience a force that will accelerate it towards a place of lower potential energy. Charges fall down the gradient (or up it if they are negative charges), in the same way a ball falls along a gravitational potential gradient.
Without the electric field, all places have the same electrical potential. Without a potential gradient for the charge to "fall" along, a charge would experience no force to accelerate it. With no net force anywhere in the space, it has the same potential energy at all places. Without a field, no work can be done on the charge. It is the field itself which "does the work" to accelerate the charge.