As I understand it, voltage is an energy difference between two
points.
The potential difference, $V$, between to points is the work required per unit charge to move the charge between the two point. So voltage is not "energy difference" between two points, but the energy required per unit charge to move charge between the two points.
But consider a toaster and a refrigerator each using their own 120V
outlets. They don't use the same amount of energy right? Why?
Correct. Because the power they each use is the product of the voltage (Joules per coulomb) times the in phase current (coulombs per second) that each draws from the voltage source, which equals energy per second (joules/sec or power in watts). Multiplying that times the time in hours that each is in operation, gives you watt-hours of energy consumed.
For a toaster, whose load is primarily resistive, the product of voltage and current is power in watts. For a refrigerator, whose voltage and current are not in phase, the real power will be less as it is $P= VI cos θ$ where $cos θ$ is the power factor. Which one uses the most energy depends on the real power of each and the time each one operates.
So I recognize that, depending on the resistance, two circuits with
the same voltage could have very different currents. but current is
not a measure of energy, correct?
Correct. Current is a measure of charge (coulombs) passing a point per unit time (seconds). That is not energy. Voltage is a measure of the work (energy) required per unit charge to move the charge between two points (joules per coulomb). Multiplying the two we get:
J/coul x coul/sec = J/sec = power (watts). Multiplying that times the amount of time in seconds we get the total energy delivered in Joules.
So why wouldn't a toaster and a refrigerator both use the same amount
of energy to function?
Actually a toaster uses more power (watts, or Joules/sec) than a refrigerator. A typical pop up toaster uses between 800 and 1500 watts of power. The power of a typical domestic refrigerator is only between 100 and 200 watts. The difference is the toaster is used, maybe, 15 minutes (1/4 hr) every day. While the refrigerator is cycling on and off 24 hours a day. According to one web site over a full day a refrigerator is likely to use around 1 to 2 kilowatt hours of energy. A high power consuming 1500 watt toaster operated 1/4 hour each day will consume 0.375 kilowatt hours of energy per day, making its energy consumption less than the refrigerator.
Hope this helps.