Honestly, they don't really seem swapped to me.
You must keep in mind that both the pictures on The World of Ice and Fire and the show depictions are just that, depictions, not "photographs".
There are some factors that should be taken into account:
For the book depictions, the art is by Ted Nasmith, a really skilled artist when it comes to represent landscapes, locations and architectural features; you can check for example also his works for the Tolkien Legendarium, his depictions of the various Middle-Earth places are easily the most accurate and the most faithful to the author descriptions, and the same thing can be said for his work about A Song of Ice and Fire; despite that, there is always a component of personal interpretation, taste and will to leave a personal touch by the artist.
For the show depictions, the same thing can be held true, but you must also consider that there is an additional problem to overcome: you must find in the real world some places that actually resemble the fictional locations of the books. I suppose that having a real place to shot, and making some CGI modifications in post-production, is easier and cheaper than having to entirely build CGI locations; considering that there are place that you have to entirely build from scratch without using real life locations (think about The Wall or Braavos, in example), using existing places for locations that are less impressive and distinctive means administrating your budget. And nor Casterly Rock or Highgarden are main locations in the show, so it is understandable that the greatest efforts were not put into them.
For the show locations, you have to pick places that are resembling the fictional locations, that are suitable, that don't have that many modern features, that are placed in the right climate, you must obtain the permissions to shot there... well, one does not simply walk into Mordor.
The show locations could thus never be as accurate as a painting, drawing, or another artist's impression, you have to face real life issues and take into account reality.
Highgarden
Highgarden is surrounded by three rings of white stone whose walls increase in height. Between the outer and middle walls is a famous briar labyrinth which serves to entertain as well as slow invaders. The oldest towers, squat and square, from the Age of Heroes. Newer towers are tall and slender, round fortifications dating from after the Andal invasion.
Also within Highgarden's walls are groves, fountains, and courtyards. The sept is matched only by the Great Sept of Baelor and the Starry Sept. The castle's godswood contains three weirwoods known as the Three Singers. The palatial keep is decorated with statues and colonnades. The castle's structures are covered in ivy, grapes, and climbing roses.
Highgarden is filled with flowers, singers, pipers, fiddlers and harpers. The stables have a fine selection of horseflesh, and there are pleasure boats to sail along the Mander. There are fields of golden roses that stretch as far as the eye can see. Fruits grown nearby include melons, peaches, and fireplums.
from A Wiki of Ice and Fire
Nasmiht's illustration from The World of Ice and Fire is perfectly matching the book description. The chosen location for the show, the castle of Almodovar del Rio, honestly, seems to me a very fitting location for Highgarden (without having anything that could resemble Casterly Rock); check this image of the castle (and the whole gallery on wikimedia), it really looks like Nasmith's illustration:
Casterly Rock
Casterly Rock is carved out of a great stone hill colossal rock beside the Sunset Sea. It is popularly believed to resemble a lion in repose at sunset.
The Casterlys of antiquity built a ringfort on the peak, and as millenia have passed its natural defenses have been expanded with walls, gates, and watchtowers. The base of the Rock contains large sea-carved caverns. The stone has been mined for thousands of years, so there are hundreds of mineshafts in the depths of the Rock, as well as yet untouched gold veins.
The Rock has been measured at three times the height of the Wall or the Hightower of Oldtown. It is almost two leagues long from west to east, and contains tunnels, dungeons, storerooms, barracks, halls, stables, stairways, courtyards, balconies, and gardens. In the bowels of the Rock are rooms where caged lions were once kept, and cells for the worst prisoners. The castle contains a sept.
from A Wiki of Ice and Fire
We know that G.R.R. Martin inspiration for Casterly Rock was the Rock of Gibraltar:
You can easily see that Nasmith's depiction is really resemblant to the real Rock (I especially love his addition of the two rock protrusions at the base, that look like a couple of lion's paws); the show's castle instead is a rather generic depiction of a castle over a cliff.
I don't really know the reasons of this choice, the most obvious solution would have been to shot at Gibraltar, or in a location that could be similar to it; considering, like I said, that Casterly Rock is not one of the main locations, maybe it was deemed not worthy to put too many resources into it; or maybe the plain look was done to avoid to create a too much magnificent location and prevent a "want for more" hype, for a place that we can see only a couple of times.
It certainly is rather unimpressive, maybe too much.
