While @Machavity has a well documented and well thought-out answer (kudos and upvotes..) I do believe that there is an additional detail worth noting:
Removing the statue of Robert E. Lee and other Confederate figures is USELESS
Those who support removing such symbols are not ACTUALLY trying to improve race relations (or even revise history), what they are tying to do is stir up trouble, make a scene, and express 'power' in terms of making other people do something they don't really want to do. Left to their own devises these symbols could stand a millennia without garnering the concern of anyone.
Removing these symbols doesn't actually 'do' anything. Those who are offended by them will simply become offended by something else tomorrow, that is the nature of who they are. They have learned that yelling inflammatory remarks like "That's racist!" invokes a response by local and national politicians in the face of media machine that refuses to even consider the truth of such allegations, but are accepted at face value. These statues have stood for a century or more and suddenly NOW they are 'racist' etc.
Those who are trying to fix yesterday will NEVER succeed, you can't unring the bell. Those who are remove the symbols of yesterday are actually doing NOTHING to make tomorrow better.
The removal of these historical monuments actually FIXES nothing. Giving in to the mob simply empowers them for the next time. We have had too many symbolic actions, we actually need to "DO" things...this DOES NOTHING.
Edit: I haven't time right now to address all of the questions and comments posted (I intend to when I do have sufficient time...soon--ish) but some of the comments have caused me to think I need to clarify something.
It is not now, nor has it ever been, my contention that those wishing TO remove statues are trying to 'rewrite history', though others here do make that argument (and it is not without merit, I just don't happen to agree with it). In the present question the motive of those demanding FOR the removal is tangential. OP's question goes to the motives of those opposed, and not necessarily those who planned, organized or attended the events in Charlottesville (on either side).
To those who have offered dissenting opinions in the comments below. I have a simple challenge for you: Even if you are right that such statuary 'is racist' what GOOD does it do to remove them? Are those statues, or the men they represent, actually still oppressing anyone? causing harm? Or are they simply spending the taxpayer's dollars to make a scene that will not ACTUALLY improve the lives of ANYONE?
Additional improvements to my initial answer (supporting docs,etc) will be addressed at a later time.