Robert E. Lee wasn’t a Nazi, and surely would have had no sympathy for the white supremacist goons who made his statue a rallying point in Charlottesville, Virginia, last weekend.

That doesn’t change the fact that his statue is now associated with a campaign of racist violence against the picturesque town where Thomas Jefferson founded the University of Virginia. The statue of Lee was already slated for removal by the city, but the Battle of Charlottesville should be an inflection point in the broader debate over Confederate statuary.

The monuments should go. Some of them simply should be trashed; others transmitted to museums, battlefields and cemeteries. The heroism and losses of Confederate soldiers should be commemorated, but not in everyday public spaces where the monuments are flashpoints in poisonous racial contention, with white nationalists often mustering in their defense.

Some discrimination is in order. There’s no reason to honor Jefferson Davis, the blessedly incompetent president of the Confederacy. New Orleans just sent a statue of him to storage — good riddance. Amazingly enough, Baltimore has a statue of Chief Justice Roger Taney, the author of the monstrous Dred Scott decision that helped precipitate the war. A city commission has, rightly, recommended its destruction.

Robert E. Lee, on the other hand, is a more complicated case. He was no great friend of slavery. He wrote in a letter to his wife “that slavery as an institution, is a moral &political evil in any Country” (he added, shamefully, that it was good for blacks — “the painful discipline they are undergoing, is necessary for their instruction as a race”). After the war, he accepted defeat and did his part to promote national healing.

Yet, faced with a momentous choice at the start of the war, he decided he was a Virginia patriot rather than an American nationalist. He told one of President Abraham Lincoln’s advisers: “I look upon secession as anarchy. If I owned the four million slaves in the South I would sacrifice them all to the Union; but how can I draw my sword upon Virginia, my native state?” He betrayed the U.S. government and fought on the side devoted to preserving chattel slavery.

That is a grievous political sin, although he obviously wasn’t the only one guilty of it. The Civil War was an America conflict, with Americans on both sides. An honorable soldier, Lee is an apt symbol for the Confederate rank and file whose sacrifices in the war’s charnel house shouldn’t be flushed down the memory hole.

The Baltimore commission has called for moving a striking dual statue of Lee and Stonewall Jackson to the Chancellorsville, Va., battlefield where the two last met before Jackson’s death. This would be appropriate, and would take a page from the Gettysburg battlefield. A statue of Lee commemorates Virginia’s losses and overlooks the field where Gen. George Pickett undertook his doomed charge. If you can’t honor Robert E. Lee there, you can’t honor him anywhere.

For some of the left, that’s the right answer, but this unsparing attitude rejects the generosity of spirit of the two great heroes of the war, Abraham Lincoln and Ulysses S. Grant. Notably, Grant vehemently opposed trying Lee for treason.

For supporters of the Confederate monuments, removing them from parks and avenues will be a blow against their heritage and historical memory. But the statues have often been part of an effort to whitewash the Confederacy. And it’s one thing for a statue to be merely a resting place for pigeons; it’s another for it to be a fighting cause for neo-Nazis.

Lee himself opposed building Confederate monuments in the immediate aftermath of the war. “I think it wiser,” he said, “not to keep open the sores of war, but to follow the examples of those nations who endeavoured to obliterate the marks of civil strife and to commit to oblivion the feelings it engendered.” After Charlottesville, it’s time to revisit his advice.

Rich Lowry can be reached via e-mail: comments.lowry@nationalreview.com.