The idea that the defining thing of every issue or historical artifact is racism and people are either racist or non-racist is something that the latest wave of fringe activists believe but that's not univerally agreed upon and there is much more to the first four Virginian presidents legacy than slavery. There is also much more to their properties than slavery.
To be critical of those properties is something but to deny their existence and put pressure on them to shutting them down, is absolutely wrong. it reeks of the culture wars that the conservatives have been doing in the 1980s