Imagine you have an old house (or other building). It has a number of distinctive features - perhaps its windows are an unusual shape, or very large or small compared to how windows are now. Perhaps it has many small rooms or is one large open plan. Many people, when they "restore" a building like this, don't just fix up the surface finishes and repair holes and cracks -- they adjust the style to match what is current at the time. So the many small windows become one large one, or the large windows are replaced with several small ones, according to the fashion. The overly large (by the restorer's standards) fireplace is replaced with something smaller, or perhaps the fireplace is removed entirely, and so on. Rooms are merged or split, the kitchen is moved to a different place with all new fittings, and the like.
Some people, who like old houses better as they were, might consider such a restoration to be an act of vandalism, taking away or ruining "period features" that they hold to be "of interest". The use of a set phrase Victorian vandal days implies that a lot of people think that was being done to properties.
It very definitely doesn't mean that it was left unrestored for too long. It means that the restoration, which to be fair was almost certainly well-intentioned on the part of the restorer, took away what the speaker of your quote considers the good parts of the building.