If anything, structures should start to gather a layer of grime over them if the owner hasn't spawned in and interacted with it for awhile.
This could also be part of the upkeep of a construct. It shouldn't cause degrading as quickly as direct damage obviously, but over a span of time it would make sense for the structure to weaken naturally.