Why would it be any different from taxing property that you simply own? In fact, where I live, blighted, disused buildings are taxed at a punitive rate.