Plans to knock down a vandalised lodge and replace it with a partial replica of itself have been refused by councillors.