Grade II-listed manor house dates back to 16th century