Demolition work has started on the derelict former residential care home, Polars, on Staplers Road.
In recent months, calls have been received from the community and nearby neighbours for the building to be tidied up and made safe.
Some neighbours said they were fearing for their safety.
Ryde Demolition started work in February to clear the inside of the property and make the site suitable for building but now major work to pull down the building has commenced.
The Polars building dates back to at least 1840 and in 1955 was taken over by Somerset Care who ran the home until 2017 when financial difficulties saw the home close.
It was then sold to its current owner Laurie Gustar but has become a target of vandals and fallen into a state of disrepair.
Plans have now been placed before the Isle of Wight Council by Specialised Supported Housing (SSH) to build 24 houses to be sold on the open market plus 20 supported living units.
The supported accommodation would be aimed at disabled adults and cater for 24 residents in 12 one-bed apartments, six one-bed bungalows and two three-bed apartments.
It is proposed SSH would build the scheme and then lease the housing to providers such as Southern Housing Group or Vectis Housing, the planning statements suggest, who would then manage and maintain the property.
Through its planning agent, Andrew White Planning Consultancy, SSH says the alternative scheme is necessary because it only has ‘a limited time’ to pursue its proposal.
The planning application is still awaiting a decision by the Isle of Wight Council.