After the death of Thérèse Fraser Lizotte in 1986, the descendants of the Fraser family officially left the Manoir, regretfully taking their possessions with them, except for some items that were sold at auction. The huge house stood empty.
In 1979, The Canadian Heritage of Quebec bought Manoir Fraser with the intention of carrying out restoration work. Unfortunately, this was impossible at that time due to a lack of funds. The CHQ ensured some upkeep and surveillance, but in spite of these efforts, the manor deteriorated. There was damage to the veranda and the roof.
In a few years, the manor lost its prestige and splendour, attracting attention on the commercial artery where it was located.
The citizens of Rivière-du-Loup had been worried by the rumours of the manor’s sale in the 1970s. They were relieved when the CHQ bought it and saved it from being demolished. However, the fact that the building still needed essential repairs made them uneasy again.
During these years, the volunteer director of the CHQ met with local organizations, such as the Chamber of Commerce and the city of Rivière-du-Loup, as well as with representatives of the Quebec Ministère de la Culture et des Communications to encourage them to participate in saving and preserving the manor. The citizens, for their part, mobilized to join in the project.