Prime Minister Lee Hsien Loong said in Parliament that ministers had unanimously expressed their opposition to knocking down the house when Mr Lee Kuan Yew met the Cabinet in July 2011.