The New Conservatives group say the PM risks “eroding public trust” unless the numbers are cut by 400,000.