A commitment by Liberal Party leader Peter Dutton to hold a second referendum on Indigenous recognition has been rubbished by one of the leading conservative Voice campaigners.