Jeremy Hunt has denied that the decision to give some overseas construction workers easier access to UK jobs was a "betrayal of Brexit".