Downing Street insists "we will not be lowering workers' rights" but does not rule out changes to the EU working time directive.