Samantha Cameron says her fashion brand is finding post-Brexit trading with the EU "challenging and difficult".