Former Tory minister warned over asylum seekers 'human tide'