The 40-year-old singer - who has six-week-old daughter Madison with wife Amelia Tank - famously fell out with his twin Ben when he missed his sibling's wedding to compete in 'The X Factor' semi-finals in 2009 and he admitted he has always found it difficult to be on his own